To force Google to remove pages it is better to serve a real page with a noindex meta tag than to return a 301 or 404 response.

This note refers only to removing content on a website you control from Google. Removing content on other sites is outside the scope of this note.

Google might have indexed content from your site which you did not want in the index, or wanted to be indexed differently. One example is if you published pages which were under development on your site or even an entire copy of your site for test purposes. They could also be obsolete pages after a site update.

The best solution here would have been to tell Google not to index the site in the first place, through the robots meta tag or the robots.txt file. But if you did not do this you may find that Google found your pages and put them in the index. Google is remarkably efficient at finding your pages, even if you never put them in your sitemap or never submit them for indexing.

Having the extra pages in the index can be a problem. Having many pages in your index makes it harder to see if Google is indexing the right pages for you. If some of the pages are identical Google will detect this and may not index the pages you intend at all, keeping the wrong ones in the index. Google may also impose a penalty on your results due to duplicate content. It could also give the appearance of disorganised hosting if searches for your content come up under strange URLs.

In theory, it is simple to get rid of these pages. Google tells you to remove the content by returning a 404 (page not found) response for removed pages or a 301 (permanent redirect) for pages which have moved. The 301 solution seems the best as people who click on old links or links which are still in the index are directed to your new pages. Google says that there is no need to resubmit these URLs to their URL Removal Tool as they will discover the changed and removed content during normal web crawling and update the index accordingly.

In my experience, this does not happen. I set pages to 301 and even 404 but they remained in the index even after a couple of months. I submitted the pages to Google's URL removal tool and they were removed after a few days but then they came back.

A clue to why this was happening was in the cached pages Google was showing in the index. The pages in question were from a test version of the main site, with URLs similar to http://testsite.mydomain.com/example. In a search for site:mydomain.com these pages were included, even though clicking on the link in the index returned a 404 response. In the index a cached page was included from just a few days ago, even though the page had returned 404 for a number of weeks. I noticed that the cached page was not of the indexed link, but from the main site http://www.mainsite.com/example.

It seems that Google linked the two pages because it detected that they were duplicates. It then kept the page at the test site in the index even though it had been deleted, because the duplicate page was still available.

To cure this, I created a simple HTML file with a link to the main site. In the header I included a robots meta tag instructing Google not to index the page:

<meta name="robots" content="noindex, follow, noarchive">

I then submitted the page to the URL removal tool and within 24 hours the page was gone. So it seems the solution is to serve a page which is different from the main site so that Google understands that the pages are not duplicates. Google will also note that the old page should not be in the index.

For pages you want to redirect, perhaps because of a change in your site organisation, you may still want to redirect the page to your main site. You can do this as long as you serve the redirected page with the noindex tag. How you achieve this will depend on how you serve your site but it seems to work well. Google sees the page with the noindex tag and removes it from the index promptly. Users who follow the links will see pages from your main site as you wanted.

We use WebPack as our build platform and now automatically set pages to be marked as noindex unless the build is for a production deployment. Within a site, we can mark individual pages as noindex if we do not wish them to be included, even if the main site is indexed. Our monitoring solution includes a check to make sure that production sites are marked to be indexed, to avoid a build error leading to the entire site becoming removed from the index.

Most sites seem to encourage you to let Google cache your site so that it is available to users in case your server is offline. Not allowing Google to cache your site does not however seem to impose any penalty on your search rankings.

Our preference is to ask Google not to cache our sites (by including the noarchive instruction in the robots meta tag). We do this because:

• We are confident that our sites are always available
• Preventing Google from maintaining a cache means you do not need to be concerned with removing information from the cache if old or incorrect data was posted on your site

Not caching your pages gives you control about what is in the index and cache which is why we do not allow Google to cache our sites.