Robots.txt & url removal vs. noindex, follow?

[nicole.healthline]

When de-indexing pages from google, what are the pros & cons of each of the below two options:

1. robots.txt & requesting url removal from google webmasters

• Use the noindex, follow meta tag on all doctor profile pages
• Keep the URLs in the Sitemap file so that Google will recrawl them and find the noindex meta tag
• make sure that they're not disallowed by the robots.txt file

[Marcus_Miller]

Great, comprehensive answer from Ryan as ever.

Nothing more to see here folks.

Move along now.

Move along.

[RyanKent]

The preferred option would be the noindex, follow tag.

The robots.txt file is a choice of last resort. The best robots.txt file for a site is an empty file (i.e. no disallows). The robots.txt file is a tool that can be used when other options are either not available, or the effort is deemed as too great.

If you use robots.txt and the url removal from google, that will work, the page will get de-indexed, but then Google will never crawl that page again and therefore not follow any of the links on that page. You are blocking their crawler so your site will not be crawled as thoroughly which means pages can be missed, a lower pecentage of your pages will be indexed (mainly applies to larger sites), and the link juice which flows to any of the blocked pages will lose their value. Any anchor text or other link value on those pages will be lost as well.

If you use the "noindex, follow" tag then those pages will still be crawled, those pages will continue to contribute value to your site and the page's links will continue to offer value to their target URLs, many of which will be your site's internal pages.

A final point is the URL removal tool in Google WMT will remove the page from Google, but it wont affect Yahoo, Bing and other directories.