Block Domain in robots.txt

[zeepartner]

Hi.

We had some URLs that were indexed in Google from a www1-subdomain. We have now disabled the URLs (returning a 404 - for other reasons we cannot do a redirect from www1 to www) and blocked via robots.txt. But the amount of indexed pages keeps increasing (for 2 weeks now). Unfortunately, I cannot install Webmaster Tools for this subdomain to tell Google to back off...

Any ideas why this could be and whether it's normal?

I can send you more domain infos by personal message if you want to have a look at it.

[donford]

Hi Philipp,

I have not heard of Google going rogue like this before, however I have seen it with other search engines (Baidu).

I would first verify that the robots.txt is configured correctly, and verify there is no links anywhere to the domain. The reason I mentioned this prior, was due to this official notification on Google: https://support.google.com/webmasters/answer/156449?rd=1

While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results.

My next thought would be, did Google start crawling the site before the robots.txt blocked them from doing so? This may have caused Google to start the indexing process which is not instantaneous, then you have the new urls appear after the robots.txt went into effect. The solution is add the meta tag noindex, or block put an explicit block on the server as I mention above.

If you are worried about duplicate content issues you maybe able to at least canonical the subdomain urls to the correct url.

Hope that helps and good luck

[zeepartner]

Hi Don

Thanks for your hint. It doesn't look like there are any links to the www1 subdomain. Also, since we've let the www1-Subdomain return 404's and blocked it with robots, the indexed pages increased from 39'300 to 45'100 so this is more than anybody would link to... Really strange why Google just ignores robots and keeps indexing...

[donford]

Hi Phil,

Is it possible that google is find the links on another site (like somebody else has your links on their site)? Depending on your situation a good catch all block is to secure the www1 domain with (.htaccess/**.**htpasswd ) this would force anybody (even bots) to provide credentials to see or explore the site. Of course everybody who needs access to the site would have the credentials. So in theory you shouldn't see any more urls getting indexed.
Hope that helps,

Don

[zeepartner]

Thanks for the resource Chris! The strange thing is that Google keeps indexing new URLs even though it is clearly blocked via robots.txt...

But I guess I'll just wait for these 90 days to pass then...

[Chris.Menke]

Phillip,

If you've deleted the URLs, there's not much else for you to do.  You're experiencing the lag between when Google crawls and indexes pages new pages and when it finds and removes a 404 URL from it's index.

You should think 90 days as an approximate time frame for your page count in the index to start dropping.  Here's more from google:
https://support.google.com/webmasters/answer/1663419