Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear

-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://azwa.1clkaccess.in/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Strange landing page in Google Analytics
Hello MOZ Community, The website in question is https://x-y.com/ When i looked at the landing pages report in GA , x-y.com is appended at the end of every URL like this. https://x-y.com/x-y.com When i open the above URL in GA interface, it shows page not found. This is obvious as there is no such URL.
Reporting & Analytics | | Johnroger
The metrics like sessions, Users, Bounce rate all look good. In the property settings, The default URL is written like this http:// cell-gate.com (Please note that s is missing in property settings). But how is traffic tracked correctly How do i solve this problem. What settings should we change to make the landing pages report look ok Thanks0 -
UTM Links Showing Up as Separate Pages in Google Analytics
Hey everyone, I was just looking at landing pages in Google Analytics, and in addition to just the URL of the landing page, the UTM links are being listed as separate pages. Is this normal? I anticipated seeing the landing page URL and then using the secondary dimension to see source/medium. If this isn't normal, what would I check next?
Reporting & Analytics | | rachelmeyer0 -
% Change - Google analytics - how to calculate?
Hi All,
Reporting & Analytics | | JohnPalmer
I have two dates with two different numbers I want to calculate the "% Change" like google analytics, The numbers of June 2015 - 127,931 sessions
The numbers of June 2014 - 90914 sessions please tell me what is the %Change. Best. J0 -
Google Search Console - Why is my average mobile position better than my average desktop position?
I'm wondering why my average mobile position is much better than my average desktop position. I'm wondering if Google is comparing the same queries for both mobile vs desktop or if they're only showing me the top ranked for each type of search. Is it example 1 or 2? Example 1: Desktop may have 5,000 ranking queries that average to 21.6
Reporting & Analytics | | Pauly_Gigs
Mobile may have 1,500 ranking queries that average to 8.5 OR Example 2: example.com has 5,000 total ranking keywords, those queries' average ranking in a desktop search 21.6 and mobile search 8.5. I'm curious to know exactly what I'm seeing in Google's Search Console. https://08875344305734164866.googlegroups.com/attach/777ae98664ed418f/Mobile%20VS%20Desktop.png?part=0.1&view=1&vt=ANaJVrEHOjRLlPH43i00NnC8PxaG3ct7bsHum_TWnUoa7xVamCpRp8jrvRQJL-gz4n7Q0otqKcKxcAJA5z1VySs2naQU_Zy5tDps6bJhUSZsLRQq4uU-tJQ0 -
Why google stubbornly keeps indexing my http urls instead of the https ones?
I moved everything to https in November, but there are plenty of pages which are still indexed by google as http instead of https, and I am wondering why. Example: http://www.gomme-auto.it/pneumatici/barum correctly redirect permanently to https://www.gomme-auto.it/pneumatici/barum Nevertheless if you search for pneumatici barum: https://www.google.it/search?q=pneumatici+barum&oq=pneumatici+barum The third organic result listed is still http. Since we moved to https google crawler visited that page tens of time, last one two days ago. But doesn't seems to care to update the protocol in google index. Anyone knows why? My concern is when I use API like semrush and ahrefs I have to do it twice to try both http and https, for a total of around 65k urls I waste a lot of my quota.
Reporting & Analytics | | max.favilli0 -
Sudden Increase In Number of Pages Indexed By Google Webmaster When No New Pages Added
Greetings MOZ Community: On June 14th Google Webmaster tools indicated an increase in the number of indexed pages, going from 676 to 851 pages. New pages had been added to the domain in the previous month. The number of pages blocked by robots increased at that time from 332 (June 1st) to 551 June 22nd), yet the number of indexed pages still increased to 851. The following changes occurred between June 5th and June 15th: -A new redesigned version of the site was launched on June 4th, with some links to social media and blog removed on some pages, but with no new URLs added. The design platform was and is Wordpress. -Google GTM code was added to the site. -An exception was made by our hosting company to ModSecurity on our server (for i-frames) to allow GTM to function. In the last ten days my web traffic has decline about 15%, however the quality of traffic has declined enormously and the number of new inquiries we get is off by around 65%. Click through rates have declined from about 2.55 pages to about 2 pages. Obviously this is not a good situation. My SEO provider, a reputable firm endorsed by MOZ, believes the extra 175 pages indexed by Google, pages that do not offer much content, may be causing the ranking decline. My developer is examining the issue. They think there may be some tie in with the installation of GTM. They are noticing an additional issue, the sites Contact Us form will not work if the GTM script is enabled. They find it curious that both issues occurred around the same time. Our domain is www.nyc-officespace-leader. Does anyone have any idea why these extra pages are appearing and how they can be removed? Anyone have experience with GTM causing issues with this? Thanks everyone!!!
Reporting & Analytics | | Kingalan1
Alan1 -
Switch to www from non www preference negatively hit # pages indexed
I have a client whose site did not use the www preference but rather the non www form of the url. We were having trouble seeing some high quality inlinks and I wondered if the redirect to the non www site from the links was making it hard for us to track. After some reading, it seemed we should be using the www version for better SEO anyway so I made a change on Monday but had a major hit to the number of pages being indexed by Thursday. Freaking me out mildly. What are people's thoughts? I think I should roll back the www change asap - or am I jumping the gun?
Reporting & Analytics | | BrigitteMN0 -
Google Analytics: how many visits from country Google domains?
Hello, I manage a site with visitors from many different countries. With Google Analytics, it is normal to see the number of visitors from each search engine. However, I would like to identify the number of visitors from each Google-search contry domain. How many visitors from Google.com? How many from Google.co.uk. And from Google.co.zm? And so on. Anybody knows if this is possible and if yes, how can it be done? Thank you in advance, Dario
Reporting & Analytics | | Darioz0