Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Value of using spaces or no spaces on product category page varient keywords
Hello, all fellow Mozzers,
On-Page Optimization | | JamesDavison
I have taken over a project and this account, so can't change the username according to MOZ.🙃 We run an eCommerce website, and to me, some of the content is conflicting as some pages have more information content than what I would put in a commerce page, but this is how the boss wants it to work, personally, I would separate the content out.
The page I'm working on:
https://www.longstonetyres.co.uk/tyres/205-70-14.html
and this is an example of the rest of these types of pages, I will be tackling:
https://www.longstonetyres.co.uk/tyres/125-15.html I was tasked to improve SEO ranking, when using the MOZ page grader I had a score of 24 out of 27 83% SEO score and 3-page problems. 7th position in Google for the search term 205/70 R14 As it is a generic product listing page, It was pointless to add to the URL and the Internal links I can't reduce as these are links to products, so I went to reduce the
keyword stuffing and making the page content more natural, this improved the page to 25 out of 27, 87% SEO score and 2-page problems. Improvement to 3rd position in Google, but he wants to chase 1st place to be above his competitors, which is fair enough. It turns out that in the past, they have used this type of page to try and get a high ranking for several search terms, as it is a different variation on a tyre size terms are:
205/70 R14, 205/70R14, 205/70 R 14
205/70 X 14, 205/70X14, 205/70 X14
and so on for all the different ways you can search for this tyre size. He is also convinced Google will see these as different search terms, and while I agree to an extent, this causes Keyword Stuffing on the page, which in turn was harming the rankings. Each product listed on the page already has its own title 205/70 R14, 205/70 HR14 and so on, so my question is. What is the best practice for writing content on these types of pages to gain high rankings for several Keywords, and what value does writing the same keyword with spaces and no spaces have? Any help or advice is welcome, so I have a better understanding of how to approach this for this page and the rest of the site. Cheers Mal0 -
Category pages, should I noindex them?
Hi there, I have a question about my blog that I hope you guys can answer. Should I no index the category and tag pages of my blog? I understand they are considered as duplicate content, but what if I try to work the keyword of that category? What would you do? I am looking forward to reading your answers 🙂
On-Page Optimization | | lucywrites0 -
Can we change Title and/or Descriptions Dynamically Based on Search Query
If I recall we used to be able to change our title attributes tag dynamically based on the search query but not sure if it's possible now or if it makes sense to do so. Thoughts? Rosemary
On-Page Optimization | | RosemaryB1 -
No meta description on category page
Hi Moz is reporting no meta description on a wordpress category page like this one: http://www.dwliverpoolphotography.co.uk/category/uncategorized/ Can I add a meta description to a category? Best wishes. David.
On-Page Optimization | | WallerD0 -
Tags vs. Categories? What should I use?
I'm starting with a blog (self-hosted wordpress) and I'm thinking of the following content structure so that the readers are easily able to locate relevant content: Background: It's a blog which gives people relevant info about government jobs. To start with we will just be publishing information about these jobs but over a period of time also intend to post content that helps readers prepare for these jobs. In other words, right now it's just about detailed job notifications but in the coming months, we shall also post about preparation-related information. Typically, each of the job notifications can be bifurcated like: Jobs basis industry Banking Railways Clinical, etc. Jobs basis company ABC co. DEF co. XYZ co. etc. Jobs basis State / City City 1 City 2, etc. Jobs basis educational qualification Graduation Post-Graduation, etc. Now, I'm seriously confused how should I structure this data from the perspective of Categories & Tags such that it's reader as well as SEO-friendly. Do note that each of the government jobs post ideally falls in a couple of above mentioned categories. Thanks..
On-Page Optimization | | Shalin.TJ0 -
H2s & H3s for Category Navigation
Hi all. I am wondering how best to format a category navigation menu. Currently I don't think we're using H2s correctly on our website. Am I right to think that the top level category e.g. Games should be formatted as an H2 and the sub-categories underneath this should be formatted as H3s (to show a hierarchy)? Is there a limit on how many H2s and H3s you should use? Obviously only one H1 per page. Thanks in advance Paul
On-Page Optimization | | kevinliao0 -
Right way to block google robots from ppc landing pages
What is the right way to completely block seo robots from my adword landing pages? Robots.txt does not work really good for that, as far I know. Adding metatags noindex nofollow on the other side will block adwords robot as well. right? Thank you very much, Serge
On-Page Optimization | | Kotkov0 -
How do you block development servers with robots.txt?
When we create client websites the urls are client.oursite.com. Google is indexing theses sites and attaching to our domain. How can we stop it with robots.txt? I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy. Thanks, TR
On-Page Optimization | | DisMedia0