This is a part of our Question and Answer series. If you have any questions related to Blogging, SEO, and/or WordPress, feel free to drop your queries via our contact form. Selected questions will be answered via a blog post. You can also check out previous questions answered here at ShoutMeLoud:
- A Few Blogging Questions & My Answers
- How To Become A Professional Blogger
This is a question by Irfan Siddiqui from realtimetricks.com.
Today’s Question:
“I am facing issues with site indexing. I own a WordPress blog and despite having ‘noindex’ tags on all of my tags and categories pages, Google has indexed WordPress tags and categories pages anyway. How can I deindex these tags and category pages?”
Google Indexing Issue with WordPress Attachments, Tags & Categories
This is one of the common indexing issues where Google indexes a few pages from your blog despite you adding a noindex tag. This is normal, and as soon as Google recrawls your site, it will respect the noindex tag and will deindex these parts of your blog.
This may take some time, but here are a few things which you can do to speed up the process.
First, make sure your Robots.txt is not blocking access to the tags and categories pages.
If a few of your pages are already indexed in Google and you have added the noindex tag, Google may have stopped crawling of that part of your site and never see the noindex tag because Robots.txt is blocking access to crawlers.
Robots.txt exclusion is only useful for parts like Wp-admin and WP-content.
- Useful Tips To Increase Google Crawl Rate
- Use Robots.txt To Protect Your Blog From A Duplicate Content Issue
Moreover, it’s not a good idea to stop crawling pages like tags and categories using robots.txt as it will hamper the crawling of your deeper pages. The best way to stop indexing is by adding a noindex tag, which you can easily do by using WordPress SEO by Yoast.
Now in the case of Irfan’s blog, he has not blocked the tags and categories pages using Robots.txt, but before adding noindex tags, Google has already indexed those pages.
Here is a quick solution for anyone facing a similar problem.
First, you can check what pages of your site is indexed in Google by typing this search operator into Google:
- site:domain.com (replace domain.com with your domain)
Make sure you use the noindex option for tags and categories pages in WordPress SEO by Yoast or the Meta Robots plugin.
If you are using the Thesis theme, you can find this option under Site option > Document head > Robots meta tag > Noindex categories and tags.
- Login to your Google Webmaster Tool
- Go to Site Dashboard > Google Index > Remove URL’s
Now, usually our tags and categories pages are seen as a directory from a search engine’s point of view.
For example:
- domain.com/tag/SEO
- doamin.com/tag/WordPress
We can use the directory removal option to remove the complete directory from the noindex tag. Within 24-48 hours, all of your categories and tag pages will be out of Google’s index.
So you need to add www.domain.com/tag/ and www.domain.com/category/ and select “Temporarily hide directory”.
Hit “Submit Request” and now you’ve gotten rid of tags and categories pages from Google’s search index.
Related reading:
Getting rid of WordPress attachment ID links
This is another thing which not only creates duplicate content issues, but also adds low-quality content into Google Search. We can again fix this issue by using the Yoast SEO WordPress plugin.
Also, here is a practice that you should be following: When you upload an image, make sure your image link URL is either blank, or if you want to make it clickable, use the “Link to Image” option.
See this screenshot:
This will ensure no such links will be indexed in the future.
Here are configuration settings for plugins which I mentioned above:
WordPress SEO by Yoast setting:
WordPress SEO dashboard > Advanced > Permalinks > Put a tick on “Redirect attachment URLs to parent post URL.”
That’s it and it will redirect all such >attachment_id= links to its respective post. In a few days, Google will deindex all such links.
I hope this quick solution will help anybody facing a similar issue. Do let me know if you’ve ever faced a similar WordPress indexing issue like Irfan. Did this solution fix the problem? Let me know in the comments!
Sir, I’m not able to find particular remove Url screen as you provided in screenshot. There is no any remove option just like that.
@Akansha
I have updated the screenshots. Now, you would be able to find it. Let me know how it works out for you!
NO, Still not able to find remove complete directory option. Yes , I remove all outdated links but by going through https://www.google.com/webmasters/tools/removals?pli=1link.
Don’t know I’m doing right or not.
Hi Harsh,
Thanks for clarifying. Just had an issue in my site where the robots.txt was blocking the WordPress ‘tags’ and ‘categories’. I obviously followed some random tip from another source and paid the price. Anyway, now I am happy I understand something along the way.
Thanks again,
Pothi
Hello Harsh,
Thanks for the steps above it is really useful as i have some of my tags ranking for specific keywords.
Hello
I’m also facing the same issue in my domain & now i’ve disabled these through Yoast Plugin. Thanks you.
Hi Harsh ,
Thanks for this great article. I just discovered my site was messed up on google , due to the fact that i had too many tags and categories indexed , which caused duplicate content issue. Luckly , I’ve found this article and i’ve pulled down all tags and categories. Now i have to monitor my traffic in the next few days to see if there’s any effect.
Cheers.
Hey Kev,
Do let us know the feedback.
hello thanks harsh for the information, my website is showing results in google, but in webmaster it is temporary removal, i have already no index no follow, but still showing? what to do for permanent solution?
Hi Hursh, little confusion here, now my categories are indexed in google but If i add nofollow tag to the categories in yoast plugin will google remove all the categories from google index or I have to use webmaster tool account ??
I am having the same problem .I added noindex tags to my category and tags but still after crawling, the pages are not removed. I had also removed the directory /tag/ and /category/ and after successful removal also the tags are appearing. what is the solution?
Hello Harsh Ji,
I’m currently having a WordPress.com Blog (Free Plan) & desire to sell my Digital Stuff online without setting-up a website and relying wholly on the Blog, at first (have been short of funds). Thus, going through with this thought I have made changes to my Blog, under this after making the payment, the customer is redirected to my blog page haivng a link to download the digital content. So, I just want the payment gateway to be able to redirect the traffic to the Download page on my blog and not have it searchable via search engines, etc. & neither be it listed under the WordPress.com Posts list/Pages, etc. because if the download page is searchable by the Search Engines & listed under my Blog’s post, I wouldn’t earn anything because then the customer would directly download the content without making the payment.
Regards,
Joojo.
Hi Harsh,
Would you pls clarify : if you only need to deindex category & tag directories, to avoid duplication, why to submit sitemaps of category & tags?
Thanks for your time.
Hey harsh, I followed your steps to remove category pages from SERP but author archives is already indexed right now. I just set author archives to no-index so my question is do I have to remove http://domain.com/author directory from Google Webmaster Tools ?
@Brijesh
Yes…You need to remove Author directory from removal tool.
Hi Harsh,
How are you? I hope you are doing good, recently i have added my sitemap to google using yoast wordpress seo, but it shows error says robots blocks tag and category sitemap.
Pls can you answer me.
@Venkatesan
Not sure why you getting this. I checked your robots.txt file & it seems fine to me. Have you made changes after commenting?
Hi Harsh
Should I put a tick on “Redirect attachment URL’s to parent post URL” though I’m not affected by the attachment ID link on google search result?
Very informative article. Made a few changes so time will tell if my page ranks go up.
Much Obliged, Doc
Thanks for the tips. I have implemented every bit of the information in my Yoast SEO and I am sure it will clean up the unnecessary google crawls and improve search results. I am new to all this SEO game and happy that someone is guiding me in the right direction
Thanks Harsh for this great piece of advice. I accidentally indexed my categories and tags and saw some very strange results in search results.
I could see that the Google search was showing tags instead of the real post. Now, that I have figured out the issue and followed the steps given by you above to remove the tags and categories. I hope to see some clean indexing.
Thanks again,
Pankaj Chauhan
Hello Harsh!
Thank you for such a nice article!
Just 1 thing with which i am confused is that i have about 5-10 categories and hundreds of tags at my blog, so in Google Webmaster Tool, should i have to enter them individually (one by one) like:
domain.com/category/SEO
doamin.com/tag/WordPress
or will only submitting the main directory link will do the work like following:
domain.com/category/
doamin.com/tag/
Please help me with this. Thanks a lot in advance!
@Ubaid
Under URl removal tool, add domain.com/category/ and select option which says remove directory. Repeat the same step for Tags and that will take care of everything.
I did over a month ago, to the letter. Google confirmed the removal of each directory in my Search Console. Right now I’m looking at today’s log and watching Googlebot index category page after category page today, squandering my crawl budget. That is, not simply not yet deindexing past crawls, but rather currently indexing pages that your post suggests would be removed from crawl consideration as I type this.
Is there perhaps an alternative solution?
Hi, Harsh,
Thank you for your article.
I use this solution:
WordPress dashboard > SEO > Permalink Settings > Put a tick on “Redirect attachment URL’s to parent post URL.
but there are still google index “/?attachment_id=xxxx ” after 3 three days.
Why ? Please help
Jason
@Jason
It will take time for Google to remove those links from search index.
Thank you very much for this post! I have just removed a number of attachment_id pages from the search result. I am still very confused regarding of the categories. On seo moz I was adviced not set categories as noindexed as they will help google understand the structure of the site. I am realy very confused now. My site has property categories in the main menu and I realy do not know if I should now no indexed them:-(
You should not use the URL Removal tool in this way.
https://support.google.com/webmasters/answer/1269119?hl=en
Don’t use the URL tool to get rid of pages in these circumstances:
To clean up cruft, like old pages that 404. If you recently changed your site and now have some outdated URLs in the index, Google’s crawlers will see this as we recrawl your URLs, and those pages will naturally drop out of our search results. There’s no need to request an urgent removal.