How To Optimize WordPress Robots.txt File For Search Engine Bots

Whenever we talk about SEO of Wp blogs, WordPress robots.txt file plays a major role in search engine ranking. It helps to block search engine bots to index and crawl important part of our blog. Though, sometimes a wrong configured Robots.txt file can let your presence completely go away from search engines. So, it’s important when you make changes in your robots.txt file, it should be well optimized and should not block access to important part of your blog.


Ideal WordPress robots txt file

There are many misunderstanding regarding indexing and non-indexing of content in Robots.txt and we will look into that aspect also in this article.

SEO consists of hundreds of element and one of the essential part of SEO are Robots.txt. This small text file standing at the root of your Website can help in serious optimization of your Website. Most of Webmasters tend to avoid editing Robots.txt file, but it’s not as hard as killing a snake. Anyone with basic knowledge can create and edit his Robots file, and if you are new to this, this post is perfect for your need.

If your website hasn’t got a Robots.txt file, learn here how to do it. If your blog or website does have a Robots.txt file but is not optimized, then follow this post and optimize your Robots.txt file.

What is WordPress Robots.txt and why should we use it?

Robots.txt file helps search engine robots to direct which part to crawl and which part to avoid. When Search bot or spider of Search Engine comes to your site and wants to index your site, they follow Robots.txt file first. Search bot or spider follows this files direction for index or no index any page of your website.

If you are using WordPress, you will find Robots.txt file in the root of your WordPress installation. For static websites, if you have created one or you developer has created one, you will find it under your root folder. If you can’t simply create a new notepad file and name it as Robots.txt and upload it into Root directory of your domain using FTP. Here is ShoutMeLoud Robots txt file and you can see the content and it’s the location at the root of the domain.

How to make  robots.txt file?

As I mentioned earlier, Robots.txt is a general text file. So, if you don’t have this file on your website, open any text editor as you like ( as the example: Notepad) and make Robots.txt file made with one or more records. Every record bears important information for search engine. Example:

User-agent: googlebot

Disallow: /cgi-bin

If these lines write on Robots.txt file it’s allowed Google bot for index every page of your site. But cgi-bin  folder of root directory doesn’t allow for indexing. That means Google bot won’t index cgi-bin  folder.

By using Disallow option, you can restrict any search bot or spider for indexing any page or folder. There are many sites who use no index in Archive folder or page for not making duplicate content.

Where Can You Get names of Search bot?

You can get it in your website’s log, but if you want lots of visitors from the Search engine you should allow every search bot. That means every search bot will index your site. You can write User-agent: *  for allow every search bot. Example:

User-agent: *

Disallow: /cgi-bin

That’s why every search bot index your Website.

What You Shouldn’t do?

1. Don’t use comments in Robots.txt file.

2. Don’t keep the space at the beginning of any line and don’t make ordinary space in the file. Example:

Bad Practice:

   User-agent: *

Dis allow: /support

Good Practice:

User-agent: *

Disallow: /support

3. Don’t change rules of command.

Bad Practice:

Disallow: /support

User-agent: *

Good Practice:

User-agent: *

Disallow: /support

4. If you want no index, more than one directory or page don’t write along with these names:

Bad Practice:

User-agent: *

Disallow: /support /cgi-bin /images/

Good Practice:

User-agent: *

Disallow: /support

Disallow: /cgi-bin

Disallow: /images

5. Use capital and small letter properly. As the example, you want no index “Download” directory but write “download” on Robots.txt file. It makes miss understand for search bot.

6. If you want index all page and directory of your site write:

User-agent: *


7. But if you want no index for all page and directory of you site write:

User-agent: *

Disallow: /

After editing Robots.txt file upload via any FTP software on Root or Home Directory of your site.

Robots.Txt for WordPress:

You can either edit your WordPress Robots.txt file by logging into your FTP account of the server or you can use plugin like Robots meta to edit robots.txt file from WordPress dashboard. There are few things, which you should add in your robots.txt file along with your sitemap URL. Adding sitemap URL helps search engine bots to find your sitemap file and thus faster indexing of pages.

Here is a sample Robots.txt file for any domain. In sitemap, replace the Sitemap URL with your blog URL:


User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /archives/
disallow: /*?*
Disallow: *?replytocom
Disallow: /wp-*
Disallow: /comments/feed/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

How to make sure no content is affected by new Robots.txt file?

So now you have made some changes into your Robots.txt file, and it’s time to check if any of your content is impacted by updating robots.txt file. You can use Google Webmaster tool ‘Fetch as bot tool’ to see if your content can be accessed by Robots.txt file or not. This step is simple, login to Google Webmaster tool and go to diagnostic and Fetch as Google bot. Add your site posts and check if there is any issue accessing your post.

Fetch as Google Bot

Fetch as Google Bot

You can also check for the crawl errors caused due to Robots.txt file under Crawl error section of GWT.  Under diagnostic >Crawl error select Restricted by Robots.txt and you will see what all links has been denied by Robots.txt file.

Here is an example of Robots.txt crawl Error for ShoutMeLoud:

Google Crawl Error

Google Crawl Error

You can clearly see that Replytocom links have been rejected by Robots.txt and so any other link which should not be a part of Google. FYI, Robots.txt file is an essential element of SEO, and you can avoid many post duplication issues by updating your Robots.txt file.

Are you using WordPress Robots.txt to optimize your site? Do you wish to add more insight to Robots.txt file? Let us know via comments. Don’t forget to subscribe to our e-mail newsletter to keep receiving more SEO tips.

Subscribe on Youtube

Article By
I'm a student of almost every subject and try to be an excellent blogger.


COMMENTs ( 54 )

  1. Felix B says

    Thanks for the Great Post bro. Made my Robots.txt for one of my Viral News Site using your Guide.

    Hoping for Positive Results =)

  2. says

    And also please explain, why have you disallowed index.php file, it is the main landing page and it should be indexed by search engines. I know you better know than me. Please explain why you have disallowed it. And what will be the effect on search engines’ crawling to index page.

  3. Sagar Patil says

    hello these may be off topic but i need help on Blocked Resources
    how to remove blocked resources.
    last time when i check there was 400 links blocked because of “disallow wp-include” line, so i remove disallow wp-include from robot file. allowing it.
    but now after 2 weeks blocked links are 670, i don’t understand what to do?
    i check all blocked links in render and fetch option, pages layout are not affected showing perfectly but not showing ads banner and other minor code are banned.
    so i’m confused now, does increase link in Blocked Resources affect websites SEO or there is no problem if links increase.
    please help on these.

  4. says

    Hi Harsh,
    I add my site before 15 days into webmaster tool. Now it is not indexed. I recognized the problem and added above given robots.txt file into my server. Now some errors are showing like

    1. URL restricted by robots.txt

    2. We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

    How to resolve it?

    • says

      Thanks, I have the alerts on webmaster notifications, because google can not access my css and js files, should I do not want to block wp-includes on the robots.txt.

  5. says

    Thanks for this guys – very very helpful! Someone should develop a plugin that magically removes all duplication, rather than diving into the back end of a site and manually editing robots.txt.

  6. says

    Thanx alot harsh. Your articles have really helped me as a newbie blogger. I am confident that updating my blog as per the information given by you in shoutmeloud will help me in future also.

  7. says

    Hai harsh

    Just went trough Yoast post and they are saying we should no longer disallow wp-content and wp-includes as Google bot started fetching CSS and JS files too.
    And he said, there is no point of adding sitemap to robots file and by default wp-admin is excluded from wp 4.0
    can you please research on this and let us know more details
    Thank you

  8. says

    I really found it helpful ! I moved my site to GoDaddy but forgot to modify the robots.txt file and was searching on Google for the best robots file ! Well, Shoutmeloud can be the best blog so why not use the same robots.txt file for myself ;-)
    Thank You harsh bro ;-)

  9. adeem says

    Superb article,
    even i had faced many problem editing robot.txt file for my WordPress blog initially for better search engine optimization but when i got how to edit it exactly. Now i know the importance of Robot.txt file for Search Engine traffic.

  10. says

    Hi, I have under one hosting more then one web site, is there necessary to Disallow all other blogs under public_html except the primary one?

    Thank you.

  11. says

    One more Superb Article after reading this post i made change in my robots.txt file as you mention here, it really help me in future for search engine friendly website.
    thank you so much for sharing your knowledge. :)

  12. Augustine Nicrotech says

    Hey Harsh,i have the same problem with Anna,I can’t fetch my site as google bot.How can i show you screen shots??

  13. Siraj.M says

    Thank you so much for this helpful post. Now i will try to optimize my blog robots.txt file.
    I was using this one :
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

  14. JR says

    Thank you. This is what I use for my robots.txt file. Hope this helps

    User-agent: *
    Disallow: /wp-login.php
    Disallow: /wp-admin/
    Disallow: /wp-content/plugins/ # prevents backlinks in plugin folders
    Disallow: /wp-includes/
    Disallow: /wp-content/themes/
    Disallow: /search/ # prevents search queries being indexed
    Disallow: /*?s
    Allow: /wp-content/uploads/

  15. altamash says

    I had little knowledge about Robot.txt and I never had touched this area, As already somewhere being told before using it you have to be very careful, Moreover I neverknow about this option.. due to examples and all the examples helped a lot..
    Thanks ;)

  16. Mahendra says

    Hello Harsh
    Adsense says me that allow crawler to index for better indexing and putting targeted adds. So I have to edit my robots.text. This post really helpful for me . I just want to ask that is it better to use WP robots.text plugin instead of editing it by myself ?

  17. Mukul Bansal says

    Made a new Robot.txt file for one of my blog and updated the robot.txt file for my other blog. Hoping to see good results in google search results

  18. Rosie Cottis says

    This was a useful post, thank you. I have a standard robots.txt that I have been using for about 5 years. I knew it was way out of date but I didn’t know exactly what to change. This has given me a lot of ideas.

  19. james says

    my domain age 3 years. recently one month back i started seo for this project with in one month result good all search engines but i upload robots.txt file my all keywords dissapear i dont know why robots.txt file some time working very bad

  20. Peace says

    Harsh I am in a thick mess up right now. I am not a google adsense user so my robots.txt file looks like this

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/
    Disallow: /comments/feed/
    Disallow: /trackback/
    Disallow: /index.php
    Disallow: /xmlrpc.php
    Disallow: *?wptheme
    Disallow: ?comments=*
    Disallow: /search?
    Disallow: /?p=*
    Disallow: /ko/
    Disallow: /bg/
    Disallow: /page/
    Disallow: /archives/
    Disallow: /trackback/
    Disallow: /tag/
    Disallow: /category/
    Allow: /wp-content/uploads
    Allow: /

    User-agent: Mediapartners-Google*
    Disallow: /

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Adsbot-Google
    Disallow: /

    User-agent: Googlebot-Mobile
    Allow: /

    I have disallowed mediapartners and adbot because I am not using adsense. At the same time I have a main question that by allowing only:
    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/
    we are authorizing Google image bot but what about bing and yahoo images. Is my this line correct:
    Allow: /wp-content/uploads
    and will help images to be indexed in other search engines. Also I have mentioned tag and category separately as I had a large number of tags, categories and pages (unuseful) indexed in google and my aim is to recover from Panda effect.
    Any advice would be helpful. Thanks

  21. igor Griffiths says

    Thanks for the clear advice about how to write a robots.txt file correctly.

    My mentor has asked me to research my competition by looking at their robots file but before doing that I checked mine which was a good idea as mine had my sitemap on a non existent paid service?

    Now going to implement your advice and get on with my research.

    igor Griffiths

  22. Simone G says

    Thanks to teaching us bad and good practice of robots.txt file . I was know about this before and using but I was not clear about the mistakes

  23. Avinash says

    Hi.. Abby Stone.. thanks a lot for this info..I guess blocking using robots.txt is the easier way to block content, I am recently seen that I have a lot of 404 in GWMT, with the /href= at the end of the URL after a theme change, can this be blocked using Disallow /href= in the robots.txt without the original url being effected.

  24. shenoyjoseph says

    i just now checked robots.txt for my blog and done some modifications to optimize my blog without any errors. :)

  25. DJ ARIF says

    Nice to see your article about Robots.txt file…. Hopefully this post will be helpful for all who wants the better result from search engines… :mrgreen:

  26. fazal mayar says

    You explained everything well Abby, i need to work on my robot.txt file as it needs improvements, thanks.

  27. Eric Murphy says

    nice post on robot.txt. this file is very important while doing SEO of any site. Thanks for sharing such a nice article.

  28. Sumanth Kumar says

    This will be very useful for the newbie bloggers like me,..hav to edit my robots.txt now..thanks for sharing the great tips…

  29. Hammad says

    Thanks for these excellent tips abby. I ll definitely check perform the required changes for my blog.

  30. Amol Wagh says

    Wow, Thanks for nailing down things separately about robots.txt

    I got some standard format & used to implement same on most of my blogs. I though I understood it completely.

    But these guidelines would improve it a lot !

    • says

      i’ve issue in my webmaster panel, Google Can’t access your site robot.txt , when i tried Fetch as Google , it returns ” SUCCESS ” ,

      then i checked the CRAWL ERROR , it still shows ” Google Can’t access …. ” ,

      What do i have to do ? Please Harsh do reply …

      • says

        What kind of Links Google is not able to access? Is it main URL or part of your site? Can you show some screenshots..Will be able to understand the issue better..