WordPress Robots.txt Tutorial: How to Create and Optimize for SEO

WordPress Robots.txt
  • Save

Whenever we talk about SEO of Wp blogs, WordPress robots.txt file plays a major role in search engine ranking.

It blocks search engine bots and helps index and crawl important parts of our blog. Though sometimes, a wrongly configured Robots.txt file can let your presence completely go away from search engines.

So, it is important that when you make changes in your robots.txt file, it should be well optimized and should not block access to important parts of your blog.

Ideal WordPress robots txt file
  • Save

There are many misunderstandings regarding indexing and non-indexing of content in Robots.txt and we will look into that aspect too.

SEO consists of hundreds of element and one of the essential parts of SEO is Robots.txt. This small text file standing at the root of your website can help in serious optimization of your website.

Most of the Webmasters tend to avoid editing Robots.txt file, but it’s not as hard as killing a snake. Anyone with basic knowledge can create and edit a Robots file, and if you are new to this, this post is perfect for your need.

If your website doesn’t have a Robots.txt file, you can learn how to do it here. If your blog/website has a Robots.txt file but is not optimized, you can follow this post and optimize your Robots.txt file.

What is WordPress Robots.txt and why should we use it

Let me start with the basics. All search engines have bots to crawl a site. Crawling and indexing are two different terms, and if you wish to go deep into it, you can read: Google Crawling and indexing.

When a search engine bot (Google bot, Bing bot, 3rd party search engine crawlers) comes to your site following a link or following a sitemap link submitted in webmaster dashboard, they follow all the links on your blog to crawl and index your site.

Now, these two files – Sitemap.xml and Robots.txt – reside at the root of your domain. As I mentioned, bots follow Robots.txt rules to determine the crawling of your website. Here is the usage of robots.txt file:

When search engine bots come on your blog, they have limited resources to crawl your site. If they can’t crawl all the pages on your website with allocated resources, they will stop crawling, which will hamper your indexing.

Now, at the same time, there are many parts of your website, which you don’t want search engine bots to crawl. For example, your WP-admin folder, your admin dashboard or other pages, which are not useful for search engines. Using Robots.txt, you are directing search engine crawlers (bots), to not crawl to such areas of your website. This will not only speed up crawling of your blog but will also help in deep crawling of your inner pages.

The biggest misconception about Robots.txt file is that people use it for Noindexing.

Remember, Robots.txt file is not for Do Index or Noindex. It is to direct search engine bots to stop crawling certain parts of your blog. For example, if you look at ShoutMeLoud Robots.txt file (WordPress platform), you will clearly understand what part of my blog I don’t want search engine bots to crawl.

The Robots.txt file helps search engine robots and directs which part to crawl to and which part to avoid. When a search bot or spider of the search engine comes to your site and wants to index your site, they follow the Robots.txt file first. The search bot or spider follows the file directions for indexing or not indexing pages of your website.

If you use WordPress, you will find Robots.txt file in the root of your WordPress installation.

For static websites, if you or your developers have created one, you will find it in your root folder. If you can’t, simply create a new notepad file and name it Robots.txt and upload it into the root directory of your domain using the FTP.

Here is an example of Robots.txt file and you can see the content and it’s the location at the root of the domain.

https://www.shoutmeloud.com/robots.txt

How to generate a robots.txt file?

As I mentioned earlier, Robots.txt is a general text file. So, if you don’t have this file on your website, open any text editor as you like (Notepad, for example) and create a Robots.txt file made with one or more records. Every record bears important information for the search engine. Example:

User-agent: googlebot

Disallow: /cgi-bin

If these lines are written in the Robots.txt file, it means it allows the Google bot to index every page of your site. But cgi-bin  folder of root directory doesn’t allow for indexing. That means Google bot won’t index cgi-bin  folder.

By using Disallow option, you can restrict any search bot or spider from indexing a page or folder. There are many sites that use no index in the archive folder or page for not making duplicate content.

Where Can You Get the names of Search bots?

You can get it in your website’s log, but if you want lots of visitors from the search engine, you should allow every search bot. That means every search bot will index your site. You can write User-agent: *  for allow every search bot. For example:

User-agent: *

Disallow: /cgi-bin

That is why every search bot will index your website.

Dont’s of Robots.txt file

1. Don’t use comments in Robots.txt file.

2. Don’t keep the space at the beginning of any line and don’t make ordinary space in the file. Example:

Bad Practice:

   User-agent: *

Dis allow: /support

Good Practice:

User-agent: *

Disallow: /support

3. Don’t change rules of command.

Bad Practice:

Disallow: /support

User-agent: *

Good Practice:

User-agent: *

Disallow: /support

4. If you do not want to index more than one directory or page, don’t write along with these names:

Bad Practice:

User-agent: *

Disallow: /support /cgi-bin /images/

Good Practice:

User-agent: *

Disallow: /support

Disallow: /cgi-bin

Disallow: /images

5. Use capital and small letters properly. For example, if you want to index “Download” directory but write “download” on Robots.txt file, it mistakes it for a search bot.

6. If you want index all pages and directory of your site, write:

User-agent: *

Disallow:

7. But if you want no index for all page and directory of you site write:

User-agent: *

Disallow: /

After editing the Robots.txt file, upload it via any FTP software on Root or Home Directory of your site.

WordPress Robots.txt Guide:

You can either edit your WordPress Robots.txt file by logging into your FTP account of the server or you can use plugins like Robots meta to edit Rrobots.txt file from WordPress dashboard. There are a few things which you should add in your Robots.txt file along with your sitemap URL. Adding sitemap URL helps search engine bots to find your sitemap file and results in faster indexing of pages.

Here is a sample Robots.txt file for any domain. In the sitemap, replace the Sitemap URL with your blog URL:

sitemap: https://www.shoutmeloud.com/sitemap.xml

User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /archives/
disallow: /*?*
Disallow: *?replytocom
Disallow: /comments/feed/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Block bad SEO bots (Complete list)

There are many SEO tools like Ahrefs, SEMRush, Majestic and many others which keep crawling your website for SEO secrets. These strategies are used by your competitors for their benefits and don’t add value to you. Moreover, these SEO crawlers also add load to your server and increases your server cost.

Unless you are not using one of these SEO tools, you are better off blocking them from cralwing your site. Here is what I use on my robots.txt to block some of the most popular SEO agents:

User-agent: MJ12bot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: dotbot
Disallow:/
User-agent: AhrefsBot
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: Xenu’s
Disallow: /
User-agent: Xenu’s Link Sleuth 1.1c
Disallow: /
User-agent: rogerbot
Disallow: /

# Block NextGenSearchBot
User-agent: NextGenSearchBot
Disallow: /
# Block ia-archiver from crawling site
User-agent: ia_archiver
Disallow: /
# Block archive.org_bot from crawling site
User-agent: archive.org_bot
Disallow: /
# Block Archive.org Bot from crawling site
User-agent: Archive.org Bot
Disallow: /

# Block LinkWalker from crawling site
User-agent: LinkWalker
Disallow: /

# Block GigaBlast Spider from crawling site
User-agent: GigaBlast Spider
Disallow: /

# Block ia_archiver-web.archive.org_bot from crawling site
User-agent: ia_archiver-web.archive.org
Disallow: /

# Block PicScout Crawler from crawling site
User-agent: PicScout
Disallow: /

# Block BLEXBot Crawler from crawling site
User-agent: BLEXBot Crawler
Disallow: /

# Block TinEye from crawling site
User-agent: TinEye
Disallow: /

# Block SEOkicks
User-agent: SEOkicks-Robot
Disallow: /

# Block BlexBot
User-agent: BLEXBot
Disallow: /

# Block SISTRIX
User-agent: SISTRIX Crawler
Disallow: /

# Block Uptime robot
User-agent: UptimeRobot/2.0
Disallow: /

# Block Ezooms Robot
User-agent: Ezooms Robot
Disallow: /

# Block netEstate NE Crawler (+http://www.website-datenbank.de/)
User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/)
Disallow: /

# Block WiseGuys Robot
User-agent: WiseGuys Robot
Disallow: /

# Block Turnitin Robot
User-agent: Turnitin Robot
Disallow: /

# Block Heritrix
User-agent: Heritrix
Disallow: /

# Block pricepi
User-agent: pimonster
Disallow: /
User-agent: Pimonster
Disallow: /
User-agent: Pi-Monster
Disallow: /

# Block Eniro
User-agent: ECCP/1.0 ([email protected])
Disallow: /

# Block Psbot
User-agent: Psbot
Disallow: /

# Block Youdao
User-agent: YoudaoBot
Disallow: /

# BLEXBot
User-agent: BLEXBot
Disallow: /

# Block NaverBot
User-agent: NaverBot
User-agent: Yeti
Disallow: /

# Block ZBot
User-agent: ZBot
Disallow: /

# Block Vagabondo
User-agent: Vagabondo
Disallow: /

# Block LinkWalker
User-agent: LinkWalker
Disallow: /

# Block SimplePie
User-agent: SimplePie
Disallow: /

# Block Wget
User-agent: Wget
Disallow: /

# Block Pixray-Seeker
User-agent: Pixray-Seeker
Disallow: /

# Block BoardReader
User-agent: BoardReader
Disallow: /

# Block Quantify
User-agent: Quantify
Disallow: /

# Block Plukkie
User-agent: Plukkie
Disallow: /

# Block Cuam
User-agent: Cuam
Disallow: /

# https://megaindex.com/crawler
User-agent: MegaIndex.ru
Disallow: /

User-agent: megaindex.com
Disallow: /

User-agent: +http://megaindex.com/crawler
Disallow: /

User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: megaIndex.ru
Disallow: /

Making sure no content is affected by new Robots.txt file

So now you have made some changes to your Robots.txt file, and it’s time to check if any of your content is impacted due to the updation in the robots.txt file.

You can use Google search console ‘Fetch as Google tool’ to see if whether or not your content can be accessed by Robots.txt file.

These steps are simple.

Login to Google search console, select your site, go to diagnostic and Fetch as Google.

Add your site posts and check if there is any issue accessing your post.

  • Save

You can also check for the crawl errors caused due to Robots.txt file under Crawl error section of search console.

Under Crawl > Crawl Error, select Restricted by Robots.txt and you will see what all links have been denied by the Robots.txt file.

Here is an example of Robots.txt Crawl Error for ShoutMeLoud:

  • Save

You can clearly see that Replytocom links have been rejected by Robots.txt and so have other links which should not be a part of Google. FYI, Robots.txt file is an essential element of SEO, and you can avoid many post duplication issues by updating your Robots.txt file.

Do you use WordPress Robots.txt to optimize your site? Do you wish to add more insight to your Robots.txt file? Let us know using the comment section below. Don’t forget to subscribe to our e-mail newsletter to keep receiving more SEO tips.

Here are a few other hand-picked guides for you to read next:

Subscribe on YouTube

  • Save
Authored By
A Blogger, Author and a speaker! Harsh Agrawal is recognized as a leader in digital marketing and FinTech space. Fountainhead of ShoutMeLoud, and a Speaker at ASW, Hero Mindmine, Inorbit, IBM, India blockchain summit. Also, an award-winning blogger.

77 thoughts on “WordPress Robots.txt Tutorial: How to Create and Optimize for SEO”

  1. Thank you so much for this helpful post. Now i will try to optimize my blog robots.txt file.
    I was using this one :
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

  2. Augustine Nicrotech

    Hey Harsh,i have the same problem with Anna,I can’t fetch my site as google bot.How can i show you screen shots??

  3. Hi, i have added this line (for not indexing search query)

    Disallow: /search/
    Disallow: /search?
    Disallow: /search?q=*
    Disallow: /*?updated-max=*

    to my robots.txt file
    kindly check this URL: http://techviha.com/robots.txt and tell me is it okay? or need any changes?

    waiting for your replay. 🙂

  4. One more Superb Article after reading this post i made change in my robots.txt file as you mention here, it really help me in future for search engine friendly website.
    thank you so much for sharing your knowledge. 🙂

  5. Hi, I have under one hosting more then one web site, is there necessary to Disallow all other blogs under public_html except the primary one?

    Thank you.

  6. Superb article,
    even i had faced many problem editing robot.txt file for my WordPress blog initially for better search engine optimization but when i got how to edit it exactly. Now i know the importance of Robot.txt file for Search Engine traffic.

  7. I really found it helpful ! I moved my site to GoDaddy but forgot to modify the robots.txt file and was searching on Google for the best robots file ! Well, Shoutmeloud can be the best blog so why not use the same robots.txt file for myself 😉
    Thank You harsh bro 😉

  8. Hai harsh

    Just went trough Yoast post and they are saying we should no longer disallow wp-content and wp-includes as Google bot started fetching CSS and JS files too.
    And he said, there is no point of adding sitemap to robots file and by default wp-admin is excluded from wp 4.0
    can you please research on this and let us know more details
    Thank you

    1. yes that right ,google now can read file like css and js ,you don’t need Disallow file like this …

    2. Yes, that is true! This article is missleading as it suggest to disallow wp-includes! This alone can cause problems with indexing and can have impact on site ranking. I just recieved email from google noting that googlebot is not able to reach all parts of my site – that is why I searched a little and foud that i have wp-includes disallowed in robots.txt – by allowing it again I fixed my issue.

      1. Hi !
        di we have to remove just /wp-includes/ or wp-admin/ too ??
        thank you because I reveived the same message from google

          1. thank you so much for your response ! and how can I know if it’s work because on my robots.txt tester everything seems ok !

          1. Thank you !
            Yesterday I removed wp-include from robots.txt but I still see it on my files !!
            I removed it from my admin wordpress –> SEO yoast plugin !

  9. Enlightening article on robots.txt optimization indeed. I learned a thing or two. Thanks for the effort.

    1. @Fahim
      Yes, it’s available. Here is how you can access it from SEO by Yoast setting:
      SEO > Advanced > File editor

  10. Thanx alot harsh. Your articles have really helped me as a newbie blogger. I am confident that updating my blog as per the information given by you in shoutmeloud will help me in future also.

  11. Thanks for this guys – very very helpful! Someone should develop a plugin that magically removes all duplication, rather than diving into the back end of a site and manually editing robots.txt.

    1. Thanks, I have the alerts on webmaster notifications, because google can not access my css and js files, should I do not want to block wp-includes on the robots.txt.

  12. Hi Harsh,
    I add my site before 15 days into webmaster tool. Now it is not indexed. I recognized the problem and added above given robots.txt file into my server. Now some errors are showing like

    1. URL restricted by robots.txt

    2. We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

    How to resolve it?

  13. hello these may be off topic but i need help on Blocked Resources
    how to remove blocked resources.
    last time when i check there was 400 links blocked because of “disallow wp-include” line, so i remove disallow wp-include from robot file. allowing it.
    but now after 2 weeks blocked links are 670, i don’t understand what to do?
    i check all blocked links in render and fetch option, pages layout are not affected showing perfectly but not showing ads banner and other minor code are banned.
    so i’m confused now, does increase link in Blocked Resources affect websites SEO or there is no problem if links increase.
    please help on these.

  14. And also please explain, why have you disallowed index.php file, it is the main landing page and it should be indexed by search engines. I know you better know than me. Please explain why you have disallowed it. And what will be the effect on search engines’ crawling to index page.

  15. Thanks for the Great Post bro. Made my Robots.txt for one of my Viral News Site using your Guide.

    Hoping for Positive Results =)

  16. Thanks post is good. But now there is no need to block wp directories such as wp-includes, wp-content. Also exclude Disallow: /wp-* code from the robots.txt because it blocks these directories.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top
61 Shares
[i]
[i]
[i]
[i]
Share via
Copy link