WordPress Robots.txt Tutorial: How to Create and Optimize for SEO

• Save

Whenever we talk about SEO of Wp blogs, WordPress robots.txt file plays a major role in search engine ranking.

It blocks search engine bots and helps index and crawl important parts of our blog. Though sometimes, a wrongly configured Robots.txt file can let your presence completely go away from search engines.

So, it is important that when you make changes in your robots.txt file, it should be well optimized and should not block access to important parts of your blog.

• Save

There are many misunderstandings regarding indexing and non-indexing of content in Robots.txt and we will look into that aspect too.

SEO consists of hundreds of element and one of the essential parts of SEO is Robots.txt. This small text file standing at the root of your website can help in serious optimization of your website.

Most of the Webmasters tend to avoid editing Robots.txt file, but it’s not as hard as killing a snake. Anyone with basic knowledge can create and edit a Robots file, and if you are new to this, this post is perfect for your need.

If your website doesn’t have a Robots.txt file, you can learn how to do it here. If your blog/website has a Robots.txt file but is not optimized, you can follow this post and optimize your Robots.txt file.

What is WordPress Robots.txt and why should we use it

Let me start with the basics. All search engines have bots to crawl a site. Crawling and indexing are two different terms, and if you wish to go deep into it, you can read: Google Crawling and indexing.

When a search engine bot (Google bot, Bing bot, 3rd party search engine crawlers) comes to your site following a link or following a sitemap link submitted in webmaster dashboard, they follow all the links on your blog to crawl and index your site.

Now, these two files – Sitemap.xml and Robots.txt – reside at the root of your domain. As I mentioned, bots follow Robots.txt rules to determine the crawling of your website. Here is the usage of robots.txt file:

When search engine bots come on your blog, they have limited resources to crawl your site. If they can’t crawl all the pages on your website with allocated resources, they will stop crawling, which will hamper your indexing.

Now, at the same time, there are many parts of your website, which you don’t want search engine bots to crawl. For example, your WP-admin folder, your admin dashboard or other pages, which are not useful for search engines. Using Robots.txt, you are directing search engine crawlers (bots), to not crawl to such areas of your website. This will not only speed up crawling of your blog but will also help in deep crawling of your inner pages.

The biggest misconception about Robots.txt file is that people use it for Noindexing.

Remember, Robots.txt file is not for Do Index or Noindex. It is to direct search engine bots to stop crawling certain parts of your blog. For example, if you look at ShoutMeLoud Robots.txt file (WordPress platform), you will clearly understand what part of my blog I don’t want search engine bots to crawl.

The Robots.txt file helps search engine robots and directs which part to crawl to and which part to avoid. When a search bot or spider of the search engine comes to your site and wants to index your site, they follow the Robots.txt file first. The search bot or spider follows the file directions for indexing or not indexing pages of your website.

If you use WordPress, you will find Robots.txt file in the root of your WordPress installation.

For static websites, if you or your developers have created one, you will find it in your root folder. If you can’t, simply create a new notepad file and name it Robots.txt and upload it into the root directory of your domain using the FTP.

Here is an example of Robots.txt file and you can see the content and it’s the location at the root of the domain.

How to generate a robots.txt file?

As I mentioned earlier, Robots.txt is a general text file. So, if you don’t have this file on your website, open any text editor as you like (Notepad, for example) and create a Robots.txt file made with one or more records. Every record bears important information for the search engine. Example:

User-agent: googlebot

Disallow: /cgi-bin

If these lines are written in the Robots.txt file, it means it allows the Google bot to index every page of your site. But cgi-bin  folder of root directory doesn’t allow for indexing. That means Google bot won’t index cgi-bin  folder.

By using Disallow option, you can restrict any search bot or spider from indexing a page or folder. There are many sites that use no index in the archive folder or page for not making duplicate content.

Where Can You Get the names of Search bots?

You can get it in your website’s log, but if you want lots of visitors from the search engine, you should allow every search bot. That means every search bot will index your site. You can write User-agent: *  for allow every search bot. For example:

User-agent: *

Disallow: /cgi-bin

That is why every search bot will index your website.

Dont’s of Robots.txt file

1. Don’t use comments in Robots.txt file.

2. Don’t keep the space at the beginning of any line and don’t make ordinary space in the file. Example:

Bad Practice:

   User-agent: *

Dis allow: /support

Good Practice:

User-agent: *

Disallow: /support

3. Don’t change rules of command.

Bad Practice:

Disallow: /support

User-agent: *

Good Practice:

User-agent: *

Disallow: /support

4. If you do not want to index more than one directory or page, don’t write along with these names:

Bad Practice:

User-agent: *

Disallow: /support /cgi-bin /images/

Good Practice:

User-agent: *

Disallow: /support

Disallow: /cgi-bin

Disallow: /images

5. Use capital and small letters properly. For example, if you want to index “Download” directory but write “download” on Robots.txt file, it mistakes it for a search bot.

6. If you want index all pages and directory of your site, write:

User-agent: *

Disallow:

7. But if you want no index for all page and directory of you site write:

User-agent: *

Disallow: /

After editing the Robots.txt file, upload it via any FTP software on Root or Home Directory of your site.

WordPress Robots.txt Guide:

You can either edit your WordPress Robots.txt file by logging into your FTP account of the server or you can use plugins like Robots meta to edit Rrobots.txt file from WordPress dashboard. There are a few things which you should add in your Robots.txt file along with your sitemap URL. Adding sitemap URL helps search engine bots to find your sitemap file and results in faster indexing of pages.

Here is a sample Robots.txt file for any domain. In the sitemap, replace the Sitemap URL with your blog URL:

sitemap: https://www.shoutmeloud.com/sitemap.xml

User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /archives/
disallow: /*?*
Disallow: *?replytocom
Disallow: /comments/feed/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Block bad SEO bots (Complete list)

There are many SEO tools like Ahrefs, SEMRush, Majestic and many others which keep crawling your website for SEO secrets. These strategies are used by your competitors for their benefits and don’t add value to you. Moreover, these SEO crawlers also add load to your server and increases your server cost.

Unless you are not using one of these SEO tools, you are better off blocking them from cralwing your site. Here is what I use on my robots.txt to block some of the most popular SEO agents:

User-agent: MJ12bot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: dotbot
Disallow:/
User-agent: AhrefsBot
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: Xenu’s
Disallow: /
User-agent: Xenu’s Link Sleuth 1.1c
Disallow: /
User-agent: rogerbot
Disallow: /

# Block NextGenSearchBot
User-agent: NextGenSearchBot
Disallow: /
# Block ia-archiver from crawling site
User-agent: ia_archiver
Disallow: /
# Block archive.org_bot from crawling site
User-agent: archive.org_bot
Disallow: /
# Block Archive.org Bot from crawling site
User-agent: Archive.org Bot
Disallow: /

# Block LinkWalker from crawling site
User-agent: LinkWalker
Disallow: /

# Block GigaBlast Spider from crawling site
User-agent: GigaBlast Spider
Disallow: /

# Block ia_archiver-web.archive.org_bot from crawling site
User-agent: ia_archiver-web.archive.org
Disallow: /

# Block PicScout Crawler from crawling site
User-agent: PicScout
Disallow: /

# Block BLEXBot Crawler from crawling site
User-agent: BLEXBot Crawler
Disallow: /

# Block TinEye from crawling site
User-agent: TinEye
Disallow: /

# Block SEOkicks
User-agent: SEOkicks-Robot
Disallow: /

# Block BlexBot
User-agent: BLEXBot
Disallow: /

# Block SISTRIX
User-agent: SISTRIX Crawler
Disallow: /

# Block Uptime robot
User-agent: UptimeRobot/2.0
Disallow: /

# Block Ezooms Robot
User-agent: Ezooms Robot
Disallow: /

# Block netEstate NE Crawler (+http://www.website-datenbank.de/)
User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/)
Disallow: /

# Block WiseGuys Robot
User-agent: WiseGuys Robot
Disallow: /

# Block Turnitin Robot
User-agent: Turnitin Robot
Disallow: /

# Block Heritrix
User-agent: Heritrix
Disallow: /

# Block pricepi
User-agent: pimonster
Disallow: /
User-agent: Pimonster
Disallow: /
User-agent: Pi-Monster
Disallow: /

# Block Eniro
User-agent: ECCP/1.0 ([email protected])
Disallow: /

# Block Psbot
User-agent: Psbot
Disallow: /

# Block Youdao
User-agent: YoudaoBot
Disallow: /

# BLEXBot
User-agent: BLEXBot
Disallow: /

# Block NaverBot
User-agent: NaverBot
User-agent: Yeti
Disallow: /

# Block ZBot
User-agent: ZBot
Disallow: /

# Block Vagabondo
User-agent: Vagabondo
Disallow: /

# Block LinkWalker
User-agent: LinkWalker
Disallow: /

# Block SimplePie
User-agent: SimplePie
Disallow: /

# Block Wget
User-agent: Wget
Disallow: /

# Block Pixray-Seeker
User-agent: Pixray-Seeker
Disallow: /

# Block BoardReader
User-agent: BoardReader
Disallow: /

# Block Quantify
User-agent: Quantify
Disallow: /

# Block Plukkie
User-agent: Plukkie
Disallow: /

# Block Cuam
User-agent: Cuam
Disallow: /

# https://megaindex.com/crawler
User-agent: MegaIndex.ru
Disallow: /

User-agent: megaindex.com
Disallow: /

User-agent: +http://megaindex.com/crawler
Disallow: /

User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: megaIndex.ru
Disallow: /

Making sure no content is affected by new Robots.txt file

So now you have made some changes to your Robots.txt file, and it’s time to check if any of your content is impacted due to the updation in the robots.txt file.

You can use Google search console ‘Fetch as Google tool’ to see if whether or not your content can be accessed by Robots.txt file.

These steps are simple.

Login to Google search console, select your site, go to diagnostic and Fetch as Google.

Add your site posts and check if there is any issue accessing your post.

• Save

You can also check for the crawl errors caused due to Robots.txt file under Crawl error section of search console.

Under Crawl > Crawl Error, select Restricted by Robots.txt and you will see what all links have been denied by the Robots.txt file.

Here is an example of Robots.txt Crawl Error for ShoutMeLoud:

• Save

You can clearly see that Replytocom links have been rejected by Robots.txt and so have other links which should not be a part of Google. FYI, Robots.txt file is an essential element of SEO, and you can avoid many post duplication issues by updating your Robots.txt file.

Do you use WordPress Robots.txt to optimize your site? Do you wish to add more insight to your Robots.txt file? Let us know using the comment section below. Don’t forget to subscribe to our e-mail newsletter to keep receiving more SEO tips.

Here are a few other hand-picked guides for you to read next:

• How To Generate A Disavow File Using Ahrefs SEO Suite
• How To Fix 500 Internal Server Error In WordPress
• WordPress SEO Tutorial (From Beginner to Advanced Guide)

• Save