Well, it’s quite interesting to know that there’s a different story behind the robots.txt file for blogs run on blogger. Basically, a robots.txt file is placed on the root directory to block the search engines from crawling specific directories or pages on your blog. By this way, webmasters take control to let the spiders know what directories are accessible and what needs to be indexed. The most common question asked by BlogSpot bloggers is: How do we edit Robots.txt file in BlogSpot and answer is we can’t.
Unlike the blogs hosted on WP, that is self-hosted blogs, those blogs run on blogger do not have the option to control the robots.txt file. The complete structure including files, images and other media files you host are owned by Google. So, we have limited access when it comes to blogger – just publish content and make some tweaks on the design:) You definitely lack the complete authority on your blog.
Take a look at the screen shot of the robots.txt file for a blogspot blog below. You can find out the same for your blog by typing the following query on your address bar – http://BLOGURL.com/robots.txt
From the above screen shot or from the query you searched, it is pretty clear that the search engines are blocked to crawl the search directory on our blogs. But, this isn’t the actual case. To prove this, you may search for the following query on Google – site:*.blogspot.com inurl:search?updated-max
You’ll find thousands of results popping up. It’s strange to see that pages which redirect to other domains have also been indexed. So, what do I do to get this stuff alright? Well, remember that your on blogger. You cannot alter a word in the file. However, self-hosted blogs need to check for the trailing slash on their robots.txt file.
In other words, when you access the robots.txt file for your blog, you should find the following phrase - Disallow: /search/ instead of Disallow: /search (as you saw on the earlier screen shot). The addition of the trailing slash (search/) actually informs the crawler that it’s just a directory and not a file.
Google mistake diluting BlogSpot SEO:
It’s important to note that Google, with high standards, more often than not updates their official blog saying we are keen on what should be indexed and what shouldn’t be. As far as I know, there is no use in indexing the dynamic search result pages when compared to properly indexing a single post page on blogger.
So, it’s up to the ‘Gurus’ in Google to take a look at this small, but crucial, problem. I suggest the WP users to make the required changes on the robots.txt file while people on blogger can just sit idle. Neither can you add a noindex tag for these dynamic pages nor control the robots.txt file. So, sit back and wait for a better solution from Google just like I do. :) I may or may not be wrong with my view about the robots.txt file. So, I welcome your thoughts about this article! Share them via comments.
I would like to know the opinion from BlogSpot bloggers. Do you think Google should disallow indexing of BlogSpot search pages or give SEO options to BlogSpot bloggers to control their blog indexing?