Robots.txt file on BlogSpot Blogs: Google SEO Mistake with BlogSpot Search Pages

Well, it’s quite interesting to know that there’s a different story behind the robots.txt file for blogs run on blogger. Basically, a robots.txt file is placed on the root directory to block the search engines from crawling specific directories or pages on your blog. By this way, webmasters take control to let the spiders know what directories are accessible and what needs to be indexed. The most common question asked by BlogSpot bloggers is: How do we edit Robots.txt file in BlogSpot and answer is we can’t.

Unlike the blogs hosted on WP, that is self-hosted blogs, those blogs run on blogger do not have the option to control the robots.txt file. The complete structure including files, images and other media files you host are owned by Google. So, we have limited access when it comes to blogger – just publish content and make some tweaks on the design:) You definitely lack the complete authority on your blog.
Take a look at the screen shot of the robots.txt file for a blogspot blog below. You can find out the same for your blog by typing the following query on your address bar –

From the above screen shot or from the query you searched, it is pretty clear that the search engines are blocked to crawl the search directory on our blogs. But, this isn’t the actual case. To prove this, you may search for the following query on Google – site:* inurl:search?updated-max

You’ll find thousands of results popping up. It’s strange to see that pages which redirect to other domains have also been indexed. So, what do I do to get this stuff alright? Well, remember that your on blogger. You cannot alter a word in the file. However, self-hosted blogs need to check for the trailing slash on their robots.txt file.

In other words, when you access the robots.txt file for your blog, you should find the following phrase –  Disallow: /search/ instead of Disallow: /search (as you saw on the earlier screen shot).  The addition of the trailing slash (search/) actually informs the crawler that it’s just a directory and not a file.

Google mistake diluting BlogSpot SEO:

It’s important to note that Google, with high standards, more often than not updates their official blog saying we are keen on what should be indexed and what shouldn’t be.  As far as I know, there is no use in indexing the dynamic search result pages when compared to properly indexing a single post page on blogger.

So, it’s up to the ‘Gurus’ in Google to take a look at this small, but crucial, problem. I suggest the WP users to make the required changes on the robots.txt file while people on blogger can just sit idle. Neither can you add a noindex tag for these dynamic pages nor control the robots.txt file. So, sit back and wait for a better solution from Google just like I do. :) I may or may not be wrong with my view about the robots.txt file. So, I welcome your thoughts about this article!  Share them via comments.

I would like to know the opinion from BlogSpot bloggers. Do you think Google should disallow indexing of BlogSpot search pages or give SEO options to BlogSpot bloggers to control their blog indexing?

Subscribe on Youtube

Article By
I'm Mukund, the founder of Everything About Blogging. I'm a freelance writer on topics related to Website Optimization (SEO), blogger customizations and making money online. I'm blogging since 2008 and I'm currently a guest blogger on Shoutmeloud & Blogging With Success.


COMMENTs ( 16 )

  1. Toufiq says

    hey, i have a website 4 years old, recently google show my tag pages on search result, whats the solution, how can i get my main post above the tag result.

  2. Labnol Technology says

    i have tried to fix this problem but failed. now i don’t try because it may make Google angry. (hahaah)

  3. Zeeshan @ says

    100% agree with Talkfree7, Google wants you to create Good content not to tweak Good SEO., Google is king . He has made its algorithm may be Google give priority to blogspot domain and even we don’t know about it. ..

    • Mukund says

      Yes! That’s what even I intend to say! Rather than indexing good content, why do you index search results. There’s no use with that! So, follow the solution given by Samir and your done!

  4. Bhaveek says

    As a newbie i find these things still confusing me and I usually ignore the robots text file. :)

  5. Talkfree7 says

    Are you are teaching to GURUS??

    Google who runs the online empire and is the unmatched KING of the Virtual World : )

    To me it never matters, my blogspot blog is on high pich in comparison to many ‘high quality’, highly SEO optimized WP blogs,

    Google knows very well their job, what they are in and doing what, this is their full time biz,

    Do the Google wants a blogger to turn into a super SEO expert, never!

    Google knows a blogger should be a blogger, … SEO is none of his job, this is the job of the Google itself and it is taking a good deal of it, without worrying a non-techy blogger to focus on SEO, rather to focus on nice quality content only,

    so you too pl do not worry your self and your readers : )

    • Mukund says

      I rather posted my opinion through this post so that everyone, be it me or you on blogger enjoy a better SEO than what we do! I wish someone from the Google team looks into this problem so that everyone benefits out of the same!

  6. Akshay Kakkar says

    Hey i think you made a spelling mistake. In the first para, last line, it should be blogspot, not blogspit. :)

  7. Samir says

    Hi Mukund!
    I agree with you about the fact that by-default all blogspot blogs have this glitch in their robots.txt files. However, it is not an incurable ailment!
    At least for Google, every single blogspot blogger can remove the directory “search” from gettig indexed by using the “Remove URL” feature in Google Webmaster Tools!
    Just go to Crawler Access > Remove URL > Enter “search/” in the box> select remove directory from results and cache and you are done!

    • Mukund says

      Well, that’s an intelligent solution and I forgot about the remove URL tool! But, do you think the above method would work? Why because, you’ll notice that the web spiders have indexed a lot of page re-directions on blogger as well. So, what do we do about???

      • Samir says

        Well all my remove URL requests have worked so far. What re-directions are you talking about exactly?

        • Mukund says

          Well, I meant the stuff that Google indexed for the search result that I mentioned in the article above. I found that every stuff is indexed wrongly only due to that missing trailing slash. So, your right buddy! Remove URL tool is the only way we can remove the above stuff! Thanks for sharing the info:)