ShoutMeLoud – Shouters Who Inspire

Superlinks
≡ Menu
≡ Menu

Robots.txt file on BlogSpot Blogs: Google SEO Mistake with BlogSpot Search Pages

TA
Robots.txt file on BlogSpot Blogs: Google SEO Mistake with BlogSpot Search Pages

Well, it’s quite interesting to know that there’s a different story behind the robots.txt file for blogs run on blogger. Basically, a robots.txt file is placed on the root directory to block the search engines from crawling specific directories or pages on your blog. By this way, webmasters take control to let the spiders know what directories are accessible and what needs to be indexed. The most common question asked by BlogSpot bloggers is: How do we edit Robots.txt file in BlogSpot and answer is we can’t.

Unlike the blogs hosted on WP, that is self-hosted blogs, those blogs run on blogger do not have the option to control the robots.txt file. The complete structure including files, images and other media files you host are owned by Google. So, we have limited access when it comes to blogger – just publish content and make some tweaks on the design:) You definitely lack the complete authority on your blog.
Take a look at the screen shot of the robots.txt file for a blogspot blog below. You can find out the same for your blog by typing the following query on your address bar – http://BLOGURL.com/robots.txt

robots file 550x208
From the above screen shot or from the query you searched, it is pretty clear that the search engines are blocked to crawl the search directory on our blogs. But, this isn’t the actual case. To prove this, you may search for the following query on Google – site:*.blogspot.com inurl:search?updated-max

You’ll find thousands of results popping up. It’s strange to see that pages which redirect to other domains have also been indexed. So, what do I do to get this stuff alright? Well, remember that your on blogger. You cannot alter a word in the file. However, self-hosted blogs need to check for the trailing slash on their robots.txt file.

In other words, when you access the robots.txt file for your blog, you should find the following phrase –  Disallow: /search/ instead of Disallow: /search (as you saw on the earlier screen shot).  The addition of the trailing slash (search/) actually informs the crawler that it’s just a directory and not a file.

Google mistake diluting BlogSpot SEO:

It’s important to note that Google, with high standards, more often than not updates their official blog saying we are keen on what should be indexed and what shouldn’t be.  As far as I know, there is no use in indexing the dynamic search result pages when compared to properly indexing a single post page on blogger.

So, it’s up to the ‘Gurus’ in Google to take a look at this small, but crucial, problem. I suggest the WP users to make the required changes on the robots.txt file while people on blogger can just sit idle. Neither can you add a noindex tag for these dynamic pages nor control the robots.txt file. So, sit back and wait for a better solution from Google just like I do. :) I may or may not be wrong with my view about the robots.txt file. So, I welcome your thoughts about this article!  Share them via comments.

I would like to know the opinion from BlogSpot bloggers. Do you think Google should disallow indexing of BlogSpot search pages or give SEO options to BlogSpot bloggers to control their blog indexing?

  • Author Bio

  • Latest Post

Article by Mukund

Mukund has written 29 articles.

If you like This post, you can follow ShoutMeLoud on Twitter. Subscribe to ShoutMeLoud feed via RSS or EMAIL to receive instant updates.


{ 16 comments… add one }

  • Toufiq

    hey, i have a website 4 years old, recently google show my tag pages on search result, whats the solution, how can i get my main post above the tag result.

    Reply
  • Mani

    It would be Great if Google give SEO control to the Blogspot admin.

    Reply
  • Labnol Technology

    i have tried to fix this problem but failed. now i don’t try because it may make Google angry. (hahaah)

    Reply
    • Mukund

      I suggest you to follow the solution provided by Samir! That should definitely work!

      Reply
  • Zeeshan @ Hubspoint.com

    100% agree with Talkfree7, Google wants you to create Good content not to tweak Good SEO., Google is king . He has made its algorithm may be Google give priority to blogspot domain and even we don’t know about it. ..

    Reply
    • Mukund

      Yes! That’s what even I intend to say! Rather than indexing good content, why do you index search results. There’s no use with that! So, follow the solution given by Samir and your done!

      Reply
  • Bhaveek

    As a newbie i find these things still confusing me and I usually ignore the robots text file. :)

    Reply
    • Mukund

      Lol:) Even I used to ignore these stuffs when I started blogging!

      Reply
  • Talkfree7

    Are you are teaching to GURUS??

    Google who runs the online empire and is the unmatched KING of the Virtual World : )

    To me it never matters, my blogspot blog is on high pich in comparison to many ‘high quality’, highly SEO optimized WP blogs,

    Google knows very well their job, what they are in and doing what, this is their full time biz,

    Do the Google wants a blogger to turn into a super SEO expert, never!

    Google knows a blogger should be a blogger, … SEO is none of his job, this is the job of the Google itself and it is taking a good deal of it, without worrying a non-techy blogger to focus on SEO, rather to focus on nice quality content only,

    so you too pl do not worry your self and your readers : )

    Reply
    • Mukund

      I rather posted my opinion through this post so that everyone, be it me or you on blogger enjoy a better SEO than what we do! I wish someone from the Google team looks into this problem so that everyone benefits out of the same!

      Reply
  • Akshay Kakkar

    Hey i think you made a spelling mistake. In the first para, last line, it should be blogspot, not blogspit. :)

    Reply
  • Samir

    Hi Mukund!
    I agree with you about the fact that by-default all blogspot blogs have this glitch in their robots.txt files. However, it is not an incurable ailment!
    At least for Google, every single blogspot blogger can remove the directory “search” from gettig indexed by using the “Remove URL” feature in Google Webmaster Tools!
    Just go to Crawler Access > Remove URL > Enter “search/” in the box> select remove directory from results and cache and you are done!

    Reply
    • Mukund

      Well, that’s an intelligent solution and I forgot about the remove URL tool! But, do you think the above method would work? Why because, you’ll notice that the web spiders have indexed a lot of page re-directions on blogger as well. So, what do we do about???

      Reply
      • Samir

        Well all my remove URL requests have worked so far. What re-directions are you talking about exactly?

        Reply
        • Mukund

          Well, I meant the stuff that Google indexed for the search result that I mentioned in the article above. I found that every stuff is indexed wrongly only due to that missing trailing slash. So, your right buddy! Remove URL tool is the only way we can remove the above stuff! Thanks for sharing the info:)

          Reply

Leave a Comment