[Blogger] Google will use Feeds to Crawl your Web pages from now.

by Harsh Agrawal on October 31, 2009

in SEO

This is a big news for bloggers and SEO person. Till now we keep our feeds as noindex and nofollow to prevent post duplication issue. But from now on, it’s going to change as Google officially announced that it will now use Rss/feeds to find new content for your blog.

According to official announcement from Google team

RSS/Atom feeds have been very popular in recent years as a mechanism for content publication. They allow readers to check for new content from publishers. Using feeds for discovery allows us to get these new pages into our index more quickly than traditional crawling methods. We may use many potential sources to access updates from feeds including Reader, notification services, or direct crawls of feeds.

So from now on if you disable indexing of feeds via robots.txt or any other method, you should avoid doing it and let your feed crawl able by Google search engine bots.

google crawler test thumb

You can use Google webmaster tools to check your robots.txt settings. Follow the above mentioned screenshot which will give you an idea about where to find test robots.txt settings.

If you have installed RSS cloud feeds plugin on wordpress, now your RSS feeds will also help you to make your site crawlable instantly.

Enter your email address:

Related posts:

  1. How to transfer your RSS feeds from feedburner account to another account?
  2. Help Google Bot to crawl your website effectively
  3. [Blogger tips] Change your Email delivery time for RSS feeds in feedburner
  4. Use robots.txt to protect your blog from duplicate content issue
  5. How to Analyze your Feeds? Blogger Tip
  6. Bloggers move your Feed burner feeds to Google
  7. What is RSS feeds?
  8. Feed Flare Feed Burner Option : Make your Feeds Social


{ 10 comments… read them below or add one }

1 Puneet October 31, 2009 at 13:18

Thats really great move by google….
but how much will it affect seo – that we have to wait and watch..

Reply

2 IndianCashMaker October 31, 2009 at 17:33

i hope i am not left behind…have too little feeds subscriptions

Reply

3 Gautam October 31, 2009 at 17:47

But that means that we cant have short excerpts of the posts in feeds, we need to have the full post. This might prevent feed subscribers from visiting your site :-(

Reply

4 Sourish Nath October 31, 2009 at 22:44

It will help those without a proper sitemap.

Sourish,
(Admin)
OneTrickADay

Reply

5 Shabnam Sultan November 1, 2009 at 03:00

I too have few subscribers.

Reply

6 Ajith Edassery November 1, 2009 at 19:13

I am not sure if it will make any difference at all to us. The Google bots doesn’t need to take the feed root to crawl your blog, right? Mostly with the content change, the ping servers woudl have been already alerted. Feed is only a secondary mechanism.

Or didn’t I get you right?

Reply

7 Harsh Agrawal November 2, 2009 at 14:27

Ajith I agree with you here, it will be used as a second mechanism and this is good since google will find new way to index your content. Specially RSS feed directories might become very popular after this. :)

Reply

8 himanshu November 1, 2009 at 23:25

hey since i have a wordpress.com , will it be indexed automatically?

Reply

9 Shahab November 1, 2009 at 23:54

Hey Harsh i can see dat your robots.txt file its still unedited.
I mean why you have not removed the line “Disallow: /feed/” ??

Reply

10 George Serradinho November 2, 2009 at 14:28

Thanks for this, just went and had a look to see if all is ok.

Reply

Leave a Comment

Previous post:

Next post: