Help Google Bot to crawl your website effectively

by samrat on July 8, 2009

in SEO

Everyone loves to get their web pages indexed by Google as soon as they write them. If you can guide the GoogleBot to crawl your site easily it will index your pages much faster than your expectations.

Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index. When the Googlebot visits your website it detects links on each page and adds them to the list of pages to crawl. If your site has got a very authority your pages will get indexed by Google within few minutes. Below are some useful tips to get your pages indexed by Google effectively.

Googlebot Image Credit

1. Create and submitting a Sitemap to Google. Even if your site is already indexed in Google submitting a sitemap will give Google a better way to understand about your site. If you are using Wordpress use any of the plugins that can create a sitemap for you.

2. Register for Google Webmasters Tools and check the crawl error page frequently. The crawl errors page provides details about the URLs in your site that Google couldn’t access. So you can fix the errors once you know the problem area.

3. Make sure that the required URL’s are not blocked by robots.txt file or meta tags. If you are not sure about robots.txt file use the Webmasters Tools to create a robots.txt for your website/blog.

4. Search engines cannot understand images and videos. So if your website has good number of images or videos try to provide relevant information about them.

These are just some of the tips which will help you to get your site crawled by Google effectively. If you have any other tips then do share them with us.

Other articles which will help you to crawl your website effectively are :

Link building , Social bookmarking websites, Anchor text technique, Blog post roundup

Related posts:

  1. [Blogger] Google will use Feeds to Crawl your Web pages from now.
  2. Google announced new Sitemap Generator Tool
  3. How to submit your Blog sitemap to Google Webmaster Tool
  4. Use robots.txt to protect your blog from duplicate content issue
  5. How To Get Quick Backlinks From High PR Sites
  6. SEO: How to Index your Website Within 24 hours in Search Engine
  7. Google XML Sitemap : Wordpress essential plugin
  8. Wordpress SEO : 5 basic practice you should implement


{ 1 trackback }

FAQPAL Blog
December 14, 2009 at 20:29

{ 11 comments… read them below or add one }

1 Chethan July 8, 2009 at 13:05

Excellent Post!
I like the picture which you have posted!
I am Google Bot, I control the earth!
lols
.-= Chethan ´s last blog ..Expand and List: Twitter’s New Tool =-.

Reply

2 Michael Aulia July 8, 2009 at 13:58

I’ve been having sitemap error for a whole week and my search engine traffic went down dramatically. Thankfully it’s solved today :( Gonna write up a post on it in case other bloggers having the same problem
.-= Michael Aulia ´s last blog ..Don™t be a Darren Rowse, Don™t be a John Chow =-.

Reply

3 Harsh Agrawal July 9, 2009 at 20:34

Bloggers usually avoid checking sitemap error and Google webmaster tool.. IMO one should make a weekly check of their webmaster tool for any possible error. Will wait for your post Michael. :)

Reply

4 Anish K.S July 8, 2009 at 17:12

Good tips :)
.-= Anish K.S ´s last blog ..Catch ˜Things We Lost in the Fire™ only on HBO =-.

Reply

5 Ajith Edassery July 8, 2009 at 17:46

Nice post Samrat buddy….

And Harsh, your aquamarine based customization looks awesome. Honeytech has done a good job. How much did you pay for it??
.-= Ajith Edassery ´s last blog ..10 Innovative & Interesting Search Engines =-.

Reply

6 Harsh Agrawal July 9, 2009 at 10:25

Thanks Ajith I’m glad you like it. Honey and I, share a very good relation apart from blogging. Hope this answered your question :)

Reply

7 Atul July 9, 2009 at 13:11

i implement all of them image is really superb….hahah
.-= Atul ´s last blog ..Railway Budget Highlights 2009 =-.

Reply

8 Nihar July 9, 2009 at 23:44

Nice post samrat,

Yeah it is very much important to submit a sitemap and also use sitemap generator plugin.
.-= Nihar ´s last blog ..New GMail Account requires Mobile number to activate it! =-.

Reply

9 Harsh Agrawal July 10, 2009 at 01:17

That’s true Nihar, Sitemap is an essential part of SEO for any blog.

Reply

10 Rajesh July 14, 2009 at 13:10

@ Amit Sir,
I am using your All tips. Although I have Installed 2 Sitemaps i.e Dragon Sitemap & Google Sitemap.

Is their any problem if I uses post?
.-= Rajesh ´s last blog ..Download Luxury Apple Wallpapers =-.

Reply

11 Martin L. October 29, 2009 at 10:46

Great info and tips… i am try to index my site at google SERP. Hope this google webmaster tools can help.

Reply

Leave a Comment

Previous post:

Next post: