Help Google Bot to Crawl your website effectively

Everyone loves to get their web pages indexed by Google as soon as they write them. If you can guide the GoogleBot to crawl your site easily it will index your pages much faster than your expectations.

Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index. When the Googlebot visits your website it detects links on each page and adds them to the list of pages to crawl. If your site has got a very authority your pages will get indexed by Google within few minutes. Below are some useful tips to get your pages indexed by Google effectively.

How to make Search engine bots crawl your site effectively:

1. Create and submitting a Sitemap to Google:

Even if your site is already indexed in Google submitting a sitemap will give Google a better way to understand about your site. If you are using WordPress use any of the plugins (Google XML sitemap)that can create a sitemap for you. Also, place your sitemap link in your robots.txt and in your sitemap file, as it will help Google bots to crawl your Website effectively.

2. Register for Google Webmasters Tools and check the crawl error page frequently:

The crawl errors page provides details about the URLs in your site that Google couldn’t access. So you can fix the errors once you know the problem area. Harsh has covered a detailed article on understanding Google WebMaster tools crawl errors, and using it, you can find why bots are facing issue with crawling of your website.

3. Robots.txt:

Make sure that the required URL’s are not blocked by robots.txt file or meta tags. If you are not sure about robots.txt file use the Webmasters Tools to create a robots.txt for your website/blog. Also, block bots access to those part of Website, which is not important for search engines. Hint: your admin folder.

4. Image and video sitemap:

Search engines cannot understand images and videos. So if your website has good number of images or videos try to provide relevant information about them. For images you need to add ALT tag, which could easily by done with the help of “SEO friendly images” plugin. For videos, you can create a video sitemap and submit it to Google.

These are just some of the tips which will help you to get your site crawled by Google effectively. If you have any other tips then do share them with us.

Other articles which will help you to crawl your website effectively are :

Link building ,Anchor text technique, Blog post roundup

Subscribe on Youtube

Article By


COMMENTs ( 5 )

  1. Chris says

    Can’t get Google Webmaster Tools to verify my site no matter what I do. Finally gave up. I get most of my traffic from Yahoo anyway :) Screw ’em.

  2. Martin L. says

    Great info and tips… i am try to index my site at google SERP. Hope this google webmaster tools can help.

  3. Michael Aulia says

    I’ve been having sitemap error for a whole week and my search engine traffic went down dramatically. Thankfully it’s solved today :( Gonna write up a post on it in case other bloggers having the same problem

    • says

      Bloggers usually avoid checking sitemap error and Google webmaster tool.. IMO one should make a weekly check of their webmaster tool for any possible error. Will wait for your post Michael. :)

  4. Chethan says

    Excellent Post!
    I like the picture which you have posted!
    I am Google Bot, I control the earth!