Everyone loves to get their web pages indexed by Google as soon as they write them. If you can guide the GoogleBot to crawl your site easily it will index your pages much faster than your expectations.
Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index. When the Googlebot visits your website it detects links on each page and adds them to the list of pages to crawl. If your site has got a very authority your pages will get indexed by Google within few minutes. Below are some useful tips to get your pages indexed by Google effectively.
How to make Search engine bots crawl your site effectively:
1. Create and submitting a Sitemap to Google:
Even if your site is already indexed in Google submitting a sitemap will give Google a better way to understand about your site. If you are using WordPress use any of the plugins (Google XML sitemap)that can create a sitemap for you. Also, place your sitemap link in your robots.txt and in your sitemap file, as it will help Google bots to crawl your Website effectively.
2. Register for Google Webmasters Tools and check the crawl error page frequently:
The crawl errors page provides details about the URLs in your site that Google couldn’t access. So you can fix the errors once you know the problem area. Harsh has covered a detailed article on understanding Google WebMaster tools crawl errors, and using it, you can find why bots are facing issue with crawling of your website.
Make sure that the required URL’s are not blocked by robots.txt file or meta tags. If you are not sure about robots.txt file use the Webmasters Tools to create a robots.txt for your website/blog. Also, block bots access to those part of Website, which is not important for search engines. Hint: your admin folder.
- Robots.txt stops search engine bots to crawl part of your blog
- Optimize WordPress Robots.txt file for SEO
4. Image and video sitemap:
Search engines cannot understand images and videos. So if your website has good number of images or videos try to provide relevant information about them. For images you need to add ALT tag, which could easily by done with the help of “SEO friendly images” plugin. For videos, you can create a video sitemap and submit it to Google.
These are just some of the tips which will help you to get your site crawled by Google effectively. If you have any other tips then do share them with us.
Other articles which will help you to crawl your website effectively are :