ShoutMeLoud – Shouters Who Inspire

Superlinks
≡ Menu
≡ Menu

12 Solid Tips to Increase Google Crawl Rate Of Your Website

rainmaker
12 Solid Tips to Increase Google Crawl Rate Of Your Website

Site crawling is an important aspect of SEO and if bots can’t crawl your site effectively, you will notice many important pages are not indexed in Google or other search. A site with proper navigation helps in deep crawling and indexing of your site. Specially, for a news site it’s important that Search engine bots should be indexing your site within minutes of publishing and that will happen when bots can crawl site ASAP you publish something.

There are many things which we can do to increase the effective site crawl rate and get faster indexing. Search engines use spiders and bots to crawl your website for indexing and ranking. Your site can only be included in search engine results pages (SERP’s) if it is in the search engine’s index. Otherwise, customers will have to type in your URL to get to your site. Hence you must have a good and proper crawling rate of your website or blog to succeed. Here I’m sharing some of the effective ways to increase site crawl rate and increase visibility in popular search engines.

Simple and Effective tips to Increase Site Crawl Rate

Increase Google Crawl Rate As I mentioned you can do many things to help search engine bots find your site and crawl them. Before I get into technical aspect of crawling, in simple words: Search engine bots follow links to crawl a new link and one easy way to get search engine bots index quickly, get your site link on popular sites by commenting, guest posting.

If not, there are many other things we can do from our end like Site pinging, Sitemap submission and controlling crawling rate by using Robots.txt. I will be talking about few of these methods which will help you to increase Google crawl rate and get bots crawl your site faster and better way.

1. Update your site Content Regularly

Content is by far the most important criteria for search engines.  Sites that update their content on a regular basis are more likely to get crawled more frequently. You can provide fresh content through a blog that is on your site. This is simpler than trying to add web pages or constantly changing your page content.  Static sites are crawled less often than those that provide new content.

Many sites provide daily content updates. Blogs are the easiest and most affordable way to produce new content on a regular basis.  But you can also add new videos or audio streams to your site. It is recommended that you provide fresh content at least three times each week to improve your crawl rate. Here is little dirty trick for static sites, you can add a Twitter search widget or your twitter profile status widget if it’s very effective. This way, at least a part of your site is constantly updating and will be helpful.

2. Increase your Pagerank

This is one of the most important factor. A lot of the search engines crawl patterns are dictated by PageRank, with higher PR sites/blog generally being crawled more regularly. But as usual this isn’t the only factor, while you build links to your site there are other things you should be doing. Though it could be considered as myth too, as Google repeatedly mention Page rank has nothing to do with site ranking and traffic, but again to increase page rank we have to build links, and that helps a lot for faster crawling.

3. Server with Good Uptime

Host your blog on a reliable server with good uptime. You don’t want Googlebots to visit your blog during downtime right na? In fact, if your site is down for long, Google crawlers will set their crawling rate accordingly and you will find it harder to get your new content indexed faster. There are many Good hosting sites which offers 99%+ uptime and you can look at them at suggested WebHosting page.

4. Create Sitemaps

Sitemap submission is one of the first few things which you can do to make your site discover fast by search engine bots. In WordPress you can use Google XML sitemap plugin to generate dynamic sitemap and submit it to Webmaster tool.

5. Avoid Duplicate Content

Copied content decreases crawl rates. Search engines can easily pick up on duplicate content.  This can result in less of your site being crawled. It can also result in the search engine banning your site or lowering your ranking.  You should provide fresh and relevant content.  Content can be anything from blog postings to videos. There are many ways to optimize your content for search engines.  Using those tactics can also improve your crawl rate.  It is a good idea to verify you have no duplicate content on your site. Duplicate content can be between pages or between websites. There are free content duplication resources available online you can use to authenticate your site content. Also see below tips on using robots.txt.

6. Reduce your site Loading Time

Mind your page load time, Note that the crawl works on a budget– if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.

7. Block access to unwanted page via Robots.txt

There is no point letting search engine bots crawling useless pages like admin pages, back-end folders as we don’t index them in Google and so there is no point letting them crawl such part of site. A Simple editing on Robots.txt will help you to stop bots from crawling such useless part of your site. You can learn more about Robots.txt for WordPress below:

8. Monitor and Optimize Google Crawl Rate

Now You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyse. You can manually set your Google crawl rate and increase it to faster as given below. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here.

how to increase crawl rate

9. Use Ping services:

Pinging is a great way to show your site presence and let bots know when your site content is updated. There are many manual ping services like Pingomatic and in WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post.

10. Submit your site to online Directories like DMoz etc.

Directories are proved to be very beneficial in driving traffic from search engines in large amount. Since Technoarti and DMoz are considered as authoritative & active directories, bots will come to your site by following your site listing pages on such directories.

11. Interlink your blog pages like a pro:

Interlinking not only helps you to pass link juice but also help search engine bots to crawl deep pages of your site. When you write a new post, go back to related old posts and add a link to your new post there. This will no directly help in increasing Google crawl rate but will help bots to effectively crawl deep pages on your site.

12. Don’t forgot to Optimize Images

Crawlers are unable to read images directly.  If you use images, be sure to use alt tags to provide a description that search engines can index.  Images are included in search results but only if they are properly optimized. You an learn about Image optimization for SEO here and you should also consider installing Google image sitemap plugin and submit it to Google. This will help bots to find all your images and you can expect decent amount of traffic from search engine bots, if you have taken care of image alt tag properly.

Well, these are few tips that I can think of which will help you to increase site crawl rate and get better indexing in Google or other search engine. A last tip which I would like to add here is, add your sitemap link in the footer of your site. This will help bots to find your sitemap page quickly and they can crawl and index deep pages of your site from sitemap.

Do let us know if you are following any other method to increase Google crawl rate of your site? If you find this post useful, don’t forget to tweet and share it on Facebook.

  • Author Bio

  • Latest Post

Article by Karan Singh Chauhan

Karan has written 1 articles.

If you like This post, you can follow ShoutMeLoud on Twitter. Subscribe to ShoutMeLoud feed via RSS or EMAIL to receive instant updates.


    { 15 comments… add one }

    • Himanshu Jain

      Do anyone knows how to set Custom Crawl rate at Google Webmasters Tool.

      Reply
    • Rafael

      We just added a twitter timeline. We hope it increases our crawl rate. Question, in our timeline we choose to display tweets with certain keywords. will those keywords help our onpage seo also.

      Reply
    • chyardi

      For point 10 submit site to directories DMOZ is pretty easy, but to be accepted into directories like DMOZ need time :)

      Reply
      • Harsh Agrawal

        @Chyardi
        I agree…Getting inside DMOZ is tough, and not sure what all criteria they actually look before accepting any website. They should come up with paid solution for small businesses, as DMOZ is still seen as a standard directory.

        Reply
    • Nikhil Saliya

      Does Improved Crawl rate Increase Traffic?
      Is there any Direct or indirect Relations Between theme?
      If yes, Please Provide me Over View.

      Reply
      • karan singh chauhan

        Yes Nikhil, Improved Crawl rate can also Increase web traffic as well. For ex- If you have to post some latest news you will get done by now and your post will rank higher and clicked by many users instead of waiting your post to get listed on Google search later on.

        Reply
    • Nizam

      Awesome tips. Submitting blog to online Directories is very useful and also submitting sitemap to Google is important. Thanks Karan for these excellent tips :)

      Reply
    • manoj

      i do all the things you have mentioned above but still i found out that my site was last crawled on july 26….is there any way we can force google into crawling our site,,,and the option of fetch as google bot,,does it make google to crawl the site

      Reply
    • Ammar Ali

      I’ve a question mate. Why we need to crawl our website fast in google? What’s advantage dude? :)

      I’d love to hear from you!

      Reply
    • Vijay Reddy

      Choose web hosting carefully, because it effects crawl rate if there is more down time and loading time…

      Reply
    • Ambika Mahajan

      Read the whole of it.
      Retweeted it, LIKE-d it on facebook and shared it on Google plus because it contains lots of useful information.
      Written in an extremely understandable language (making it easier for a person with a non technical base like me to understand it), it would benefit any new blogger immensely.
      Could you suggest me a few good directories for submission in the health niche`?

      Reply
    • mohan

      I tries above steps 1,4,6,10,11,12 it’s really worked.Useful tip for new bloggers.

      Reply
    • Nirmal Kumar Pandit

      It will very helpfull to increase traffic for new website. All of the mentioned point definetly help us to grow crawl. Thanks to provide such kind of information.

      Reply
    • Gautham Nekkanti

      SEO Smartlinks is a great plugin that can help interlink like a pro

      Reply
    • Ashokkumar

      Wow….Great tips dude…i will do every steps today onwards….thanks lot…keep sharing dude..

      Reply

    Leave a Comment