What is Google Crawling and Indexing

SEO is a very big sea and to understand SEO, we should know all the basic terminology of SEO. Crawling and indexing are two such words, which are the basis of SEO. If you have been in the web world for a while you must be aware of the terms Google Crawling and Indexing. These are the two terms on which the whole web world  depends. Lets understand both the words and get more in-depth information about the same.

Crawling means the visit of Google to track your web page. This process is done by Spider of Google crawler. Indexing means when crawling has been done putting the results  on Google’s index .

Google Crawling Indexing

What is Google Crawling:

Crawling means follow a path. In SEO word, crawling means following your links and crawling your Website. When bots come to your Website (On any page), it follows other linked page and crawl your site. This is one reason we create site map, as it contains all the links of our blog and bots uses it to deep crawl our pages.

Usually we stop crawling of a certain part of our site is by using Robots.txt file.

What is Google indexing:

In layman word, Indexing is the process of adding Webpages into Google search. Depending upon what meta tag you used for your page (Index or NO-index), Google crawl and index your pages. No-index means that page will not be added in Web search. By default any WordPress post or pages are indexable.

A good idea for better search engine ranking is to let only vital part of your blog/Website be in index and do not index un-necessary archive like tags, categories and all such pages.

There are millions of websites on this earth but are everyone satisfied with the crawling rate?? No!! Even you must have felt this at the beginning that why aren’t your articles getting indexed. Let us see some major factors which play important role at the backend of crawling and indexing:

Domain Name:

Since the Google Panda update domain name importance has risen significantly and domains that include the main keyword are given importance. Also the crawling rate is high for those domains that have good PR.


  1. How to select great domain name
  2. How to find best domain name for your website


The more the number of backlinks you have more trust worthy and reputation you are in the eyes of search engine. If you have good rankings but certainly you aren’t earning any backlinks for your site Search engine may assume that you have low quality content.


  1. Top 5 online backlink checker tools
  2. 101 ways to get quality backlink to your blog

Internal Linking

There have been so many discussions regarding the internal linking also known as deep linking. People even suggest to use the anchor text within the same article as it helps in Deep linking of site.


  1. Interlink blog post like a pro using SEO smart link plugin
  2. Interlink your old post to keep your readers for long
  3. Quickly interlink your blog post with Insight WordPress plugin

XML Sitemaps:

The moment you setup a site in WordPress you are advised to use XML sitemap so that your sitemaps can be auto generated and google can be informed about your blog/website pages and it can be crawled.


  1. How to submit sitemap to Google

Duplicate Content:

This could be really penalizing for you. Recently I found a site which had so many repetitions of same para’s in different posts which was banned by Google because of this practice. Try not to have any duplicate content in your site. Fix all your re-directs 301 and 404 pages for better optimization.


  1. How to fix replytocom issue to avoid duplicate content issue
  2. Use Robots.txt to fight against duplicate content on your site

URL canonicalization:

Create SEO friendly URL for each page on your site as this keeps huge importance in SEO terms.
Meta Tags:
Have unique and less competitive meta tags for your website so that you can be at top rankings .


  1. How to use Meta tag analyzer while doing SEO


Make sure you have added all the main ping sites to your WordPress. WordPress also has auto-pinging feature which will inform search engines and ping sites about your updates available.

Optimizing using these basic techniques can really help you to get the best crawl rate for your website so make sure you try all these.

Subscribe on Youtube

Article By
I write about wordpress,Blogging , Online money making with a zeal to earn and flourish my knowledge.


COMMENTs ( 13 )

  1. says

    thanks a lot for this useful article. One thing I am sure is that I never find it difficult in getting index by Google or any other search engine. I will sharing this post with my friends who are bloggers.
    Keep up with the good work.

  2. Surendra says

    Hello @ Harsh,
    Today I started learning SEO From starting from your first post ! Please update some posts as yoast SEO plugin as been updated a lot.

  3. doroki says

    Yeah, You will have to add your site to webmaster tool for adding and indexing. robot.txt will be generating sitemap for you site that can be used as your site map on webmastertool. Backlinks and right contents are always good to get indexed by google as well as PR related stuffs.

  4. shenoyjoseph says

    i agree these fact and following these method for my blog to get better results. “The more the number of backlinks you have more trust worthy and reputation you are in the eyes of search engine.”

  5. sureshpeters says

    hey author , you missed one important recipe which is called robots.txt, if you don’t make it. bots won’t be come effectively . making a sitemap is not only the task we have to submit to google, yahoo,bing, and more webmaster account and if you need indexing soon then you have to post recently then you will get into first page if the keyword is less competition

  6. WpRack says

    Good post with important points. I have read in somewhere that the keywords included in the domain names are not given more preferrence after the panda update. Is that true?

  7. ajay s bisht says

    ashish sir i would like to state my view on this article..
    although you have covered a lot topics in this but a little elaboration would have been of great help…
    hope you come up with a detailed article…

  8. Fayaz Ahmad says

    If you need better crawling of your web pages the you should submit a sitemap to you root directory , for search engine and also create a site map for your users also