Before Google can rank a page, it has to know the page exists. Crawling is that discovery step. A program called a crawler (Google’s is Googlebot) requests a URL, downloads the HTML, notes every link on it, and adds those links to a queue of pages to fetch next. It repeats this across billions of pages, revisiting popular ones often and quiet ones rarely.
Crawling is separate from indexing. A crawled page has been fetched. An indexed page has been fetched, understood and stored so it can appear in results. Plenty of pages get crawled and never indexed.
Why it matters
Take Coastline Plumbing in Austin. They add a new page for water heater repair in Round Rock. If nothing links to it and it isn’t in their sitemap, Googlebot may never find it, and a page Google never finds can’t bring in a single call. The same happens when a developer accidentally blocks a folder in robots.txt or puts important links behind a script that only runs after a click.
How crawlers find your pages
- Links from pages they already know, both your own internal links and links from other sites.
- XML sitemaps that list the URLs you want found, along with when they last changed.
- Previous visits, since Google keeps a record of URLs it has seen before and checks them again.
How to check crawlability
In Google Search Console, the Crawl Stats report (under Settings) shows how many requests Googlebot makes per day, the response codes it gets, and how fast your server answers. The URL Inspection tool tells you when Google last crawled a specific page and whether it was allowed to.
A site crawler gives you the same view from the outside. RankThat’s Site Audit crawls your site as RankThatBot and flags pages that return errors, pages blocked by robots.txt, and pages no other page links to.
Common mistakes
- Blocking CSS or JavaScript files that Google needs to render the page.
- Relying on links that only appear after a user clicks or scrolls.
- Leaving staging-site rules such as
Disallow: /in place after launch. - Assuming a page in the sitemap will be crawled even when every internal link to it is broken.
Example
A basic crawl path looks like this: Googlebot fetches your homepage, finds a link to /services/, fetches that, finds /services/water-heater-repair/, and queues it. If any link in that chain is missing or blocked, the pages after it depend on your sitemap or outside links to be discovered at all. Keeping the crawl budget focused on useful pages helps larger sites get new content found sooner.