Google doesn’t have unlimited time for any one website. Crawl budget describes how much attention Googlebot gives yours. Google explains it as two things combined: the crawl capacity limit (how much crawling your server can handle without slowing down) and crawl demand (how much Google wants to crawl, based on how popular and how often updated your pages are).
Does your site need to worry?
Probably not, if you run a typical local business site. Google’s own guidance says crawl budget is mainly a concern for sites with more than about a million pages, or more than about 10,000 pages that change daily, and for sites with a large share of URLs stuck as “Discovered, currently not indexed”.
Coastline Plumbing with 60 pages has nothing to manage here. Bean & Barrel Coffee, with 400 products, is fine too, unless its filters generate 200,000 URL combinations. That is where crawl budget becomes a real issue.
What wastes crawl budget
- Faceted navigation creating near-endless filter and sort URLs (
?roast=dark&size=1kg&sort=price). - Session IDs and tracking parameters in internal links.
- Redirect chains and long lists of internal links that point at redirects.
- Soft 404s and duplicate pages that Google keeps re-checking.
- A slow server. If responses slow down or return 5xx errors, Google crawls less.
How to improve it
- Block crawling of low-value parameter URLs in robots.txt, and link to filtered pages only when they deserve to rank.
- Keep your XML sitemap limited to canonical URLs that return 200.
- Link internally to final URLs, not redirects.
- Speed up server response time, especially for pages Googlebot requests most.
- Remove or consolidate thin and duplicate pages.
How to check
Search Console’s Crawl Stats report (Settings, Crawl stats) shows total crawl requests, average response time and which file types and status codes Googlebot is getting. If a big share of requests go to parameter URLs or redirects, that’s wasted budget.
Server logs give the full picture. Log file analysis shows exactly which URLs Googlebot requested and how often, so you can see whether it spends its time on your best pages or on filter combinations nobody searches for.
Common mistake
Using noindex to save crawl budget. Google still has to crawl a page to see the noindex tag, so it doesn’t reduce crawling much. Robots.txt rules and cleaner internal linking do.