Every time anyone, or anything, requests a file from your server, the server writes a line in its access log. Those lines are the only first-hand record of how crawlers actually behave on your site. Tools like Search Console give you summaries and samples. The logs give you every request.
What a log line contains
66.249.66.1 - - [14/Oct/2026:06:12:44 +0000] "GET /shop/decaf/?sort=price HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"That’s the IP address, date and time, the URL requested, the status code returned, the response size, and the user agent, which here identifies Googlebot Smartphone.
Why it matters
For a small site, Search Console’s Crawl Stats report is usually enough. Logs earn their keep on larger or troubled sites. Bean & Barrel Coffee notices new products take weeks to appear in Google. The logs show why: 70% of Googlebot’s requests in the last month went to filter and sort URLs like the one above, while some product pages weren’t requested at all. That’s a clear crawl budget leak, and now there’s evidence for exactly which parameters to block.
What to look for
- Where crawl activity goes. Which sections get the most requests, and which important pages get none.
- Status codes. How many requests hit redirects, 404s or 5xx errors.
- Crawl frequency. How often key pages are re-crawled after you update them.
- AI crawler activity. Whether GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot visit, and whether your firewall is returning 403 errors to them (see AI crawlers).
- Response times for crawler requests.
Verify the bot is real
Anyone can put “Googlebot” in a user agent string. To confirm a request really came from Google, run a reverse DNS lookup on the IP address: it should resolve to a host ending in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that host should return the same IP. Google also publishes its crawler IP ranges as JSON files. OpenAI and Perplexity publish IP ranges for their crawlers too.
Common mistakes
- Analyzing a few hours of logs. Look at at least two to four weeks.
- Forgetting the CDN. If you use Cloudflare or a similar service, many requests never reach your origin server, so the CDN’s logs are the ones you need.
- Trusting user agents without verification.
How to get logs
Many hosts let you download raw access logs from the control panel. Managed WordPress hosts sometimes keep only a few days, so export regularly. For a crawler’s-eye view without logs, a Site Audit crawl shows what a bot can reach by following links.