When anything requests a page from your server, it sends a User-Agent header saying what it is. A visitor on an iPhone sends a string naming Safari and iOS. Google’s crawler sends a string containing “Googlebot”. Servers use this to log visits, and sometimes to decide what to send back.
In SEO, user agents matter in two places: robots.txt, where rules are grouped by user agent, and server logs, where they tell you which crawler made each request.
Common crawler user agents
| Product token | Who runs it | Purpose |
|---|---|---|
| Googlebot | Crawling for Google Search (smartphone and desktop) | |
| Google-Extended | Not a crawler; a robots.txt token that controls use of content for Gemini training | |
| Bingbot | Microsoft | Bing search, which also feeds Copilot |
| GPTBot | OpenAI | Collecting training data |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ClaudeBot | Anthropic | Collecting training data |
| PerplexityBot | Perplexity | Perplexity search index |
| RankThatBot | RankThat | Site audits run by RankThat users |
The full string is longer. Googlebot Smartphone, for example, includes a recent Chrome version number plus (compatible; Googlebot/2.1; +http://www.google.com/bot.html).
In robots.txt
The robots.txt User-agent line matches the short product token, not the full string:
User-agent: Googlebot
Allow: /
User-agent: GPTBot
Disallow: /private-guides/A crawler follows the most specific group that matches it and ignores the rest, so a group for GPTBot replaces the * rules for that bot rather than adding to them.
Why it matters
Coastline Plumbing’s host adds a firewall that blocks “unknown bots”. Without anyone noticing, it starts returning 403 errors to OAI-SearchBot and PerplexityBot, so ChatGPT and Perplexity stop being able to read the site. Checking which user agents get through is now part of routine maintenance, especially for AI crawlers.
Common mistakes
- Trusting the string. User agents are easy to fake. Verify important bots by reverse DNS or published IP ranges, as covered in log file analysis.
- Serving different content by user agent. Showing Googlebot something different from visitors is cloaking and breaks Google’s spam policies.
- Typos in robots.txt tokens, which make the rule apply to nobody.
RankThat’s own crawler identifies itself honestly as RankThatBot and never pretends to be a browser, so site owners can allowlist it. Use the free AI Crawler Checker to see which AI user agents your site currently allows.