Search engines have always sent crawlers to read websites. AI companies now do too, and their bots do different jobs. Some collect pages to train future models. Some fetch pages to answer a question someone is asking right now. Some do both. Deciding which ones to let in has become part of running a website.
The main AI crawlers
| User agent | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content for training models |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page when a user’s chat needs it |
| ClaudeBot | Anthropic | Collects content for training |
| Claude-SearchBot | Anthropic | Indexes pages for Claude’s search results |
| Claude-User | Anthropic | Fetches a page when a user asks |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers |
| Perplexity-User | Perplexity | Fetches pages for a user’s question |
| Google-Extended | Not a crawler; a robots.txt token controlling use in Gemini training and grounding | |
| Applebot-Extended | Apple | A token controlling use in Apple’s AI training |
| CCBot | Common Crawl | Open web archive widely used to train models |
The companies document these on their own sites, and the list changes, so check current documentation before writing rules.
Training versus search
This distinction drives most decisions. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) keeps your content out of future model training. Blocking search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) means those assistants can’t read or cite your pages when people ask them questions, which reduces your AI visibility.
Many businesses choose to allow search bots and make a separate decision about training bots. Blocking Google-Extended doesn’t remove you from Google Search or AI Overviews, which use Googlebot.
Example robots.txt
# Allow AI search and user fetches
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /Why it matters for a real business
Coastline Plumbing’s host enabled a “block AI bots” toggle in its firewall. Nobody noticed until the owner asked ChatGPT about local water heater repair and found the answer cited competitors and old directory listings instead of Coastline’s own pages. The firewall had been returning errors to every AI search bot for months.
Common mistakes
- Blocking everything with one firewall setting, including the search bots you want
- Rules in robots.txt that conflict with firewall rules
- Assuming user agents are genuine. Verify important bots with published IP ranges where available
- Relying on llms.txt to grant access. It doesn’t
How to check
The free AI Crawler Checker shows which AI bots your robots.txt allows or blocks. RankThat’s Site Audit flags blocked AI crawlers, and on WordPress the Bridge plugin can update AI crawler access in robots.txt as a reviewable fix.