This AI crawler checker answers one narrow question well: does your robots.txt let each AI bot read a given page? Below is what each of the ten bots in the results does, what “Blocked” really costs you for each one, ready-to-paste robots.txt rules for the three policies most sites choose, and the reasons a site can pass this check and still never show up in ChatGPT or Perplexity.
What the checker actually tests
When you enter a site, the checker:
- Fetches robots.txt from the root of the address you typed.
- For each bot, finds the group of rules that applies. A group that names the bot’s user agent wins over the catch-all
User-agent: *group. - Tests the homepage (or the page path you typed) with Google’s matching rules: the longest matching rule wins, and Allow wins a tie.
If there is no robots.txt, every bot is allowed. If your server answers /robots.txt with an HTML page (some themes and hosts do this instead of a proper “not found”), the checker treats the file as missing, because that’s what a crawler would conclude too.
To test a deeper page, such as yoursite.com/blog/, type the full address. A rule like Disallow: /blog/ won’t show up when only the homepage is tested.
The ten bots, and what blocking each one does
| Bot | Run by | What it fetches pages for | If it’s blocked |
|---|---|---|---|
| GPTBot | OpenAI | Collecting content to train future models | You opt out of OpenAI training. ChatGPT search can still cite you |
| OAI-SearchBot | OpenAI | ChatGPT’s search results | Your pages can’t appear as sources in ChatGPT search |
| ChatGPT-User | OpenAI | Fetching a page when a ChatGPT user asks it to | ChatGPT can’t open your page mid-conversation |
| ClaudeBot | Anthropic | Anthropic’s main crawler, used for model training | You opt out of Claude training data |
| Claude-SearchBot | Anthropic | Claude’s search results | Claude is less able to find and cite your pages |
| PerplexityBot | Perplexity | Perplexity’s answer index | Perplexity is unlikely to cite you |
| Google-Extended | A permission token, not a separate crawler. Googlebot fetches; this says whether Gemini models may use the content | You opt out of Gemini training and grounding. Google Search is not affected | |
| Applebot-Extended | Apple | A permission token for training Apple’s AI models | You opt out of Apple model training. Applebot still crawls for Siri and Spotlight |
| CCBot | Common Crawl | A public web archive that many AI models are trained on | You leave future Common Crawl snapshots |
| Bytespider | ByteDance | Crawling for ByteDance and TikTok AI products | You opt out of ByteDance’s crawler |
The bots fall into two groups, and you can treat them differently. The ones that fetch pages to answer a question right now decide if you can be recommended and cited. The ones that collect pages for training decide if your words help build future models. Blocking the second group is a fair business choice. Blocking the first group by accident is the expensive mistake this check is designed to catch.
Note that RankThat’s Website SEO Checker marks its “AI crawler access” check as a Fix if any of OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot or Claude-SearchBot is blocked.
Three robots.txt policies, ready to paste
Pick the one that matches your view, add it to your robots.txt, and keep your existing rules for search engines.
Policy 1: allow every AI bot. You don’t need to add anything. With no AI-specific groups, they all follow your User-agent: * rules. Most local service businesses should be here: you want every assistant to know your services, area and phone number.
Policy 2: allow answer bots, block training bots.
User-agent: GPTBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpSeveral User-agent lines in a row share the rules that follow them, so this blocks the five training bots in one group. OAI-SearchBot, ChatGPT-User, PerplexityBot and both Claude bots have no group of their own, so they use the * rules and can read the site.
Policy 3: block all AI bots. Add OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot to the group above. Be clear about the trade: you are choosing not to be cited by AI assistants.
Worked example: Bean & Barrel Coffee
Bean & Barrel Coffee sells beans online and runs two cafés. Their checker results showed OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Claude-SearchBot all blocked. Their robots.txt had this, pasted from a “block AI scrapers” blog post a year earlier:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /The owner’s real worry was training, not being recommended. They deleted the four answer-bot lines, thought separately about ClaudeBot and decided to allow it too, and kept GPTBot blocked. A rerun showed only GPTBot blocked, which was the policy they actually wanted. The next question is whether assistants now mention them, which you can track with AI Visibility: it asks ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews questions about your services and area and records who gets named.
Common mistakes
- Giving a bot its own group and forgetting it skips the
*rules. Once a bot has a named group, it ignores the catch-all group entirely. If you addUser-agent: PerplexityBotwithDisallow: /checkout/, it no longer sees your other*rules. Copy any rules it still needs into its group. - Stacking agents by accident. Two
User-agentlines with nothing between them share the next set of rules. Put a blank line and a rule between groups you mean to keep separate. - Blocking Googlebot to stop Gemini. That removes you from Google Search. Use
Google-Extendedinstead. - Editing the wrong file. On WordPress, robots.txt is often generated on the fly. If a physical
robots.txtfile exists in the site root, it overrides the generated one, and edits in your SEO plugin won’t show. The robots.txt checker shows the exact text crawlers receive.
If robots.txt allows the bots and you’re still never cited, check your firewall. CDN and security settings that block “AI bots” act before robots.txt is ever read, and this checker can’t see them.
Passing the check is step one
Allowing a crawler makes you eligible. It doesn’t make you the answer. Assistants cite pages that state facts plainly (who you are, where, what it costs, how long it takes), that other sites mention, and that load as real HTML rather than scripts. Our guide to getting recommended by ChatGPT covers the rest, and the glossary entries on AI crawlers and LLM citations explain the terms.
The full Site Audit checks AI crawler access and llms.txt as part of its 100+ checks, and on WordPress the free Bridge plugin can set AI crawler access in robots.txt for you, with undo. If you want a short guide for assistants as well, the llms.txt generator drafts one from your homepage.