Every well-behaved crawler checks one file before it fetches anything else on your site: https://www.example.com/robots.txt. The file holds simple rules grouped by user agent. Each group says which paths that crawler may not request (Disallow) and any exceptions (Allow). It can also point to your XML sitemap.
The rules are voluntary. Googlebot, Bingbot and the major AI crawlers follow them. Scrapers and bad bots often don’t, so robots.txt is not a security tool.
Why it matters
One wrong line can hide an entire site. A common story: a developer builds the new Coastline Plumbing site on a staging server with Disallow: / to keep it out of Google, then copies everything to the live domain, file included. Within a few weeks rankings slide as Google stops refreshing pages. The fix takes ten seconds once someone notices.
Robots.txt is also where you decide which AI crawlers can read your content, for example allowing OAI-SearchBot so ChatGPT search can cite you while blocking a training crawler.
Example
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap_index.xmlThis lets all crawlers in except for admin, cart and checkout pages, blocks OpenAI’s training crawler entirely, and lists the sitemap.
Robots.txt does not remove pages from Google
This is the most misunderstood part. Blocking a URL stops Google from crawling it, but if other pages link to it, Google can still index the bare URL and show it with no description. To keep a page out of results, let Google crawl it and add a noindex tag. Blocking the page in robots.txt would stop Google from ever seeing that tag. Google also stopped supporting noindex rules inside robots.txt in 2019.
Common mistakes
- Blocking CSS and JavaScript folders, which stops Google rendering the page properly.
- Forgetting that paths are case-sensitive:
/Blog/and/blog/are different. - Using
Crawl-delayfor Google. Googlebot ignores it. - Putting the file anywhere other than the root of the host. Each subdomain needs its own.
How to check
Paste your domain into the free Robots.txt Checker to see which crawlers are allowed or blocked and test individual URLs against your rules. Search Console’s robots.txt report also shows the version Google last fetched and any parsing errors.