Robots.txt Checker

A robots.txt tester is the quickest way to answer “can Google crawl this page?” without waiting for Search Console to tell you weeks later. This guide explains how this robots.txt checker decides Allowed or Blocked, shows how real rules behave on real paths, gives safe starting files for WordPress and WooCommerce, and lists the mistakes that most often hide pages from search.

What the checker reads and tests

Type a site or a full page address. The checker then:

1. Fetches robots.txt from the root of that exact host. https://www.yoursite.com/robots.txt and https://yoursite.com/robots.txt are separate files as far as crawlers are concerned, and so is https://shop.yoursite.com/robots.txt. Test the host your visitors actually land on.
2. Shows you the file as crawlers receive it, plus how many Sitemap lines it lists.
3. Tests one path for Googlebot (Google Search), Bingbot (Bing and Copilot) and ten AI crawlers. If you typed a page address, that path is tested. If you typed only the site, the homepage is tested.

It applies the same matching rules Google uses:

• Each crawler follows the one group whose User-agent names it. If no group names it, it follows the User-agent: * group.
• Within that group, the longest matching rule wins.
• If an Allow and a Disallow rule are the same length, Allow wins.
• * matches any run of characters and $ marks the end of the address.
• An empty Disallow: line allows everything.

Google’s robots.txt documentation is the official reference for these rules.

How rules behave on real paths

Most robots.txt surprises come from prefix matching: a rule blocks every path that starts with it, not just the exact folder you had in mind.

When a result surprises you, find the longest rule in the crawler’s group that matches the start of the path. That’s the rule deciding the outcome.

Reading your results

If the checker says there is no robots.txt but you know there is one, open it in a browser. A file that returns a server error is worse than a missing one, because Google treats a server error there as a reason to pause crawling the site.

Safe starting files

A typical WordPress site:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://coastlineplumbing.com/sitemap_index.xml

A WooCommerce store:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*add-to-cart=
Disallow: /*orderby=

Sitemap: https://beanandbarrelcoffee.com/sitemap_index.xml

The last two Disallow lines stop crawlers spending time on endless sorted and add-to-cart versions of the same category pages. On stores with filters, this is where most wasted crawl budget goes; the glossary entry on faceted navigation explains the bigger picture. Use your real sitemap address: Yoast and Rank Math use sitemap_index.xml, while WordPress on its own uses wp-sitemap.xml.

Blocked from crawling is not the same as out of Google

A Disallow rule stops Google reading the page. It doesn’t stop Google listing the address if other pages link to it. That’s how sites end up with results that say “No information is available for this page”, and with the Search Console status “Indexed, though blocked by robots.txt”.

To take a page out of Google properly:

1. Remove the Disallow rule so Google can read the page again.
2. Add a noindex tag to the page.
3. Wait until the page drops out of Google (use Search Console’s URL Inspection to check).
4. Only then, if you still want to save crawling, add the Disallow back.

Doing step 4 first is the most common reason a noindex “doesn’t work”.

Common mistakes

• Launching with the staging file. A development site’s Disallow: / copied to the live site blocks everything. Test the homepage after every launch or migration.
• Blocking theme files. Rules like Disallow: /wp-content/ or Disallow: /wp-includes/ stop Google loading your CSS and JavaScript, so it can’t see the page the way visitors do.
• Using robots.txt to hide private pages. The file is public. Anyone can read it, so listing /staff-portal/ in it advertises the address. Put private pages behind a sign-in.
• Relying on Crawl-delay. Google ignores it. Bing respects it, which can slow Bing’s crawling more than you meant to.
• Editing a file nobody serves. On WordPress, a physical robots.txt in the site root replaces the one your SEO plugin generates. If your edits don’t show up in the checker, look for that file.
• Forgetting other hosts. A www version, a shop. subdomain or an old http setup each answer with their own robots.txt.

After any change, paste a few important page addresses into the checker one by one: your homepage, a top service or product page, and a blog post. It takes a minute and catches nearly every robots.txt accident.

When one path isn’t enough

This tool tests one page at a time. The Site Audit tests every page it finds against your robots.txt and flags important pages blocked by mistake, a file that blocks the whole site, a robots.txt with syntax errors, a missing Sitemap line, and the WordPress “discourage search engines” setting left on after launch. On WordPress, the free Bridge plugin can also set AI crawler access in robots.txt for you, with one-click undo. See how the WordPress fixes work, and for the wider picture of pages Google can’t reach, our guide to fixing crawl errors and the glossary entry on crawling.