How to fix crawl errors in Google Search Console

A crawl error means a search engine tried to fetch a page and something went wrong: the page was missing, the server failed, a redirect went nowhere, or something blocked the request. Some crawl errors cost you traffic. Many do not matter at all. The skill is knowing which is which and fixing the important ones first.

This guide explains where to find crawl errors, what each one means and how to fix it.

Where to find crawl errors

Google Search Console is the main source, because it shows what Google itself ran into.

• Pages report (Indexing > Pages): lists every reason pages are not indexed, with example URLs. Most crawl errors appear here.
• Crawl stats (Settings > Crawl stats): shows how often Googlebot visits, the response codes it gets and whether your server had availability problems.
• URL Inspection: checks a single URL, shows the last crawl, and lets you test the live page and request indexing.

A site crawler gives you the other half. It follows every link on your site and finds broken links and redirect chains before Google does, including ones on pages Google rarely visits. RankThat’s Site Audit crawls as RankThatBot and lists each issue with why it matters and how to fix it.

Search Console tells you what Google saw. A crawler tells you what is wrong right now. Use both.

First: decide what matters

Before fixing anything, ask of each error:

1. Is this a page that should exist and be indexed? An error on your main service page is urgent. An error on an old tag archive or a URL someone mistyped is not.
2. Do other pages or sites link to it? Broken URLs with backlinks or internal links waste value and send visitors to a dead end.
3. Did it get traffic before? Check Search Console’s Performance report for the URL.

A 404 on a page that never existed is normal. A 404 on a page that ranked for “emergency plumber austin” last month is an emergency.

Not found (404)

What it means: the server says the page does not exist.

When it is fine: the page was deleted on purpose and has no useful replacement, has no important links pointing at it and had no traffic. Google will drop it from the index over time. Returning 404 (or 410 for “gone for good”) is the correct response.

When to fix it:

• The page moved. Add a 301 redirect from the old URL to the new one.
• The page was deleted but there is a close replacement. Redirect to the most relevant page, such as the replacement product or the parent category. Do not redirect everything to the homepage; Google often treats that as a soft 404.
• Your own site links to it. Update the internal links to point straight at the right page.
• It was a typo or a mangled URL linked from another site. Redirect if the link is valuable.

The free redirect checker confirms the redirect returns a single 301 to a working page.

Soft 404

What it means: the page returns a 200 (success) code, but Google thinks it looks like an error page: empty, “no products found”, or a “sorry, page not found” message served with a 200.

Common causes:

• Empty category or search results pages.
• Out-of-stock product pages with almost no content.
• A theme that shows a “not found” message without sending a 404 code.
• Many old URLs redirected to the homepage.

Fix: if the page should not exist, make it return a real 404 or 410. If it should exist, add real content. If it is a redirect target problem, redirect to a more relevant page.

Server error (5xx)

What it means: the server failed to respond properly. Googlebot slows down crawling when it sees many server errors, and pages that keep failing can drop out of the index.

Common causes:

• Overloaded or underpowered hosting, especially during traffic spikes or heavy crawling.
• A broken plugin or theme update causing PHP errors.
• Database connection problems.
• Timeouts on slow pages, such as huge category pages or slow external calls.

Fix:

1. Check whether the error happens now. Open the URL and use URL Inspection’s live test.
2. If it was temporary, check Crawl stats for the pattern. Repeated spikes point to hosting limits.
3. Check your server or hosting error logs for the exact error.
4. Roll back recent plugin, theme or code changes if errors started after them.
5. Talk to your host about resources if the site regularly runs out.

Server errors on important pages deserve attention the same day.

Redirect error

What it means: Google followed a redirect and it failed: a loop (A goes to B goes back to A), a chain that is too long, a redirect to an empty or bad URL, or a URL that grew too long.

Common causes:

• HTTP to HTTPS and non-www to www rules fighting each other.
• Redirect plugins and server rules both redirecting the same URL.
• Several site migrations, each adding a hop.

Fix: every old URL should go to its final destination in one hop. Check each one with the redirect checker, remove duplicate rules, and update internal links so they point at final URLs instead of redirects.

Blocked by robots.txt

What it means: your robots.txt file tells Googlebot not to crawl the URL.

When it is fine: for admin areas, internal search results, cart and checkout pages, and endless filter combinations.

When to fix it: when an important page or folder is blocked. Common mistakes include Disallow: / left over from a staging site, blocking CSS or JavaScript folders, and broad patterns that catch more than intended.

Fix: edit robots.txt to allow the URLs, then test them with the free robots.txt checker.

Note: robots.txt stops crawling, not indexing. A blocked URL can still appear in results, without a description, if other sites link to it. To keep a page out of the index, allow crawling and use noindex instead.

Excluded by ‘noindex’ tag

What it means: the page tells search engines not to index it, through a meta robots tag or an X-Robots-Tag header.

Fix: check the list for pages you want indexed. Common causes are:

• WordPress’s “Discourage search engines” setting left on after launch.
• An SEO plugin setting noindex on a whole content type, such as all products or all categories.
• A page builder or landing page template with noindex on by default.

Remove the noindex, then use URL Inspection to request indexing for the most important pages.

Blocked due to access forbidden (403) or unauthorized request (401)

What it means: the server refused Googlebot. A 401 means a login is needed; a 403 means access is denied.

Common causes:

• Firewall, CDN or security plugin rules blocking crawlers.
• Pages behind a login or member area.
• Country or IP blocks that catch Google’s crawlers.

Fix: for public pages, adjust the firewall rule so verified search engine crawlers can reach them. Your firewall or CDN can usually verify real Googlebot traffic. For pages that should be private, this is fine, but remove them from your sitemap and internal links. The same kind of blocking often hits AI crawlers too; the free AI crawler checker shows whether ChatGPT, Perplexity and others can reach you.

Not errors, but often confused with them

Some statuses in the Pages report are not crawl failures but are worth understanding:

• Discovered, currently not indexed: Google knows the URL but has not crawled it yet. Common on new or large sites. Improve internal links to the page and check the server is fast enough.
• Crawled, currently not indexed: Google fetched the page and chose not to index it. Usually a content quality or duplication signal. Improve, merge or remove the page.
• Duplicate without user-selected canonical: Google found similar pages and picked one. Add a canonical tag pointing to the version you want.
• Alternate page with proper canonical tag: working as intended.
• Page with redirect: the URL redirects. Fine, as long as it is not in your sitemap or internal links.

After fixing: validate and monitor

1. Click “Validate fix” on the issue in the Pages report. Google re-checks the affected URLs over the following days or weeks and reports whether they pass.
2. Update your XML sitemap so it lists only live, indexable URLs, then resubmit it.
3. Request indexing with URL Inspection for a few of the most important fixed pages.
4. Re-crawl the site to confirm nothing new broke.
5. Keep watching. Crawl errors come back after redesigns, plugin updates, migrations and product changes.

A quick priority order

When you open Search Console and see hundreds of issues:

1. Server errors and robots.txt blocks on important pages.
2. Noindex on pages that should rank.
3. 404s on pages with traffic, backlinks or internal links.
4. Redirect errors and chains on main navigation paths.
5. Soft 404s on product and category pages.
6. Everything else, in batches.

On WordPress sites, RankThat can draft redirects for broken pages and fix links that point at redirects, then apply them through the free WordPress Bridge plugin after you review them. Each change can be undone, and the live page is re-checked after the fix. For the wider picture, see how to do an SEO audit.