Noindex

Noindex is the reliable way to keep a page out of search results. Unlike a canonical tag, which is a hint, Google obeys noindex once it has crawled the page and seen the instruction. The page drops out of the index, usually within days to a few weeks depending on how often Google visits it.

Two ways to set it

As a meta tag in the page’s <head>:

<meta name="robots" content="noindex, follow">

Or as an HTTP response header, which also works for PDFs and other non-HTML files:

X-Robots-Tag: noindex

The follow part is optional and is the default anyway. It lets crawlers keep following links on the page.

When to use it

Good candidates are pages that serve visitors but have no business ranking: thank-you pages after a form, internal search results, cart and account pages, thin tag archives, and staging copies. Brightside Dental, for example, might noindex the “appointment confirmed” page so it never shows up when someone searches the practice name.

Pages with weak or duplicate content are a judgment call. Sometimes the better move is to improve the page or merge it, rather than hiding thin content.

The robots.txt trap

If you block a page in robots.txt, Google can’t crawl it, so it never sees the noindex tag. The URL can stay in the index, often shown with no description. To remove a page, allow crawling and use noindex. Once it has dropped out, you can block it if you want to save crawl requests.

Common mistakes

• Launching with noindex still on. WordPress has a “Discourage search engines from indexing this site” checkbox under Settings, Reading. Ticked on a live site, it adds noindex everywhere. This is one of the most common reasons a new site never appears in Google.
• Noindexing pages you want to rank by applying a plugin setting to a whole post type.
• Combining noindex with a canonical pointing elsewhere. Pick one signal.
• Expecting instant removal. For something urgent, use the Removals tool in Search Console as a temporary measure while noindex takes effect.

How to check

URL Inspection in Search Console reports “Excluded by ‘noindex’ tag” when it finds one. A site crawl with Site Audit lists every noindexed URL, which makes it easy to spot important pages that were hidden by mistake, and to confirm whether they ever reached the index.