A page is missing from search results for one of a small set of reasons: it was never discovered, it cannot be crawled, it returns an error or a redirect, it carries a noindex instruction, another URL was chosen as the canonical version, or it was crawled and Google chose not to index it. Diagnose in that order — discovery, crawl, response, directives, canonical, judgment — because each step rules out a whole family of causes. Search Console's Page indexing report names the family for you with exact status strings; this guide maps each string to the check that confirms it and the fix that clears it, using our free SEO Checker, which fetches a URL the way a crawler does and reports the same signals.

Crawling is not indexing

Three separate things have to happen before a page can appear in results. Discovery means Google knows the URL exists, usually from a link or a sitemap. Crawling means Googlebot fetches it, which robots.txt and your server can prevent. Indexing means Google analyses the fetched page and stores it so it can be returned for a query. A failure at any stage produces the same symptom — the page is not in results — which is why guessing wastes time. The triage below isolates the stage first, then the cause.

Step 1: Read the exact status in Search Console

Open the Page indexing report (Indexing → Pages) and find your URL's status. Google uses precise strings, and each one points at a different step in this guide. We verified the following strings against Google's own Page indexing documentation while writing this guide:

Search Console status What it means Go to
Discovered – currently not indexed Google knows the URL but has not crawled it yet Step 2
Blocked by robots.txt The crawler was not allowed to fetch the page Step 3
Excluded by 'noindex' tag The page itself instructs Google not to index it Step 4
Page with redirect The URL redirects elsewhere, so this URL is not indexed Step 5
Not found (404) / Soft 404 The URL errors, or returns 200 with error-page content Step 5
Alternate page with proper canonical tag Google indexed the canonical URL instead — usually correct Step 6
Duplicate without user-selected canonical Google found near-duplicates and picked one itself Step 6
Crawled – currently not indexed The page was fetched and stored nowhere; a quality/relevance judgment Step 7

If the URL is not in the report at all, Google has not discovered it: link to it from a page that is indexed and list it in your sitemap, following How to Create an XML Sitemap.

Step 2: Confirm the page can be fetched at all

Before interpreting any status, fetch the URL yourself. In a terminal:

curl -sI https://example.com/definitely-not-a-real-page-xyz

Run against example.com while writing this guide, that command returned a real error page's first line:

HTTP/1.1 404 Not Found

That is the healthy behaviour for a URL that does not exist — and it proves the opposite case matters: if your missing page returns 404, 403, or a 5xx here, no indexing instruction on the page matters until the response is fixed. The full code-by-code breakdown is in HTTP Status Codes Explained for Troubleshooting. The SEO Checker reports the final status and every redirect hop in one pass, which is faster when you are working through a list of URLs.

Step 3: Check whether robots.txt blocks the crawl

If the status is "Blocked by robots.txt", fetch /robots.txt and look for a Disallow rule covering the URL's path. Remember the order of operations: a blocked page is never fetched, so Google cannot see anything on it — including a fix you already made. Remove or narrow the rule, fetch the page again to prove it loads, and only then expect the status to clear. The matching rules (longest rule wins, paths are case-sensitive) are covered in How to Configure robots.txt Correctly. Note also that a blocked URL can still be indexed as a bare link if other sites point to it, because indexing a URL and fetching it are different operations.

Step 4: Check for noindex — in the page and in the headers

A noindex can arrive three ways: a <meta name="robots" content="noindex"> tag in the HTML, an X-Robots-Tag: noindex HTTP header, or a CMS setting that emits one of those without you noticing. This is a real capture, not a reconstruction: while preparing this guide we fetched an outdated MDN address with curl, then ran the site's own SeoAnalyzer class — the analysis code behind the SEO Checker — on that real response. It reported:

Signal Captured value
Final status 404
Meta robots noindex, nofollow
Canonical https://developer.mozilla.org/en-US/404
High-priority issue raised "Page is blocked from indexing (noindex)" — evidence: "Robots directive: noindex, nofollow"

MDN deliberately marks its error page noindex — correct behaviour, and a perfect specimen of the mechanism. On your own missing page the same finding is a bug: a template, a staging setting, or a "discourage search engines" toggle left on. Remove the directive at its source (editing the rendered HTML does nothing if the CMS re-emits it), then re-fetch the page and confirm the directive is gone before requesting indexing.

The same method applied to https://example.com — real response fetched with curl, analysed by the same SeoAnalyzer class — shows the healthy contrast: status 200, no meta robots directive at all, and no X-Robots-Tag header — nothing on that page tells a search engine to stay away.

Step 5: Check the response the crawler actually gets

Two statuses hide here. A hard error (404, 403, 5xx) means there is no page to index. A soft 404 is worse because it looks healthy: the server returns 200, but the content is an error message, an empty category, or a "not found" template. Google classifies it as an error from the content, not the code. Check what a visitor sees, not just the status line. If the URL redirects, follow the chain to its end: Google indexes the destination, not your URL, and lists yours under "Page with redirect" — correct behaviour unless the redirect itself is the mistake, in which case How to Fix Redirect Chains and Redirect Loops covers the repair.

Step 6: Check which URL Google chose as canonical

"Alternate page with proper canonical tag" means your page declares a different canonical URL and Google agreed — usually nothing to fix. "Duplicate without user-selected canonical" means the duplicate signals were left to Google to resolve. Fetch the page and read its canonical tag: if it points at another URL, that URL is the one that can rank, and this page will stay excluded however often you request indexing. If there is no canonical tag, add a self-referencing one and make sure your sitemap and internal links agree with it. Canonical Tags Explained covers the conflicting-signal cases in depth.

Step 7: Crawled, fetched fine — and still not indexed

If every mechanical check passes and the status is "Crawled – currently not indexed", there is no switch to flip. Google fetched the page and decided it adds too little to store: thin content, a near-duplicate of a page already indexed, or a page nothing links to. The fixes are editorial, not technical — merge it into the stronger page it duplicates, expand it with information the indexed alternative lacks, or link to it from relevant indexed pages. Requesting indexing repeatedly does not change the judgment; changing the page can. A full sweep of these checks across a site, in a fixed order, is the audit routine in How to Perform a Website SEO Audit Step by Step.

Frequently asked questions

How long after a fix until the page is indexed?

There is no fixed interval. After you remove the blocking cause, Google has to re-crawl the page before any status changes, and re-crawling happens on Google's schedule. Use the URL Inspection tool's "Request indexing" once the live page verifies clean — it places the URL in a crawl queue but does not guarantee a date or a result.

My page is indexed but does not rank. Is that an indexing problem?

No. Indexing means the page is stored and eligible to appear; ranking is the separate question of where. If the URL Inspection tool says "URL is on Google", your problem is relevance, content, or links — not anything in this guide.

Can robots.txt remove a page that is already indexed?

Not by itself. Blocking the crawl stops Google from fetching the page again, so an indexed blocked page can linger as a link-only result. To remove it, unblock it, serve a noindex, and let Google re-crawl — counter-intuitive, but the crawler has to be allowed in to read the instruction that removes the page.

Why does Search Console show more excluded pages than my site has pages?

The report counts URLs Google has encountered, including parameter variants, old URLs from before a redesign, and URLs other sites invented by linking wrongly. Work by status family and by the URLs you recognise as real pages first; large counts of "Alternate page" or redirect entries are often just your URL variants being resolved correctly.

Should every page on my site be indexed?

No. Thank-you pages, internal search results, login screens, and print views are correctly excluded, often by design. An indexing problem is a page you want in results being excluded for a reason you did not intend.

References

Related reading: duplicate metadata, structured data with JSON-LD.