The search-bot delivery route
A search system can discover and crawl a page without placing it in an index. Imagine a library intake desk: finding a book, requesting it, reading it and cataloguing it are separate jobs. Only catalogued candidates can be considered for a particular search; being catalogued does not guarantee a ranking.
Text equivalent: a system discovers a URL, may request it, processes a successful response, may index it, then may consider that indexed page for a specific query.
Three different checkpoints
Crawled
The bot requested the URL. The response might still be an error, duplicate or blocked from indexing.
Indexed
The system chose a version for its searchable catalogue. This is not a visibility promise.
Ranked
The indexed candidate was selected and ordered for a particular search, place, time and context.
How discovery and controls interact
Search engines can find pages through a descriptive internal link or an XML sitemap. A sitemap is a discovery aid, not an indexing guarantee. Robots.txt guides crawler access; it is not a dependable way to remove a URL already indexed. A noindex directive generally must remain crawlable so a crawler can see it. A canonical URL is a preferred-version hint, not a guaranteed command.
The crawler cannot fetch the page, so it may not see the noindex instruction.
Fetch succeeded, but the system may choose a different canonical or not index it.
No internal route points to it; a sitemap can aid discovery, but cannot guarantee inclusion.
The address reports that the resource is missing and should not serve an indexable page.
Google-specific reference: compare the official guidance for robots.txt, noindex and canonicalisation. Other search engines can implement their own behaviour.
Related dictionary terms
Level checkpoint / Mission review
Check what you learned
Answer all three questions correctly, then complete the lesson. Retry any answer whenever you need.
Complete all three questions to unlock lesson completion.
Checkpoint cleared
Lesson complete
You can revisit the check above at any time.
Next lesson: Performance and Core Web Vitals →