A generative engine can't cite what it never retrieved. Structured, accurate content behind a crawl block, a JavaScript wall, or a chunking boundary that splits the fact in half is functionally invisible — no different in outcome from content that doesn't exist.
By the end of this lesson, you should be able to inspect a page's retrieval accessibility, assign evidence-based readiness scores, identify which barriers block crawlers versus which merely degrade what gets retrieved, and explain why a human-readable page is not the same claim as a machine-retrievable one.
Check robots.txt, authentication walls, and bot-blocking rules against the specific crawlers behind the engines that matter — a block aimed at scrapers can just as easily block the retrieval pipeline you want to reach.
Confirm the facts are present in server-rendered HTML — a retrieval pipeline that doesn't execute client-side scripts sees an empty shell where a human sees a fully rendered page.
Write claims so each one is self-contained within a plausible chunk boundary — a fact that only makes sense combined with a sentence three paragraphs earlier may get retrieved without the context that made it true.
Publish an llms.txt or equivalent canonical reference — a dedicated, structured entry point removes the need for a retrieval pipeline to reconstruct your ground truth from scattered prose.
A company's leadership page is fully server-rendered and passes a crawl check with no robots.txt restrictions. The founder's title and start date appear in a sentence that begins two paragraphs earlier with "As mentioned above," relying on unstated prior context. The domain has no llms.txt file.
Crawl access earns a 2 — this was actually tested against robots.txt, not assumed. Render access earns 1 at best: server-rendering is confirmed, but nothing here tests what a specific retrieval pipeline's fetcher actually receives. Chunk-level clarity is 0 — a fact that depends on "as mentioned above" is a documented failure mode, not a passing score; a chunk containing only that sentence is retrievable but meaningless on its own. Direct access points is 0: no llms.txt exists. The appropriate next step is rewriting the leadership claim to be self-contained within one paragraph — not adding an llms.txt file before the underlying prose can survive being chunked.
STOP → DECOUPLE