01 · Sitemap
Which canonical pages are listed?
Include public URLs on the same host, keep dates truthful, and remove retired routes. The protocol describes a list of URLs and optional metadata; its optional values are hints, not commands.
Foundation 03 · Discovery boundaries
List the route.
Do not script the reader.
A sitemap, robots file, route list, or `llms.txt` file makes a publisher’s public structure easier to inspect. Those records describe available material and preferences; they do not compel any crawler, search engine, application, or model to access, index, rank, retrieve, cite, or interpret it.
Learning boundary. This lesson is not crawler-control, search, legal, security, or access-control advice. `robots.txt` expresses crawler rules; it is not authorization. A sitemap lists URLs; it is not an indexing or ranking command. `llms.txt` is a publisher convention, not a standards-track requirement for model behavior.
Learning objective
Publish a route inventory that is accurate, current, and modest about its effects.
By the end of this lesson, you should be able to distinguish a canonical page, sitemap entry, robots rule, descriptive route list, access-control requirement, and recheck event without using any file as a promise about downstream behavior.
02 · Robots
Which crawl preferences are declared?
State patterns carefully and remember that exclusion rules are not authentication or authorization. Sensitive material needs an actual access-control mechanism, not an obscured path.
03 · Route list
What additional context helps a reader or system choose?
Use a descriptive list such as `llms.txt` to name the page, purpose, scope, source boundary, and related material. Keep it current and do not claim it changes a model’s behavior.
04 · Recheck
When must the discovery record change?
Update the route inventory when a public page is added, moved, materially revised, held, or retired. A stale discovery file misrepresents the public record even when every linked page still loads.
Practice lab
Release one new public guide without overstating discovery.
A team publishes a technical guide and wants “AI and search engines to find it.” The page has a canonical URL, a dated author line, two primary sources, and a correction route. The team wants to add it to a sitemap, `robots.txt`, and an `llms.txt` list.
- Define which records should name the new route and what truthful metadata belongs in each one.
- Write a route-list description that states the guide’s reader job and boundary without saying a model will use it.
- Identify which request needs access control rather than `robots.txt` if the guide were not actually public.
- Define the event that requires all three discovery records to be rechecked or changed.
Source shelf
Read the protocol boundaries before treating them as a growth mechanism.
- Sitemaps.org: Sitemap XML formatRead the required URL-list structure, date requirements, and guidance that optional metadata is only a hint to crawlers rather than a position-control mechanism.
- IETF RFC 9309: Robots Exclusion ProtocolRead the protocol’s distinction between crawler-requested rules and access authorization. Do not use a disallow line as a substitute for protecting private material.
Complete the pathway