<!-- llms-full grammar: mintlify — per page: '# Title' / 'Source: <url>' / blank / body -->

# The attribute rubric
Source: https://llms-explorer.com/reference/attributes/

# The attribute rubric

Every attribute an llms file is judged on, with bars and severities.

Every finding `/ldo` raises names one attribute below. An attribute has: the file kinds it
applies to, how it is measured (deterministic check, model judgment, or a live agent test),
the bar, and the severity of a miss. "Index" = `llms.txt`; "full" = `llms-full.txt` (and
`llms-small.txt`); "facts" = `llms-facts.txt` (a hub extension); "family" = a nested index
that links other indexes.

1. Identity and shape (I1–I6)
2. Navigation (N1–N7)
3. Descriptions (D1–D6)
4. Content fidelity (C1–C7)
5. Provenance and trust (P1–P6)
6. Size and budget (S1–S6)
7. Retrieval readiness (R1–R7)
8. Family / nesting (F1–F6)
9. Hygiene and serving (H1–H8)
10. The three kinds side by side

## 1. Identity and shape

| Id | Attribute | Applies | Measure | Bar | Miss |
|---|---|---|---|---|---|
| I1 | Exactly one H1 naming the site/product (not a page) | index, facts, family | deterministic | 1 H1; title = product/site | High |
| I2 | Blockquote summary immediately after H1, 1–3 sentences, self-contained | index, family | deterministic + judgment | present; says what the thing is and who it is for | Medium |
| I3 | Free-form info before the first H2 (how to read this file, versions, languages) | index, family | judgment | only if it changes how a reader should use the links | Low |
| I4 | Sections are H2 only; each is a link list; no prose after the first H2 except list notes | index, family | deterministic | no H3+, no stray paragraphs | Medium |
| I5 | Link entries match `- [name](url)` + optional `: notes` | index, family | deterministic | 100% of list items | High if <90%, else Medium |
| I6 | Kind is unambiguous from the first 20 lines (index vs full vs facts) — a full file is never served as an index | all | deterministic | grammar detected with one candidate | High |

## 2. Navigation

| Id | Attribute | Applies | Measure | Bar | Miss |
|---|---|---|---|---|---|
| N1 | Two hops: index → page (or family → index → page); no index links a bare directory of more indexes | index, family | deterministic (link targets) | ≤2 hops to any page | High |
| N2 | Section design mirrors how users ask (task/topic groups), not the URL tree or an alphabet | index | judgment | ≥80% of sections are task/topic named | Medium |
| N3 | Ordering by expected query frequency: quickstart/auth/reference/errors first; the first 20% of links should answer 80% of questions | index | judgment + agent test | hot pages in the first section | Medium |
| N4 | `## Optional` holds only skippable material (changelog, legal, old posts, appendices); it is the last section | index | deterministic + judgment | last; no reference/pricing inside | Medium |
| N5 | Every page the source publishes that a reader would need is reachable (coverage) | index | deterministic vs source page list | ≥95% of `reference`+`guide` pages linked | High if <80% |
| N6 | No dead ends: each link resolves (200, markdown or `.md` twin), no redirect to an HTML app shell | index, family | deterministic (`--check-links`) | 0 dead links | High |
| N7 | Cross-cutting material (errors, auth, glossary) linked once, not once per section | index, family | judgment | no duplicate targets | Low |

## 3. Descriptions

| Id | Attribute | Applies | Measure | Bar | Miss |
|---|---|---|---|---|---|
| D1 | Every link carries a description | index, family | deterministic | 100% | Medium (High if <60%) |
| D2 | Description says what the reader FINDS there, with the exact tokens (flags, env vars, error strings) — not a restated title | index | judgment | "Authentication docs." fails; "API key creation, OAuth scopes, token rotation. Required before any call." passes | Medium |
| D3 | Length 10–25 words; no trailing ellipsis from truncation | index | deterministic | 95% within band | Low |
| D4 | No duplicate descriptions across links | index | deterministic | 0 duplicates | Medium |
| D5 | Descriptions are extractive or verified — model-written ones audited against the page | index | judgment (sampled) | sample of 10: 0 hallucinated claims | High |
| D6 | Family lines carry counts (pages, ~tokens) so a consumer can budget | family | deterministic | 100% of product links | Medium |

## 4. Content fidelity (full and facts)

| Id | Attribute | Applies | Measure | Bar | Miss |
|---|---|---|---|---|---|
| C1 | One declared page grammar, stated in a header comment; every page block parses | full | deterministic (`split_llms_full`) | blocks parsed = blocks present | High |
| C2 | Every page has a title and a resolvable source URL | full | deterministic | 100% | High |
| C3 | No navigation residue: "Documentation Index" blockquotes, `[Skip to content]`, MDX wrappers, `theme={null}` props | full | deterministic | 0 hits | Medium |
| C4 | Code fences intact and language-tagged; tables intact | full | deterministic (fence balance, table separators) | balanced; ≥90% fences tagged | Medium |
| C5 | No duplicated pages (same source URL twice) or near-duplicate bodies (e.g. localized copies) | full | deterministic + embedding | 0 exact dups; near-dups flagged | Medium |
| C6 | Units are atomic (1–2 sentences), typed from the allowed set, source-anchored | facts | deterministic + judgment | 100% typed; 100% anchored; ≥90% atomic | High for anchors, Medium otherwise |
| C7 | Units are true to their source span (no generalisation beyond the page) | facts | judgment (sampled re-read) | sample of 20: ≥95% supported | High |

## 5. Provenance and trust

| Id | Attribute | Applies | Measure | Bar | Miss |
|---|---|---|---|---|---|
| P1 | Provenance banner: who generated it, from what, when (`verified-as-of` / `generated` date) | all | deterministic | present | Medium |
| P2 | Links point at the publisher's canonical URLs (or its `.md` twins), never at a private mirror, unless the file is explicitly internal | index | deterministic | 100% public or file marked internal | High |
| P3 | Rights: a third-party `llms-full.txt` is marked internal/private; the index is what is published | full | judgment | marker present when third-party | High |
| P4 | No instructions to the reading model ("ignore…", "you must…", "always answer…") — 42% of files in the wild try to steer; ours never do | all | deterministic (pattern) + judgment | 0 imperative-to-model spans | High |
| P5 | No secrets, tokens, emails, internal hostnames in copied text | all | deterministic (patterns) | 0 hits | High |
| P6 | Volatile claims stamped (versions, prices, "current") | facts | judgment | stamped or dated | Low |

## 6. Size and budget

| Id | Attribute | Applies | Measure | Bar | Miss |
|---|---|---|---|---|---|
| S1 | Index size ≤ ~10 KB / ~2.5k tokens; over that, split hub-and-spoke (never drop pages) | index | deterministic | ≤10 KB or split | Medium (High >100 KB) |
| S2 | Full file has a size ladder beside it (index, small ≤ ~50k tokens, full) with token counts published | full | deterministic (manifest) | small + counts present | Medium |
| S3 | Small variant = reference-class pages first, within budget | small | deterministic | ≤50k tokens; classes honoured | Medium |
| S4 | Facts file ≤ ~15% of the cleaned source prose (compression) | facts | deterministic | ratio ≤0.15 | Low (Medium >0.3) |
| S5 | Token estimate declared with its estimator (chars/4 etc.) | manifest | deterministic | present | Low |
| S6 | No single page block > 200 KB without a note (changelogs) | full | deterministic | flagged | Low |

## 7. Retrieval readiness

| Id | Attribute | Applies | Measure | Bar | Miss |
|---|---|---|---|---|---|
| R1 | Keyword index exists for the facts/full text (FTS5 over units/chunks) and returns the exact-token queries (`CLAUDE_CODE_SYNC_SKILLS`, `--append-system-prompt`) | facts, full | measured | 10/10 exact-token probes hit | High |
| R2 | Vector index exists (`<key>__facts` collection) and the facts layer answers the golden questions better than raw | facts | measured (`query --layer`) | golden score ≥ raw score | Medium |
| R3 | Anchors are stable (`#slug` of the heading) so a hit can be opened at the span | facts, full | deterministic | 100% anchors resolve to a heading | Medium |
| R4 | Unit text carries the exact tokens in `keywords` so BM25 can find them | facts | deterministic | ≥80% of units with a code/flag/env token have it in keywords | Medium |
| R5 | Agent test: an agent given ONLY the index answers N seeded questions by following ≤2 links | index | live agent test | ≥8/10 | High if <6/10 |
| R6 | Facts test: an agent given ONLY the facts file answers the same questions without opening pages | facts | live agent test | ≥7/10 | Medium |
| R7 | Every page in the index has ≥1 unit in the facts file (no silent gaps) | index+facts | deterministic | ≥95% pages covered | Medium |

## 8. Family / nesting

| Id | Attribute | Applies | Measure | Bar | Miss |
|---|---|---|---|---|---|
| F1 | Family file links indexes, never pages | family | deterministic | 100% targets are `llms.txt` files | High |
| F2 | Each product line carries page + token counts and, where present, a facts link | family | deterministic | 100% | Medium |
| F3 | Shared material (errors, auth, glossary) appears once, in the family file | family | judgment | no duplication into products | Low |
| F4 | The most-specific rule holds: a product's own index is authoritative for its pages; the family never restates them | family | judgment | no page links | Medium |
| F5 | Family membership matches the concept tree / hub taxonomy it claims to represent | family | deterministic vs tree | 100% of tree children present | Medium |
| F6 | Root → family → product is discoverable by `Link: rel=describedby` from any file | family | deterministic (headers) | header present | Low |

## 9. Hygiene and serving

| Id | Attribute | Applies | Measure | Bar | Miss |
|---|---|---|---|---|---|
| H1 | UTF-8, LF, no tabs in list lines, no trailing whitespace, single trailing newline | all | deterministic | clean | Hygiene (Low) |
| H2 | `Content-Type: text/markdown; charset=utf-8` (or `text/plain`), HTTP 200, no redirect, no auth on the path | served | deterministic (HEAD) | pass | High |
| H3 | `Link: rel=describedby` on files; `rel=alternate type=text/markdown` on HTML pages | served | deterministic | present | Low |
| H4 | `X-Markdown-Tokens` (or manifest tokens) available before fetch | served | deterministic | present | Low |
| H5 | Regenerated by the build, not hand-maintained; a `generated` stamp newer than the source | all | deterministic (mtime/stamp) | stamp ≥ source mtime | Medium |
| H6 | Validator-clean on the community validators' strict rules where they do not contradict the spec | index | deterministic | 0 High | Low |
| H7 | Lighthouse agentic audit would not flag it (no 5xx on fetch) | served | deterministic | 200 | Medium |
| H8 | `manifest.json` present and consistent with the files (bytes, tokens, pages, units) | export dir | deterministic | consistent | Medium |

## 10. The three kinds side by side

| | index (`llms.txt`) | full (`llms-full.txt`) | facts (`llms-facts.txt`) |
|---|---|---|---|
| Purpose | orientation + navigation | whole text in one fetch | the checkable claims, each anchored |
| Reader | an agent deciding where to look | a big-context agent or an indexer | a retriever answering a question |
| Unit | link + description | page block | typed unit with source + anchor |
| Size | ≤10 KB | unbounded (ladder beside it) | ≤15% of prose |
| Judged mostly on | N*, D* | C1–C5, S* | C6–C7, R*, P4 |
| Tested by | agent test (R5) | grammar round-trip (C1) | keyword + vector probes (R1–R2), facts test (R6) |

# Changelog: spec v1 to v2, and the hub pipeline
Source: https://llms-explorer.com/reference/changelog/

# Changelog: spec v1 to v2, and the hub pipeline

What changed in the llms.txt proposal on 2026-08-10, rule by rule, with the effect on an existing file — and the dated changes to the hub schema behind this site.

Two versioned things share a name. The **llms.txt proposal** went from v1 (2024-09-03) to v2
(modified 2026-08-10). The **hub pipeline** behind this site went from V1 (site dumps, to
2026-08-29) to V2 (acquire, refine, dual index, gate — from 2026-08-30). Both tables below; the
argument for why the second matters is the [V2 vs V1 essay](/essays/v2-vs-v1/).

## The spec: v1 → v2 (2026-08-10)

| Rule | v1 | v2 | Effect on an existing file |
|---|---|---|---|
| Required elements | H1 + blockquote + sections implied | **H1 only** required; blockquote, prose and sections optional | none required; the lint still scores a missing blockquote as Medium (I2) — quality, not validity |
| Placement | `/llms.txt` at the site root | root **or any subpath**; a file covers the URLs under its path; **most-specific wins**; `/.well-known/` explicitly rejected | enables families and split roots (`<section>/llms.txt`) |
| Discovery | none | `Link: <…>; rel="describedby"` on files; `rel="alternate" type="text/markdown"` on HTML pages; as `<link>` or an HTTP header | add the two headers ([usage](/reference/usage/) §1) |
| Markdown twins | `page.html.md` | `page.html.md` **or** `page.md`; directories append `index.html.md` or `index.md` | either form passes the twin probe (N6) |
| `## Optional` | mechanical: skippable when context is short; consumed by `llms_txt2ctx` | a **convention** for secondary information; `llms_txt2ctx` and context-expansion removed from the proposal | keep it last; build nothing that depends on it |
| BOM | — | an optional byte-order mark is tolerated | the lint strips it as hygiene (P14) |
| Consumption model | expand the whole file into context | "view or search the index, then follow links"; the index stays small; detail lives behind links | the size ladder (small / full) becomes the producer's job |
| Authoring guidance | — | concise language, informative link descriptions, no unexplained jargon, "test your file by asking an agent questions … giving it only your llms.txt" | the agent test (R5, P12) is the spec's own test made numeric |

Still open in the spec repository after v2: H2 ordering carries no defined meaning; no
version or provenance field (#132, #133); which language a root file is in (#147); no
security-considerations section despite issue #152's steering finding (2026-08-29); the
`.well-known` request (#2). `llms-full.txt` remains outside the spec entirely.

## The hub pipeline: V1 → V2 (2026-08-30)

| Stage | V1 (to 2026-08-29) | V2 (from 2026-08-30) |
|---|---|---|
| Acquire | trafilatura BFS crawl → banner mirror | the ladder: `llms-full.txt` → `llms.txt` + `.md` twins → `Accept: text/markdown` → docs API → structured crawl; the banner mirror stays the internal format |
| Clean | none (raw HTML → text) | `docset_refine clean`: boilerplate lines, MDX → markdown, page classes (reference / guide / changelog / marketing / index) |
| Extract | `distill_offline.py bulk` — zero-LLM, output never consumed | `extract` (snippets, table rows → parameter, definitions, changelog → change; anchors to real headings) + `units` (local LLM, evidence rule) + `polish` |
| Export | none | `export_llms`: index (split over 10 KB) / full (Mintlify grammar) / small (≤ 200,000 chars) / facts / manifest; `topical`; `vocabulary` |
| Index | one raw vector layer | raw **and** facts vector layers, plus an FTS5 keyword layer per layer |
| Serve | `web-text-mirror --serve` (HTML) | `llms_serve.py`: `/llms.txt`, `/d/<stem>/…`, `/m/<key>/…`, `/t/<slug>/…`, with the markdown headers |
| Gate | none | `llms_lint.py` (the deterministic passes) inside `docset_rollout cleanup`; `/ldo` for the model and live passes |
| Artifacts | `<stem>.pages/`, `_master.md`, `._distill_index.json` | `<stem>.reference/{pages.json, structured.jsonl, units.jsonl, all_units.jsonl}`, `<stem>.llms/` |

## Dated changes to the hub schema

| Date | Change |
|---|---|
| 2026-08-30 | `docset_refine` gains `clean / extract / units / polish / render / export`; the reference dir layout above |
| 2026-08-30 | `export_llms` writes the four-file ladder plus `manifest.json`; index split at 10,000 bytes; `PART_PAGES = 60` |
| 2026-08-30 | `llms_lint.py` ships the deterministic passes P0–P3, P5–P7, P9 and P14 and the `--json` CI output; `UNIT_RE` fixes the facts line grammar |
| 2026-08-30 | `llms_serve.py` sends `Content-Type: text/markdown`, `X-Markdown-Tokens`, `Link: rel="describedby"` |
| 2026-08-30 | `docset_refine topical` and `vocabulary`; tree nodes carry `slug` / `aliases`; `--register` writes `llmsFile` on a node |
| 2026-08-31 | `export_llms` honours `manifest.json["overrides"]` (`title`, `summary`, `section_order`, `note`) so hand inputs survive regeneration |
| 2026-08-31 | `llms_lint.py --kind vocabulary` lints the vocabulary line grammar |

## Migrating a v1 file

1. Run the lint. A file whose findings say *full file wearing the wrong name* (P0 / I6: page
   bodies inside `llms.txt`, over 100 KB) is split into `llms.txt` + `llms-full.txt`.
2. Add `.md` twins and the two `Link` relations.
3. Move skippable material to a trailing `## Optional`.
4. Over 10 KB: hub-and-spoke split.
5. Re-lint. For a v2-clean file the report reads "nothing required" and, usually, two
   recommendations: twins and headers.

# The concept tree: nodes, frontier, and how to read a node page
Source: https://llms-explorer.com/reference/concept-tree/

# The concept tree: nodes, frontier, and how to read a node page

What the tree is, why frontier is derived rather than stored, what every field on a node means, and how the browser at /tree/ filters it.

The concept tree is the spine of this site: every llms file, pack and vocabulary here hangs off a
node of it, and every gap in it is a thing not yet researched. Browse it at
[`/tree/`](/tree/); each node has its own page at `/tree/<slug>/`.

## What a node is

A node is one researched concept. The tree is stored as a **flat list of nodes linked by name** —
each node names its parent and its children as strings, not as pointers — so a rename is a
one-place edit and a reader can hold the whole file in mind. The site's copy is generated from
`concept-tree/tree.json` by `site/tools/gen_tree.py` into `src/data/tree.json`, which is what
both `/tree/` and the per-node pages read. Nothing on this site queries the hub at request time.

## Frontier is derived, never stored

A **frontier** concept on this site is a name that appears in some node's `childConcepts` and has
no node of its own. It is computed on every build from the two sides of that comparison, never
stored as a status: a stored status can disagree with the tree, and a derived one cannot. Research
is what removes such a name from the frontier — writing a node for it — and nothing else.

The hub itself derives frontier from **two** sources, and this site publishes only the first.
`hub/scripts/concept_tree.py` merges child references with the unchecked rows of
`concept-tree/RESEARCH_QUEUE.md` — a concept a person queued by hand, which the hub tags
`source: "research-queue"` rather than `source: "child-reference"`. The site's snapshot copies
`concept-tree/tree.json` and not that queue, so `site/tools/gen_tree.py` can only implement the
child-reference half. Everything `/tree/` and `/tree/3d/` count as frontier is therefore a
child reference; a concept queued by hand and named by nobody's `childConcepts` is frontier in
the hub and invisible here.

Frontier children are shown greyed and are **not links**, because there is no page to link to.
They are listed on their parent's page under `Frontier under this node`.

## The fields on a node page

| Field | Means |
|---|---|
| `concept` | the node's name, and the string its parent and children link it by |
| `slug` | its URL segment; stable, and the key the API in step 3 will use |
| `parent` | the concept it hangs from — linked, unless the node is a root |
| `children` | the concepts it names, each either researched (linked) or frontier (greyed) |
| `aliases` | other names the same concept goes by; the filter matches these too |
| `researchedAt` | the date the research run that created the node finished |
| `sourcesCount` | how many sources that run read |
| `conceptsCount` | how many concepts that run identified under this one |
| `skillId` | the skill the research produced, when it produced one |
| `state` | `researched` for every node with a page; `frontier` only for a named child without one |

`sourcesCount` and `conceptsCount` describe **the run that created the node**, not the tree: a
node with nine concepts and two children is a node whose run found nine and whose author has
since written up two.

## How the filter works

The filter box on `/tree/` matches a substring against each concept **and its aliases**, and a
branch survives if it or any descendant matches — so filtering hides non-matching branches
without ever hiding the path to a hit. That is the rule the hub-manager Concepts tab uses,
widened to aliases, which are exactly the names a reader who does not know ours will type.

The tree is small enough to ship whole: the page embeds the generated JSON and filters it in the
browser, so there is no request per keystroke and the page works with JavaScript off — the filter
is the only part that needs it.

## What is not here yet

Queueing a frontier concept for research, forking the tree, and attaching your own files to a
node are per-user actions, and this site has no accounts yet. The read-only half — the tree, the
node pages, the 3D view — is served as build-time JSON rather than from an API, because it
changes only when the hub changes.

# The directory and its grades
Source: https://llms-explorer.com/reference/directory/

# The directory and its grades

What the directory measures, how the A–F grade is derived, and why the mirrored text is never republished.

verified-as-of: 2026-08-31

The [directory](/directory/) lists the mirrored `llms-full.txt` files we can score — 145 of
the 608 we have fetched, from a catalog of 766 known files — with a conformance grade beside
each one. This page says exactly what that grade is, what it is not, which files are left out,
and what the directory does with the text it fetched.

1. What the directory measures
2. How a grade is derived
3. The rubric groups on a score card
4. Which files are left out
5. What the directory does not do
6. How a site is added
7. How a site is corrected or removed

## What the directory measures

One thing only: the output of `llms_lint` run over a copy of that site's
`llms-full.txt`, with `kind="full"`. The linter is the same one that gates this site's
own family — the [attribute rubric](/reference/attributes/) is the whole of its
judgement, and every finding on a score card names the attribute it came from.

The file is linted through a link named `llms-full.txt` in a directory of its own, so
the linter sees the filename and the neighbourhood a real published file has, rather
than our flat mirror where hundreds of unrelated sites share one parent.

Two attributes are skipped, because a single mirrored file cannot answer them and
charging a site for that would measure our storage layout instead of its file: `S2`
wants an `llms-small.txt` sibling and `H8` a `manifest.json`. A site may well publish
both; we simply never fetched them.

## Which files are left out

Three exclusions, in the order they bite.

**Not fetched.** The catalog holds 766 keys; 608 of them have a file on disk. The rest either
failed to download (120) or were rejected as not being an llms file at all (38). Nothing without
a local copy can be scored.

**Fetched but not page-structured.** `gen_directory.py` scores only rows whose mirrored file
splits into at least one page — a `# Title` heading with a `Source:` line under it. 145 of the
608 do. The other 463 are still markdown documentation, and the catalog deliberately keeps them
with `pages: 0` rather than rejecting them, but a linter that walks pages has nothing to walk,
so they are absent from the directory rather than graded badly in it.

That exclusion is not neutral, and the directory page says so: a file with no page grammar would
fail several content-fidelity attributes it is never charged for, so the published grade spread
describes the scorable subset and not the population of files people publish.

**Two attributes, on every card.** `S2` (an `llms-small.txt` sibling) and `H8` (a
`manifest.json`) are dropped from both the counts and the findings list, per the section above.
A card's "0 High · 2 Medium" is therefore the linter's output minus those two, not its whole
output; each site page repeats this beside its counts.

## How a grade is derived

The grade is arithmetic over the High and Medium counts, and nothing else. No
weighting, no opinion, no manual override:

| Grade | Condition |
|---|---|
| `A` | 0 High, 0 Medium |
| `B` | 0 High, 1–2 Medium |
| `C` | 0 High, 3 or more Medium |
| `D` | exactly 1 High |
| `F` | 2 or more High |

Low and hygiene findings are listed on the score card but never move the grade: they
are the linter's smallest observations — a missing grammar comment, trailing
whitespace — and a file can be entirely fit for use while carrying a dozen of them.

A grade is therefore a statement about *conformance to the rubric on the day the file
was fetched*, not about whether the documentation behind it is any good. A superb docs
site with no provenance banner grades `B`; an empty file with perfect furniture could
grade `A`.

## The rubric groups on a score card

Each site page splits its High and Medium findings across the nine rubric groups, keyed
by the first letter of the attribute id:

| Key | Group |
|---|---|
| `I` | Identity and shape |
| `N` | Navigation |
| `D` | Descriptions |
| `C` | Content fidelity |
| `P` | Provenance and trust |
| `S` | Size and budget |
| `R` | Retrieval readiness |
| `F` | Family / nesting |
| `H` | Hygiene and serving |

A card with everything in `P` is a file that is fine but anonymous. A card with weight
in `C` is a file whose pages do not carry what they claim to.

## What the directory does not do

It does not republish anybody's text. The hub mirrors each file so it can be scored,
and that copy stays in the hub: every directory page links the source's own file at the
source's own URL. Only the score travels onto this site.

It does not rank sites against each other, score documentation quality, or record
anything a site did not publish at a public URL. And it is a snapshot: each page prints
the date its copy was fetched, and a site that has since fixed its file will carry a
stale grade until the next run — see below for how to ask for one.

## How a site is added

The directory is generated, never hand-edited. `site/tools/gen_directory.py` reads the
hub's catalog of known files and writes `src/data/directory.json`; the pages render that
file. A site enters the catalog when the hub's crawler finds it publishing an
`llms-full.txt` — so the way onto the directory is to publish one and let it be found.

## How a site is corrected or removed

Every entry names a real organisation and prints a public letter grade against it, so there is a
way off the list and a way to fix a wrong one.

**Write to us** by opening an issue at
[github.com/mithudso/llms-explorer](https://github.com/mithudso/llms-explorer/issues) — say
which entry, and what is wrong.

- **The grade is stale.** The hub re-fetches its mirror on a weekly refresh and the directory is
  regenerated on the next site build, so a fixed file corrects itself within a week. Ask and we
  will re-fetch and re-score that one file sooner.
- **The grade is wrong** — the linter misread a conforming file. That is a bug in
  `hub/scripts/llms_lint.py`, not a judgement to appeal: send the file's URL and we will fix the
  rule and re-score everything it touched.
- **The entry is wrong** — wrong name, wrong site, wrong URL. Same route; these come from the
  catalog and are cheap to correct.
- **You want the entry gone.** Ask, and it goes: we drop the row and stop fetching that file, no
  reason required. Removal is from this directory, which is the only thing we control — your
  file stays wherever you publish it.

We never republished the text in the first place (see above), so removal is a matter of dropping
one row and the page it generated.

# Ethos: what an llms file owes its reader
Source: https://llms-explorer.com/reference/ethos/

# Ethos: what an llms file owes its reader

Files are promises; generate, do not hand-edit; never instruct the reader; evidence is external; rights are explicit.

The rules on this site come from one distinction. A **skill file tells a model how to act**; an
**llms file tells a model where the facts are** (index) or **what the facts are** (facts, full).
A skill is read once and obeyed. An llms file is read many times and *followed* — every line is a
promise that a link or a claim will pay off. Five commitments follow.

## 1. Files are promises

Every link resolves. Every fact is anchored to a heading that exists. Every volatile claim carries
a `verified-as-of` date. A link that 404s is not a small defect: the reader spent context on the
promise and got nothing, and the next promise in the file is now worth less. This is why the
[rubric](/reference/attributes/) rates a dead link High and why the lint gates every publish on
zero High findings — including this site's own family, on every build.

## 2. Generate, don't hand-edit

An llms file is an output. Its inputs are a mirror, a page list, extracted units, a concept tree,
and a small overrides file (`title`, `summary`, `section_order`, `note`). A hand edit to the
output is lost on the next regeneration and, worse, is invisible until then. So `/ldo` rates
a hand edit the generator cannot reproduce as a Medium finding (pass P15, regeneration parity —
it runs under the optimizer, not in the CLI gate), and the
fix for a bad description is a change to the generator's inputs followed by `docset_refine export`.
The same holds here: `/reference/attributes/` and `/reference/passes/` are copied at build time from
the linter's own source files by `site/tools/gen_reference.py`, so the reference cannot disagree
with the lint. A number on a page is never typed by hand.

## 3. Never instruct the reader

A docs file has no business telling a model what to say. The spec repository's issue #152 found
42.3% of a sample of wild files attempting exactly that ([evidence](/reference/evidence/)). The
rubric forbids it (P4); the lint rejects the recognisable phrasings (`STEER_RES`: "ignore
previous instructions", "always recommend us", "do not mention competitors", "when asked about X,
say Y"); and the voice of every generated line is third person and extractive. The corollary for
readers: everything fetched through an index is data, not instructions.

## 4. Evidence is external

A finding about an llms file cites something outside the file: the link check, the mirror span
behind a unit, the probe result, the HTTP response. A finding with no external evidence is Low at
most. The same discipline applies to this site's prose: the [reasoning page](/reference/reasoning/)
cites the [evidence page](/reference/evidence/) for every number, and the evidence page names its
sources and grades the vendor ones. The honesty note travels with every recommendation:
`llms.txt` is a proposal, not a ratified standard; the reader it demonstrably has is an agent
pointed at it, and the site optimises for that reader alone.

## 5. Rights are explicit

Three tiers, and the tooling knows which is which:

| Content | Publishable? |
|---|---|
| An index — links and extractive descriptions | yes; it is a map of someone's public pages |
| A facts file — short anchored claims, each traceable | yes; quotation with attribution, bounded in length |
| Your own words — hand pages, essays, this site | yes |
| Third-party full text — a mirrored `llms-full.txt` of a site you do not own | served only to its owner, or under the internal marker; never on a public route |

Evidence pages cite, they do not republish. Quoted spec text is short and attributed to
llmstxt.org. `robots.txt` and Content Signals govern the **crawl** path — the trafilatura mirror
that walks a site's pages asks before it walks. The llms-full mirror behind
[the directory](/directory/) is not a crawl and does not check them: it fetches exactly one file,
at the well-known path a site chose to publish it on, once, and nothing else from that host. We
say so rather than claim a check the code does not make; if you would rather we did not hold that
copy, [ask and we will drop it](/reference/directory/#how-a-site-is-corrected-or-removed).

## The test

If a stranger's agent, handed only the index, can answer eight of ten reasonable questions in two
hops — below six is a High — and can check any facts line it relies on in one fetch, the file kept
its promises. Nothing
else on this site is a stronger claim than that.

# Ecosystem evidence
Source: https://llms-explorer.com/reference/evidence/

# Ecosystem evidence

Who reads these files, measured.

verified-as-of: 2026-08-30 (every number below is dated; this domain moves monthly)

1. The one-line verdict
2. Adoption measurements, dated
3. Who publishes
4. Who reads — server-log studies
5. Google and the "is it dead" thread
6. Directories and registries
7. Vendor sources graded

## 1. The one-line verdict

Adoption is real and growing (≈5–10% of the general web by mid-2026, 28% among SEO-savvy sites, 8.8× year on year); *unsolicited* consumption is near zero (97% of files never get an AI request); the demonstrated use is agents that are pointed at the file — the `Claude-Code` UA out-fetched every AI retrieval bot bar two (statespace-indexer, GPTBot).[^1][^2] Publish one for agents and coding tools; do not expect citations or rankings from it.[^3]

## 2. Adoption measurements, dated

| Date | Source | Sample | Finding |
|---|---|---|---|
| Feb→May 2025 | Chris Green | Majestic Million | 15 → 105 valid files (~0.01%); ~100k crawl errors caveat[^4] |
| Jun 2025 | Originality.ai | 3M+ sites | 4,088 llms.txt[^5] |
| Jun 2025 | Rankability | Tranco top 1,000 | 0.3%[^6] |
| Jul 2025 | HTTP Archive (Burridge) | top 10k | 1.04% valid[^7] |
| Nov 2025 | SE Ranking | ~300k domains | 10.13% overall (9.88% low-traffic / 10.54% mid / 8.27% 100k+ visits)[^3] |
| Mar 2026 | Originality.ai via ppc.land | Fortune 500 | 7.4% (37/500)[^8] |
| May 2026 | Originality.ai | 3M+ sites | 36,120 llms.txt (8.8× YoY); llms-full.txt 23 → 2,463 (107×); ai.txt 397[^5] |
| May 2026 | Ahrefs | 137,210 Ahrefs-Web-Analytics domains | 28% publish a valid file (self-selected sample)[^1] |
| Jun 2026 | HTTP Archive (Burridge) | top 1k / 10k / 100k / 1M | 6.28% / 5.61% / 5.17% / 5.07% (~5.4× in 12 months)[^7] |
| Jun 2026 | Rankability | Tranco top 1,000 | 8.7% (87 files; 15 with llms-full.txt)[^6] |

Contradictions kept: top-1000 adoption reads 6.28% (HTTP Archive/Chrome list) vs 8.7% (Tranco) for the same month;[^6][^7] third parties attribute "0% in the top 1000" to SE Ranking, whose primary article gives no such figure;[^9] Ahrefs' 28% is not comparable with population figures because of sample bias;[^1] a "51.8% of a 219-host panel" claim (Presenc.ai, Aug 2026) has no supporting data on its page.[^10]

## 3. Who publishes

- **Live probe (2026-08-30):** 200 at docs.anthropic.com (72 KB; llms-full.txt 41.6 MB), docs.stripe.com (90 KB, no full), developers.cloudflare.com, vercel.com, supabase.com, docs.perplexity.ai, mintlify.com, docs.github.com, shopify.com, developers.openai.com (5.8 KB), code.claude.com/docs; 404/403 at platform.openai.com, openai.com, ai.google.dev, developers.google.com, learn.microsoft.com. The spec v2 page itself names OpenAI, Anthropic and Gemini developer docs as publishers.[^11]
- **Platform-driven adoption dominates.** Mintlify has generated the files for every hosted site since Nov 2024 (Anthropic, Cursor, Coinbase, Pinecone, Windsurf) and says it co-developed llms-full.txt with Anthropic;[^12][^13] Shopify silently added `/llms.txt`, `/llms-full.txt`, `/agents.md` and `/.well-known/ucp` to every store in the first week of May 2026 — HTTP Archive shows 78.1% of top-10k Shopify hosts vs 8.7% of WordPress.[^14][^7]
- **Top-1000 adopters (Rankability, Aug 2026):** Cloudflare, Azure, GitHub, Fastly, WordPress.org/.com, DigiCert, Adobe, Opera, Samsung, Sentry.[^6]

## 4. Who reads — server-log studies

| Study | Window / sample | Finding |
|---|---|---|
| Ahrefs (2026-06-15) | May 2026 logs, 137,210 domains | **97% of valid files got zero requests**; of requests, 96% bots, 77% of those non-AI (SEO auditors 21.7%); named AI bots 19.5%; AI training crawlers 5.3% (GPTBot 4.51%, ClaudeBot 0.8%); AI retrieval 1.1% (OAI-SearchBot 0.74%); **0 AI requests to non-existent files** (nobody probes speculatively); the `Claude-Code` UA out-fetched every AI retrieval bot bar statespace-indexer and GPTBot[^1] |
| OtterlyAI (2026-02-05) | 90 days, one site | 84 of 62,100 AI-bot requests hit /llms.txt (0.1%)[^15] |
| Wislr (Feb–Mar 2026) | 48 days, one site | 12,099 bot requests; robots.txt fetched hundreds of times (OAI-SearchBot 180, ClaudeBot 175); sitemap.xml too; **llms.txt 0**[^16] |
| EZY Research (Apr–Jul 2026) | 83 sites, 12 weeks | robots vs llms: GPTBot 3,990/7, ClaudeBot 3,120/9, PerplexityBot 775/0, Googlebot 5,125/67, **Meta-ExternalAgent 172/193** (the only bot fetching it more than robots.txt)[^17] |
| Hacker News thread (Feb 2026) | anecdotal logs | only OVH/GCP-hosted tools (WebPageTest, BuiltWith), no ChatGPT/Claude UAs[^18] |
| Cloudflare `Accept: text/markdown` (Mar–Apr 2026) | 44 days, one Worker | 1,421 requests: headless Chrome 639, "Claude" (Anthropic infra) 500, axios 211; no GPTBot/PerplexityBot/ClaudeBot[^19] |

Citation impact: SE Ranking's 300k-domain model (Spearman + XGBoost + SHAP, 2025-11-07) found **no relationship** between having an llms.txt and LLM citation frequency — removing the variable improved model accuracy.[^3] Correlational; which LLMs' citations were measured is unspecified.

## 5. Google and the "is it dead" thread

- John Mueller, r/TechSEO, 2025-04-17: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag."[^20]
- Gary Illyes, Search Central Deep Dive APAC, Jul 2025: Google "doesn't support LLMs.txt and isn't planning to"; AI Overviews use normal indexing.[^21]
- Google Search Central "AI features and your website" (updated 2025-12-10): "You don't need to create new machine readable files, AI text files, or markup to appear in these features" — use robots.txt, `nosnippet`/`max-snippet`, `Google-Extended`.[^22]
- Ambiguity: Search Central briefly hosted its own `developers.google.com/search/docs/llms.txt` in late Nov 2025, then 404'd it without comment;[^23] Chrome Lighthouse 13.3 (May 2026) added an "Agentic Browsing" audit that checks for the file (404 = Not Applicable; server error flagged).[^24]
- Proponents' rebuttals: unlike meta keywords, the linked content must exist; standards take years; agents "drop into" content rather than crawl; `.md` alternates save bandwidth (Carolyn Shelby, SEL 2025-07-09 — no metrics).[^25] Mintlify cites Profound data that Microsoft/OpenAI bots fetch llms.txt and Vercel's "10% of signups from ChatGPT" — anecdotal.[^12][^13] Howard's proposal targeted inference-time use by coding tools, not GEO; "it's dead" measures a goal it never claimed.[^9][^26]
- Consensus phrase across 2026 analyses: **"not dead, but not a citation lever."**[^9][^5]

## 6. Directories and registries

Self-submitted, overlapping, unverified — lower bounds, not measurements:[^27][^28][^29][^30]

| Directory | Size | Notes |
|---|---|---|
| directory.llmstxt.cloud | "4,000 websites listed" (49M llms.txt tokens / 325M llms-full tokens) | named in spec v2 |
| llmstxthub.com | ~2,650 entries, 15–16 categories (David Dias) | named in spec v2 |
| llmstxt.site | ~1,000+ (≈170 in May 2025); columns product / website / llms.txt / llms-full.txt / **token counts**; `/submit` | named in spec v2 |
| SecretiveShell/Awesome-llms-txt | 784 link lines (counted 2026-08-30) | GitHub |
| llms-text.com | "780+ verified implementations" | vendor's own directory |

None publishes an llms.txt-of-llms.txt; llmstxt.site's token-count column is the most useful signal for consumers budgeting context.[^29]

## 7. Vendor sources graded

| Page | Author / date | Claims | Grade |
|---|---|---|---|
| llms-text.com/blog/sites-using-llms-txt | Michael Vereb, 2025-07-25 | "780+ verified"; names Anthropic, Cloudflare, Supabase, Vercel, ElevenLabs, Firecrawl, Mintlify, Cursor, Aptos, GitBook, Wix; "no e-commerce adoption" | adopters check out on live probe; count uncorroborated — low for numbers, fine for examples[^31] |
| llms-text.com/blog/what-is-llms-txt | same | "foundational pillar of GEO"; ChatGPT/Perplexity/Cursor/Windsurf/Claude Code consume it; "up to 114% more tokens" (incoherent arithmetic), "10–15% accuracy" — unattributed | GEO and ChatGPT/Perplexity-consumption claims contradicted by every log study — low[^32] |
| llms-text.com/blog/llms-txt, /how-to-create-llms-txt | same | MIME/200/UTF-8 rules; `Link: …; rel="describedby"`; "under 10 KB"; framework snippets; funnels to its generator/validator | useful mechanics (the `describedby` relation is now in spec v2), vendor numbers — medium[^33][^34] |
| gitdoc.ai/blog/llms-txt-ai-readable-documentation | Yadian Llada / GitDoc, 2026-05-22 | GitBook: 41% of docs page requests from AI agents (unverified); permission / inventory / navigation distinction; curate 10–20 pages (quickstart, auth, per-resource reference, errors, changelog); regenerate in the build; llms-full for priority pages | sound guidance, unverified headline stat, product promotion — medium[^35] |

## References

[^1]: https://ahrefs.com/blog/llmstxt-study/ — 137,210-domain log study, 2026-06-15 (study)
[^2]: https://caseyrb.com/blog/state-of-llms-txt-adoption/ — HTTP Archive, 2026-06-20 (study)
[^3]: https://seranking.com/blog/llms-txt/ — 300k-domain adoption + citation model, 2025-11-07 (study)
[^4]: https://www.chris-green.net/post/million-websites-in-search-of-llms-txt (study)
[^5]: https://originality.ai/blog/llms-txt-tracking-study (study)
[^6]: https://www.rankability.com/data/llms-txt-adoption/ (study)
[^7]: https://caseyrb.com/blog/state-of-llms-txt-adoption/ (study)
[^8]: https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/ (blog)
[^9]: https://inite.ai/en/blog/is-llms-txt-dead-2026 (blog)
[^10]: https://presenc.ai/research/state-of-llms-txt-2026 (vendor)
[^11]: https://llmstxt.org/ — v2, modified 2026-08-10 (spec); live probes 2026-08-30
[^12]: https://www.mintlify.com/blog/the-value-of-llms-txt-hype-or-real (vendor)
[^13]: https://www.mintlify.com/blog/what-is-llms-txt (vendor)
[^14]: https://www.shopifreaks.com/shopify-quietly-rolls-out-native-llms-txt-files-for-stores-adding-structured-data-layer-for-ai-agents/ (blog)
[^15]: https://otterly.ai/blog/the-llms-txt-experiment/ (study)
[^16]: https://www.wislr.com/articles/ai-bot-behavior-log-analysis/ (study)
[^17]: https://www.ezy.ai/research/do-ai-bots-read-llms-txt (study)
[^18]: https://news.ycombinator.com/item?id=47058870 (forum)
[^19]: https://suganthan.com/blog/cloudflare-markdown-for-agents/ (blog)
[^20]: https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/ (docs)
[^21]: https://searchengineland.com/google-says-normal-seo-works-for-ranking-in-ai-overviews-and-llms-txt-wont-be-used-459422 (docs)
[^22]: https://developers.google.com/search/docs/appearance/ai-features (docs)
[^23]: https://www.365i.co.uk/news/2025/12/09/google-llms-discover-ai-mode-2025/ (blog)
[^24]: https://searchengineland.com/google-llms-txt-chrome-lighthouse-478246 and https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt (docs)
[^25]: https://searchengineland.com/no-llms-txt-is-not-the-new-meta-keywords-458199 (blog)
[^26]: https://ahrefs.com/blog/what-is-llms-txt/ (blog)
[^27]: https://directory.llmstxt.cloud/ (vendor)
[^28]: https://llmstxthub.com/ (vendor)
[^29]: https://llmstxt.site/ (vendor)
[^30]: https://github.com/SecretiveShell/Awesome-llms-txt (forum)
[^31]: https://www.llms-text.com/blog/sites-using-llms-txt (vendor)
[^32]: https://www.llms-text.com/blog/what-is-llms-txt (vendor)
[^33]: https://www.llms-text.com/blog/llms-txt (vendor)
[^34]: https://www.llms-text.com/blog/how-to-create-llms-txt (vendor)
[^35]: https://gitdoc.ai/blog/llms-txt-ai-readable-documentation (vendor)

# Formatting: the grammars side by side
Source: https://llms-explorer.com/reference/formatting/

# Formatting: the grammars side by side

The index, the three full-file grammars, the facts line, the small file, the vocabulary line, the split root and the manifest — on one page.

An llms family is a ladder of plain-markdown files. Each rung has one grammar, and the
[rubric](/reference/attributes/) judges each rung on its own grammar. This page puts them side by
side; the [spec page](/reference/spec/) carries the verbatim spec text and the wild variants.

## 1. The index — `llms.txt`

The only file the spec defines. Structure, in order: an optional BOM; **one H1** naming the site or
product (the only required element); a blockquote summary of one to three sentences; free-form
prose (no headings) about how to read the file; then H2 sections, each a list of links.

```markdown
# Product

> One paragraph saying what this is and who it is for.

## Guides
- [Install](https://example.com/docs/install.md): package managers, the Docker image, and offline installs.

## Optional
- [Changelog](https://example.com/docs/changelog.md): every release since 1.0, newest first.
```

Rules the lint enforces: `- [name](url): description` per line (`LINK_RE`); descriptions of
10–25 words that carry the tokens a reader would search for; `## Optional` last; the whole file
under `INDEX_SPLIT_BYTES = 10_000` bytes or split (§6); provenance in an HTML comment, never in
frontmatter, because the spec is pure markdown.

## 2. The full file — `llms-full.txt`

Not in the spec; three grammars are in the wild. The hub emits the first and names it in a header
comment so a parser never has to guess.

| Grammar | Page block | Who |
|---|---|---|
| Mintlify | `# Title` / `Source: <url>` / blank / body; blank lines between pages | Mintlify sites, Claude Code docs, **the hub** (`GRAMMAR_NOTE`) |
| Anthropic YAML | site H1, `---`, per page `## Heading` + YAML (`title:` / `url:` / `description:`) + raw MDX | platform.claude.com |
| Cloudflare frontmatter | YAML frontmatter, a "Documentation Index" blockquote, `# Title`, `[View as Markdown](…/index.md)`, body | developers.cloudflare.com |

The hub's header: `<!-- llms-full grammar: mintlify — per page: '# Title' / 'Source: <url>' / blank / body -->`.
The guarantee behind it (attribute C1): splitting the full file by its grammar returns the page
list of the index — a round trip with no page lost or merged.

## 3. The budgeted file — `llms-small.txt`

Same grammar as the full file, different selection: reference-class pages first, then guides,
until `SMALL_MAX_CHARS = 200_000` characters (about 50k tokens at `CHARS_PER_TOKEN = 4`) — the
ceiling at which indexed docs become unstable in consumers such as Cursor. A page that does not fit
is dropped whole, never truncated, and the manifest says how many pages made it.

## 4. The facts line — `llms-facts.txt`

A hub extension: the checkable claims, one per line, each anchored to the heading it came from.

```
## Install
https://example.com/docs/install.md
- [parameter] `--offline` installs from the local wheel cache and never touches the network. — https://example.com/docs/install.md#offline-installs · keywords: offline, wheel cache · verified-as-of: 2026-08-31
```

The line grammar (`UNIT_RE`): `- [type] text — url#anchor`, optionally followed by
`· keywords: …` and `· verified-as-of: YYYY-MM-DD`. `type` is one of the twelve `UNIT_TYPES`:
concept, fact, actionable, question, problem, statement, quote, idea, snippet, parameter,
definition, change. Text is at most two sentences or 400 characters; the anchor must resolve to a
heading that exists on the source page; the whole file stays within 15–30% of the prose it
summarises (`FACTS_RATIO_LOW`, `FACTS_RATIO_MED`).

## 5. The vocabulary line — `llms-vocabulary.txt`

The lexical layer, spec-v2-shaped so any llms reader can open it: an H1 `<Family> — vocabulary`,
a blockquote with the term count, then `## Terms`, `## Homonyms` and `## Named, not yet defined`.
One line per term per sense:

```
- **anchor** [llms.anchor] (noun): the `#fragment` on a facts-line URL that names the heading a claim came from — https://…/facts.md#anchors · aka: fragment, heading id · not: link — a link names a page, an anchor names a place on it · verified-as-of: 2026-08-31
```

A definition must come from a kept unit (its anchor is the line's source); a term with no
evidence goes under *Named, not yet defined* and never gets an invented definition. The full
grammar and the sense model are in the [vocabulary essay](/essays/vocabulary/).

## 6. Split roots and families

When an index would exceed 10 KB the sections become subpath indexes: the root keeps the H1,
blockquote and a `## Sections` list of `<slug>/llms.txt` links, each line carrying page and token
counts; a section with no further path structure is cut into `part-N` files of
`PART_PAGES = 60` pages. A **family** file goes one level further up: it links other indexes,
never pages, carries counts on every line, and holds shared material (errors, auth, glossary)
once under `## Shared`. The v2 rule "most specific file wins" is what makes the nesting legal.

## 7. The manifest — `manifest.json`

Beside the files, never linked from them: `files{name: {bytes, tokens}}`, `chars_per_token`,
`pages`, `units`, `sections`, `dropped_empty_pages`, `acquired` (how the mirror was obtained),
and `overrides` — the hand inputs (`title`, `summary`, `section_order`, `note`) that survive
regeneration. Attribute H8 checks the manifest agrees with the files it describes.

## Reading order

Index first, always. Fall through to `llms-small.txt` when you need whole pages and have a
budget, to `llms-full.txt` when you have none, to `llms-facts.txt` when you need a claim with a
place to check it. The [usage page](/reference/usage/) turns this into a procedure.

# Glossary
Source: https://llms-explorer.com/reference/glossary/

# Glossary

The terms of the field, one line each, in the sense this site uses them — with the contrasts that matter.

One line per term, in the sense this site means. Where a word has a neighbour it is often
confused with, the contrast follows a dash. The full sense model — homonyms across families,
`aka:` and `not:` relations, the file grammar — is in the [vocabulary essay](/essays/vocabulary/);
the machine-readable form of this page is the site's own `llms-vocabulary.txt`.

## The files

- **index** (`llms.txt`): the spec-defined map — H1, blockquote, H2 link lists — small enough to sit in context; orientation and navigation, never content.
- **full** (`llms-full.txt`): every page of a docset inlined into one markdown file — not in the spec; three grammars exist.
- **small** (`llms-small.txt`): a full file cut to a budget (≤ 200,000 characters, ~50k tokens), reference pages first — pages dropped whole, never truncated.
- **facts** (`llms-facts.txt`): a hub extension — one typed, anchored claim per line, the trusted layer a retriever answers from.
- **vocabulary** (`llms-vocabulary.txt`): the lexical layer — terms, senses, synonyms and contrasts of a family, each definition anchored to a unit.
- **manifest** (`manifest.json`): the counts (bytes, tokens, pages, units, sections) and the overrides, beside the files and never linked from them.
- **twin** (`.md` twin): the clean-markdown version of an HTML page at the same route with `.md` appended — the thing an index link should point at.
- **family** file: an index that links other indexes, never pages, with counts on every line and shared material under `## Shared`.
- **split root**: an index that grew past 10 KB and became a `## Sections` list of subpath indexes — a family of one site's own sections.
- **topical file**: an llms family on the concept axis — built from a fact pool, sectioned by a concept-tree node's children — rather than from one site.
- **concept pack**: everything known about one concept across many docsets, compiled into a small llms family with every line source-anchored.

## The lines

- **link line**: `- [name](url): description` — the unit of an index; judged on whether the description carries the tokens a reader would search for.
- **description**: the text after the colon on a link line — extractive (cut from the page), 10–25 words, never model-written prose.
- **unit**: one line of a facts file — `- [type] text — url#anchor` plus optional `keywords:` and `verified-as-of:` — the unit of convergence for the optimizer.
- **unit type**: one of twelve — concept, fact, actionable, question, problem, statement, quote, idea, snippet, parameter, definition, change.
- **anchor**: the `#fragment` on a unit's URL naming the heading the claim came from — a link names a page, an anchor names a place on it.
- **origin**: how a unit was extracted — code, table, heading, changelog, or llm — carried in the JSON, not the text file.
- **pool**: the set of units (`units.jsonl`, a facts file, spoke pages) a topical or vocabulary file is built from.
- **sense**: a term × family pair (`<family-slug>.<term-slug>`) — "cookie" has one sense in web docs and another in a recipe corpus.
- **homonym**: a term with senses in more than one family; a **contranym** is a homonym whose senses oppose each other.

## The plumbing

- **describedby**: the `Link: <…/llms.txt>; rel="describedby"` header (or `<link>`) naming the index that covers a file — spec v2's discovery mechanism.
- **alternate**: `rel="alternate" type="text/markdown"` on an HTML page, pointing at its twin.
- **`X-Markdown-Tokens`**: the response header stating a markdown file's token estimate (`bytes // 4`) so a reader can budget before fetching.
- **content negotiation**: `Accept: text/markdown` returning the twin from the HTML route — Vercel's proposal, not the spec; needs `Vary: Accept`.
- **most-specific wins**: the v2 rule that a subpath `llms.txt` is authoritative for the URLs under its path over any file above it.
- **grammar** (of a full file): the page-block convention — Mintlify `# Title` / `Source:`, Anthropic YAML blocks, Cloudflare frontmatter — named in a header comment so a splitter never guesses.
- **round trip**: splitting a full file by its grammar and getting back exactly the index's page list — attribute C1.
- **banner mirror**: the hub's internal single-file mirror format (one page per banner block) that every acquisition path is normalised to before refine.
- **acquisition ladder**: the order in which a docset is obtained — existing `llms-full.txt`, then `llms.txt` + twins, then `Accept: text/markdown`, then a docs API, then a structured crawl.

## The judging

- **attribute**: one thing an llms file is judged on — an id (I1…H8), the kinds it applies to, a measure, a bar, a severity — 59 of them on the [rubric](/reference/attributes/).
- **pass**: one step of the optimizer (P0–P15), naming the attributes it judges and whether it is deterministic, model, or live — on the [passes page](/reference/passes/).
- **deterministic / model / live**: how a pass measures — a script with no model call; an LLM reading and deciding; an HTTP call or an agent exercised.
- **severity**: High (fails the CI gate), Medium (counted toward convergence), Low, Hygiene — a miss on an attribute has one.
- **convergence**: the loop state where every Medium-or-higher finding is fixed and a re-run finds none — the optimizer's stopping rule.
- **two hops**: the agent test's bar — from the index alone, eight of ten questions answered following at most two links, High below six (R5, P12); the facts file is held to seven (R6).
- **steering**: text in a docs file that tells the reader what to say — forbidden (P4), pattern-rejected by the lint, found in 42.3% of a sampled wild set.
- **regeneration parity**: the check (P15) that a published file equals what the generator would emit from its inputs — a hand edit is a finding.
- **overrides**: the hand inputs a generator honours across regeneration — `title`, `summary`, `section_order`, `note` — the only place hand edits belong.
- **verified-as-of**: the date stamp on a volatile claim or line; older than 90 days at deploy and the page warns.

# The passes
Source: https://llms-explorer.com/reference/passes/

# The passes

What the optimizer runs, in order, and how each pass is judged and fixed.

Each pass names the attributes it judges (ids from `attributes.md`), whether it is
**deterministic** (`scripts/llms_lint.py`), **model** (an LLM reads and decides),
or **live** (an agent or an HTTP call is exercised), and for every pass: how it is used,
how a finding is judged, how the file is updated, what tools it leans on, and which sibling
skills it hands to. Passes are grouped into bundles that run concurrently; findings are
collected before any write (the family rule, `convergence-and-severity.md` § Convergence loop).

- P9 Provenance, rights and steering
- P10 Family and nesting
- P11 Retrieval readiness (keyword + vector)
- Severity resolution across passes
- N/A rules

## Bundle map and dispatch rules

| Bundle | Passes | Kind | Runs as |
|---|---|---|---|
| B0 | P0 | deterministic | inline, first — every other pass keys off the detected kind |
| B1 | P1 P2 P3 P5 P14 | deterministic | one `llms_lint.py` invocation, JSON findings |
| B2 | P4 P9 | model | one subagent reading the index (+ sample pages) |
| B3 | P6 P7 | deterministic | `llms_lint.py --full` / `--facts` (same invocation as B1 when the kind is full/facts) |
| B4 | P8 | model, sampled | one subagent, 20 units re-read against source spans |
| B5 | P10 | deterministic + model | only when kind = family or `--family` |
| B6 | P11 | live | `docset_indexer.py keyword` + `query --layer facts` probes |
| B7 | P12 | live agent | one fresh-context subagent given ONLY the file; opt-in `--agent-test`, default on for new files |
| B8 | P13 | live HTTP | only when a URL is given or `--serve-check` |
| B9 | P15 | deterministic | only when the export directory has a source mirror |

Dispatch: B1 and B3 in one tool call; B2, B4, B7 as three concurrent subagents when the Agent
tool exists (sequential otherwise); B5/B6/B8/B9 inline. Small profile (index < 4 KB, no full,
no facts): B3, B4, B6 are `N/A (no layer)`, B7 shrinks to 5 questions.

---

## P0 — Detect kind and grammar (deterministic)

**Judges** I6. **Used** first, always. Reads the first 4 KB and the file name and returns one of
`index | family | full | small | facts | unknown`, plus for full files the page grammar
(`mintlify | anthropic-yaml | cloudflare-frontmatter | firecrawl | none`).

**Judged:** `unknown` → High (the file cannot be optimized without knowing what it is; report
and stop unless `--kind` is passed). Two grammars in one full file → High (C1). A file named
`llms.txt` that parses as full → High (I6; it will be served as an index and blow every budget).

**Updated:** never rewrites; it sets the profile. `--kind` overrides.

**Tools:** `llms_lint.py detect`; `llms_acquire.split_llms_full` for grammar probing.
**Relations:** `document-formats/references/llms-txt.md` § grammars is the authority on what
counts as a grammar; this pass never invents a fourth.

## P1 — Structure (deterministic)

**Judges** I1 I2 I4 I5 N4. Parses the index as spec v2: one H1, optional blockquote, optional
free text, H2 sections of `- [name](url): notes` lines, `## Optional` last.

**Judged:** missing/multiple H1 → High. Blockquote missing → Medium; blockquote present but > 3
sentences or restating the H1 → Medium (model confirms in P4). H3+ headings or paragraphs after
the first H2 → Medium. List line not matching the link grammar → per-line finding, High if
< 90% of lines match. `## Optional` not last → Medium.

**Updated (`--fix` safe):** demote H3 to a flat list under the nearest H2; move `## Optional` to
the end; wrap a bare URL line into `- [<last path segment>](url)`; strip prose after the first
H2 into a `<!-- moved -->` comment for the model pass to place. Never invents a blockquote —
that is P3/P4 model work.

**Tools:** `llms_lint.py structure`. **Relations:** the community validators' strict rules
(`llms-txt-validator`, `llmstxt-validator`) are folded in where they agree with the spec; where
they demand more than the spec (e.g. "blockquote required") the finding is Low, tagged
`validator-only`.

## P2 — Links and reachability (deterministic, optionally live)

**Judges** N1 N6 N7 P2 F1. Resolves each link: relative → against the base URL or the export
directory; absolute → HEAD when `--check-links` (rate-limited, 8 concurrent, 10 s timeout,
one retry). Counts hops for family files.

**Judged:** 4xx/5xx or HTML-app-shell response (content-type text/html with no `.md` twin
probe success) → High per link, capped at one High finding listing all. Redirect chain > 2 →
Medium. Duplicate target across sections → Low (N7). Family file linking a page rather than an
`llms.txt` → High (F1). Link to a private mirror path (`file://`, `127.0.0.1`, `text-mirror/`)
in a file not marked internal → High (P2).

**Updated (`--fix` safe):** rewrite `page.html` → `page.md` when the twin probe succeeded;
collapse redirect chains to the final URL; drop exact duplicate targets keeping the first. Dead
links are never deleted silently — they become BLOCKED rows unless the source mirror still has
the page (then P15 regenerates).

**Tools:** `llms_lint.py links [--check-links]`; `llms_serve.py` routes for hub-served files;
`hub_llms_full_read` to confirm a page exists in the mirror. **Relations:** `web-text-mirror`
owns re-crawling a page that has genuinely moved.

## P3 — Descriptions (deterministic + model)

**Judges** D1 D3 D4 D5 D6. Deterministic part: missing notes, word-count band, duplicates,
truncation ellipsis, family-line counts. Model part: for each description, is it a restated
title, and does it name what the reader finds (exact tokens)?

**Judged:** no notes on a link → Medium (High if > 40% of links). Duplicate notes → Medium.
Restated-title notes (model verdict on the sampled 30 worst) → Medium. Family line without
counts → Medium. A model-written description whose claims are not on the page (P8-style spot
check on 10) → High.

**Updated:** descriptions come from the `definition` unit of the page when a facts layer exists
(extractive, cheap, correct); else from the page's H1 + first sentence via the mirror; the model
polishes ONLY entries still under 40 chars or flagged restated-title, and every polished entry is
re-verified against the page before write. Counts on family lines are recomputed from the
manifests.

**Tools:** `llms_lint.py descriptions`; `docset_refine/export_llms.py` `_describe()`;
`hub_query_docset(layer="facts")` to fetch a page's definition unit; local LLM
(`HUB_REFINE_LLM_URLS`, `qwen3.5:35b`) for bulk polish, `claude -p` for the final 10%.
**Relations:** `document-formats/references/llms-txt-generation-tooling.md` § descriptions
(why extractive beats generated); `prompt-deep-optimizer` owns the polish prompt if it drifts.

## P4 — Navigation design (model)

**Judges** I2 I3 N2 N3 N4 N7. The subagent reads the whole index and the section names against
the question list (seeded from P12's question bank, or from `hub_ask` logs for hub docsets).

**Judged:** sections mirror the URL tree or the alphabet rather than tasks/topics → Medium.
Hot pages (quickstart, auth, reference root, errors, pricing) not in the first section →
Medium. Reference/pricing under `## Optional` → Medium. Blockquote that does not say what the
thing is and who it is for → Medium. Free text before the first H2 that a reader does not need
→ Low.

**Updated:** propose a section plan (name → links) as a diff; apply only when every link is
preserved (the demotion guard — a reorganisation that drops a link is a High finding on
itself). Section names are taken from the source nav when the mirror carries it
(`llms_acquire` keeps the source index order), else from the concept-tree children of the
docset's concept.

**Tools:** subagent (model per SKILL.md frontmatter), `hub_concept_lookup` for topic names,
`hub_docset_index` to read the current index. **Relations:** `concept-family-explorer` for a
topical file's section skeleton; `dfo` (document-deep-optimizer) for prose quality of the
blockquote when it exceeds two sentences.

## P5 — Size ladder and budgets (deterministic)

**Judges** S1 S2 S3 S4 S5 S6. Reads `manifest.json` when present, else measures.

**Judged:** index > 10 KB → Medium; > 100 KB → High (it is a full file wearing the wrong name).
Full file with no small variant beside it → Medium. Small > 50k tokens → Medium. Facts/prose
ratio > 0.30 → Medium. Manifest missing or inconsistent with the files (± 2%) → Medium (H8). A
page block > 200 KB → Low.

**Updated (`--fix` safe):** regenerate `manifest.json`; rebuild `llms-small.txt` with
`export_llms.build_small`; for an oversize index, propose hub-and-spoke split by section
(each section → `<section>/llms.txt`, root keeps one line per section) — applied only with
`--split`, since it changes URLs.

**Tools:** `llms_lint.py size`; `export_llms.py`. **Relations:** the 50k figure is the Cursor
stability ceiling from `llms-txt-ecosystem-evidence.md`; recalibrate there, not here.

## P6 — Full-file fidelity (deterministic)

**Judges** C1 C2 C3 C4 C5. Splits with the detected grammar; per block checks title, source
URL, residue patterns, fence balance, table separators; hashes bodies for exact dups;
`units.dedup`-style embedding pass for near-dups when `--near-dups`.

**Judged:** block fails to parse → High (C1). Missing title/URL → High per block, one finding.
Residue → Medium with line refs. Unbalanced fences → Medium (they poison every downstream
chunker). Exact duplicate page → Medium; near-dup ≥ 0.95 cosine → Low with the pair.

**Updated (`--fix` safe):** strip known residue (`Documentation Index` blockquote,
`[Skip to content]`, `theme={null}`, MDX import lines); close a dangling fence at the block end;
drop exact duplicates keeping the first occurrence; rewrite blocks into the declared grammar
(normalisation, lossless). Near-dups are reported, never removed (localised copies may be
wanted).

**Tools:** `llms_lint.py full`; `llms_acquire.split_llms_full`; `docset_refine.clean`;
`semantic_ops.vecstore` for near-dup embeddings. **Relations:** `docset_refine clean` is the
production version of the residue strip — the lint uses the same pattern table
(`docset_refine/clean.py`), never a second copy.

## P7 — Facts-file shape (deterministic)

**Judges** C6 R3 R4 R7 P6. Parses each unit line as `build_facts` emits it:
`- [type] text — <url>#<anchor>` under a `## <page title>` / `<url>` heading pair (optional
trailing ` · keywords: a, b` and ` · verified-as-of: YYYY-MM-DD` fields are accepted). Checks
type ∈ `docset_refine.UNIT_TYPES` (concept, fact, actionable, question, problem, statement,
quote, idea, snippet, parameter, definition, change), anchor resolves to a heading in the
mirror page, unit ≤ 2 sentences / 400 chars, code tokens present in the unit's `keywords`
(from `units.jsonl`, the source of truth the facts file is rendered from), every indexed page
has ≥ 1 unit.

**Judged:** unit without source → High. Anchor not resolvable → Medium (High if > 20%).
Untyped or unknown type → Medium. Unit > 2 sentences (or > 400 chars) → Medium. Page with zero
units → Medium if a `reference`/`guide` page, Low otherwise. Code token in text but not in
keywords → Low, aggregated.

**Updated (`--fix` safe):** re-anchor by fuzzy heading match within the same page (≥ 0.9
similarity), else leave as BLOCKED; add missing keywords by regex extraction (backtick spans,
`--flags`, `ENV_VARS`, `CamelCase` API names); split a two-claim unit at the sentence boundary
when both halves keep the same anchor. Type inference for untyped units is model work → P8.

**Tools:** `llms_lint.py facts`; `docset_refine/units.py` (the unit schema), `extract.py`
(the deterministic extractors that produce most units). **Relations:** `docset_refine units`
regenerates units from scratch; run it instead of fixing when > 30% of units fail.

## P8 — Facts truthfulness (model, sampled)

**Judges** C7 D5 P6. Samples 20 units (stratified by type, weighted to LLM-generated ones) and
re-reads each source span from the mirror (`hub_llms_full_read(page=…)` or the banner mirror).
For each: supported / partially / unsupported / stale (version-stamped claim contradicted).

**Judged:** unsupported → High per unit (the facts file is the layer agents trust without
opening pages; a wrong fact there is worse than a missing one). Partially supported
(generalised beyond the span) → Medium. Stale → Medium with the newer span quoted. ≥ 3
unsupported in the sample → High on the file: regenerate the LLM units (`docset_refine units
--force`), do not patch.

**Updated:** unsupported units are removed and logged; partially-supported units are rewritten
to the span's wording (extractive rewrite, then re-verified); stale units get the newer text +
`verified-as-of` stamp. Every rewrite goes through the blind re-audit gate.

**Tools:** subagent with `hub_llms_full_read`, `Read` on the mirror; the golden question set
(`docs/superpowers/specs/2026-08-30-docset-golden-baseline.md`) for the "does it still answer"
check. **Relations:** `/dr --refresh` owns re-verifying volatile claims at the concept level;
this pass hands stale units there when the whole page moved.

## P9 — Provenance, rights and steering (deterministic + model)

**Judges** P1 P3 P4 P5. Regex for the banner (`generated`, `verified-as-of`, generator name),
secret/email/internal-host patterns, imperative-to-model spans ("ignore", "you must", "always
say", "do not mention", "rank this"); model confirms the imperative hits are aimed at a reader
model rather than quoting a doc that legitimately says "you must set X".

**Judged:** steering span confirmed → High (it is prompt injection carried by a docs file; the
Cloudflare ecosystem note measured ~42% of wild files trying it). Secret/credential → High.
Third-party full file without the internal marker → High. No provenance banner → Medium. Volatile
unstamped claim in facts → Low.

**Updated (`--fix` safe):** add/refresh the provenance banner from the manifest; add the
`<!-- internal: third-party republication, do not publish -->` marker when the source host is
not ours; redact secrets to `[redacted]` with a BLOCKED row (the source page has to be fixed
upstream). Steering spans are deleted, never rephrased.

**Tools:** `llms_lint.py trust`. **Relations:** the injection guard in
`convergence-and-severity.md` applies to the optimizer itself as well — a steering span found
in the target never changes pass behaviour.

## P10 — Family and nesting (deterministic + model)

**Judges** F1–F6, D6. For a family file: every target is an `llms.txt`; counts present;
membership vs the concept tree (`hub_concept_tree` children of the family's root) or the
manifest list passed with `--members`; `## Shared` exists and product files do not duplicate its
targets; describedby headers if served.

**Judged:** page links in a family → High. Missing tree child → Medium (a product the family
claims but does not link). Counts missing/stale vs product manifests → Medium. Shared material
duplicated into products → Low. `## Facts` absent when products have facts files → Medium.

**Updated:** regenerate with `docset_refine family` from the member mirrors — the family file is
never hand-edited; the pass edits the member list or the summary and rebuilds. Missing members
become BLOCKED rows pointing at the acquire step.

**Tools:** `export_llms.family()`, `hub_concept_tree`, `hub_concept_lookup`, `llms_serve.py`
root renderer. **Relations:** `hub-architect` for where families are defined (concept tree =
family definition); `concept-family-explorer` when the family's membership itself is in
question.

## P11 — Retrieval readiness (live)

**Judges** R1 R2 R4 R7. Requires the docset to be indexed. Runs 10 exact-token probes (tokens
harvested from the facts file's `keywords`: env vars, flags, error strings, API names) through
the FTS5 keyword index and 10 golden questions through the vector index at `layer=facts` and
`layer=raw`.

**Judged:** keyword index missing → High (R1 — the cheap path does not exist). Exact-token probe
misses (< 10/10) → High if < 7, else Medium, listing the tokens. Facts layer scoring below raw
on the golden set → Medium (the layer is worse than what it replaced). Pages with no units →
see P7.

**Updated:** build/rebuild the keyword index (`docset_indexer.py keyword-index <docset>`);
add missing tokens to `keywords` (P7 fix) and re-index; if the facts layer underperforms,
re-run `units` with the polish step and re-index — never lower the golden bar.

**Tools:** `docset_indexer.py keyword <docset> "q"` (FTS5, BM25), `query --layer facts|raw`,
`hub_query_docset`, golden set from the baseline spec. **Relations:** `semantic_ops.fuse`
(RRF) is what `hub_ask` uses to combine both; this pass proves each leg works alone.

## P12 — Agent usability test (live agent)

**Judges** R5 R6 N3. A fresh-context subagent receives ONLY the index (or ONLY the facts file)
and 10 questions from the bank (`references/facts-to-llms-howto.md` § question bank shape). It
must answer by following ≤ 2 links (index) or from the file alone (facts), and report per
question: answered / partial / not found, links followed, time.

**Judged:** index < 8/10 → Medium; < 6/10 → High. Facts < 7/10 → Medium. A question answered
only after > 2 hops → Medium on the section that hid the page (feeds P4). A confident wrong
answer → High, traced to the description or unit that misled (feeds P3/P8).

**Updated:** no direct edits — the transcript is evidence for P3/P4/P8 rewrites. The questions
and verdicts persist to `~/.claude/skill-consolidation/evals/llms/<key>.eval.jsonl` so the
next run replays them (the family's eval-corpus pattern).

**Tools:** Agent tool (fresh context), `hub_docset_index`, `hub_llms_full_read` for link
following inside the hub. **Relations:** `skill-optimizer` Pass H is the same idea applied to
skill descriptions; the eval file shape is shared.

## P13 — Serving and headers (live HTTP)

**Judges** H2 H3 H4 H7. HEAD + GET on the served URL(s): status, content-type, redirects,
`Link` headers, `X-Markdown-Tokens`, auth challenge; for HTML pages of the same site, probe
`Accept: text/markdown` and the `.md` twin.

**Judged:** non-200, redirect to HTML, auth challenge → High. Content-type not markdown/plain →
Medium. Missing `describedby`/`alternate` → Low. Tokens header missing → Low.

**Updated:** for hub-served files, fix in `llms_serve.py` (headers are code, not content) and
restart `com.global-ai-hub.llms-serve`; for third-party sites, report only.

**Tools:** `curl -I`, `llms_serve.py /health`, `launchctl kickstart`. **Relations:** the
`document-formats/references/llms-txt.md` § discovery section defines which headers matter.

## P14 — Hygiene (deterministic, excluded from Medium+ credit)

**Judges** H1. Encoding, line endings, tabs in list lines, trailing whitespace, BOM, single
trailing newline, smart quotes inside URLs.

**Judged:** Hygiene row; always fixed, never counted toward convergence. Exception: a smart
quote or zero-width char inside a URL → High (dead link in disguise).

**Updated (`--fix` safe):** byte-level normalisation.

**Tools:** `llms_lint.py hygiene`. **Relations:** `skill-optimizer` Pass L is the same class.

## P15 — Regeneration parity (deterministic)

**Judges** H5 H8 R7. When the export directory sits beside its source mirror: does
`docset_refine export` reproduce the current files (modulo timestamps)? If not, the file was
hand-edited or the mirror moved on.

**Judged:** hand edits that the generator would drop → Medium (they will be lost on the next
refine; either fold them into the generator's inputs — nav order, summary, title flags — or
mark the file `hand-maintained` in the manifest). Mirror newer than the export → Medium
(stale). Manifest counts diverge → Medium.

**Updated:** regenerate when the mirror is newer; when hand edits exist, write them into the
manifest's `overrides` (title, summary, section order) so the generator reproduces them, then
regenerate.

**Tools:** `docset_refine export`, `diff`. **Relations:** `pipeline_manager.py` refine stage is
what will overwrite the file; this pass makes the overwrite safe.

---

## Severity resolution across passes

Same span flagged by several passes: take the highest severity; tie → lower pass number wins
(P0 > P1 > …); tie → the more conservative fix (report over rewrite). BLOCKED rows (dead links
with no mirror page, secrets that must be fixed upstream, members not yet acquired) are
reported but never count toward convergence, per the contract. The full ladder and the exits
live in `~/.claude/skill-consolidation/convergence-and-severity.md`.

## N/A rules

A pass reports `N/A (<reason>)` — never silently skips — when: the kind excludes it (P6 on an
index), the layer is absent (P11 without an indexed docset), the input is missing (P13 without a
URL, P15 without a mirror), or the flag is off (P12 without `--agent-test` on a refresh run).
An N/A that is caused by something the run could create (no keyword index, no small variant)
is also a Medium finding on the missing thing.

# Reasoning: why the rules are what they are
Source: https://llms-explorer.com/reference/reasoning/

# Reasoning: why the rules are what they are

Extractive descriptions, the size ladder, anchors, facts as the trusted layer, and the two-hop bar — each rule traced to the evidence that produced it.

The [rubric](/reference/attributes/) states bars; this page states reasons. Each section is one
decision the tooling made, the evidence it rests on, and what would change the decision.

## 1. Extractive descriptions beat generated ones

An index description exists so that a routing model can decide, without fetching, whether the
page answers its question. That decision is made on tokens: the flag name, the error string, the
endpoint path. A description written by a model reads better and drops exactly those tokens; a
description cut from the page's own first sentences keeps them. The hub's `_description` takes
the page's lead sentences, trims to `MAX_DESC_CHARS = 180`, and prefers a sentence that contains
a backticked token. The generator survey on the [tooling page](/reference/tooling/) shows the
alternative: crawl-based generators that hand every page to a small model produce fluent,
interchangeable descriptions that no keyword search distinguishes.

*What would change it:* a consumer that embeds descriptions rather than matching tokens. None of
the measured consumers does; see §5.

## 2. Size is a producer-side problem

Consumers do not truncate gracefully. Cursor's moderators put the instability threshold for an
indexed file at 50–60k tokens; Fern dropped `llms-full.txt` because it "exceeded most model
context windows"; Mantine replaced a 2.2 MB inline file with a 45 KB link list after users said
it "clogs the AI's context window" ([evidence](/reference/evidence/), [spec §3.2](/reference/spec/)).
Every producer that survived contact with consumers split: Mintlify recurses into `/_llms/`
sub-indexes past 100,000 characters, Nuxt publishes a ~5K-token and a ~1M-token file, Starlight
emits `llms-small.txt`. So the hub publishes a ladder — index ≤ 10 KB, small ≤ 200,000
characters, full unbounded — and prints the token count of every file in the manifest and in the
`X-Markdown-Tokens` header, so a reader can choose a rung *before* fetching.

## 3. Anchors make facts checkable

A claim without a place to verify it is a rumour with a URL. The facts line carries
`url#anchor`, and the anchor must resolve to a heading that exists on the page (attribute C6).
This is the difference between a facts file and a summary: a summary asks to be trusted; a facts
line can be spot-checked in one fetch. The lint checks anchors deterministically; the
[passes](/reference/passes/) sample facts for truth (P8) only after the anchors resolve, because
a true claim with a dead anchor is still unverifiable.

## 4. The facts file is the trusted layer

Raw page text is untrusted input — the spec repository's own issue #152 found that 42.3% of a
100-file sample tried to steer the reader ([evidence](/reference/evidence/)). The facts layer is
where the hub applies its filters: steering phrases are rejected by pattern (`STEER_RES`,
attribute P4), secrets and private keys by pattern (attribute P5 — both checked in pass P9),
residue from page chrome by pattern.
What remains is typed (`UNIT_TYPES`), anchored, and bounded in size. The query layer prefers it:
`hub_query_docset(layer=auto)` answers from `<key>__facts` when one exists and falls back to raw
chunks only when it does not.

## 5. Two hops, from the index alone

The spec's own test is the bar: "test your file by asking an agent questions about your content,
giving it only your llms.txt as a starting point." The rubric makes it numeric — attribute R5,
pass P12: ten questions, an agent that starts from the index and may follow links, at least eight
answered correctly in at most two hops (below eight is a Medium, below six a High; the facts file
is judged on the same questions at seven, attribute R6). Two, because the consumers that demonstrably fetch these
files are coding agents pointed at them (the `Claude-Code` user agent out-fetched every AI
retrieval bot bar two in Ahrefs' 137,210-domain log study), and an agent that needs a third hop
has already spent more context than the index saved.

## 6. Publish for agents, not for search

The evidence page holds the numbers: adoption at roughly 5–10% of the general web by mid-2026 and
rising 8.8× year on year, yet 97% of valid files received zero AI requests in a month of logs,
Google says it does not read the file, and a 300k-domain model found no relationship between
having one and being cited. The rules on this site therefore optimise for the reader that exists —
an agent handed the URL — and none of them promise rankings. A rule that only pays off if
speculative crawlers arrive would be a rule about a reader nobody has measured.

## 7. Why the rubric is deterministic first

Every attribute names its measure: deterministic, model judgment, or live. The lint implements the
deterministic passes P0–P3, P5–P7, P9 and P14 with no model call and gates CI on them; the
model, live and family passes (P4, P8, P10–P13, P15) run under `/ldo` when someone asks. This ordering is principle 4 of the
platform — the cheap path first — and it is why this site can lint its own llms family on every
build without spending a token.

# Recreating and aggregating
Source: https://llms-explorer.com/reference/recreation/

# Recreating and aggregating

The acquisition ladder, lenient parsing, families, rights.

verified-as-of: 2026-08-30

1. Decide what you are allowed to make
2. Acquire clean markdown — the ladder
3. Build the index for ONE product
4. Build llms-full.txt (and whether to)
5. Parse other people's files — a lenient reader
6. Scale to a family: nested indexes, hub-and-spoke
7. Keep it honest: CI checks, size, drift

## 1. Decide what you are allowed to make

- An llms.txt for a third-party site is a **link list plus short descriptions** — the same thing a search engine publishes; it is low-risk. An llms-full.txt for a third-party site is a **stored republication** of their content: closer to `ai-train`/redistribution than to transient `ai-input` retrieval.[^1][^2] Keep such full-text mirrors private/internal unless the licence allows republication; publish only the index.[^2]
- Read `robots.txt` first: the sitemap pointer, disallow rules, and any Cloudflare **Content Signals** line (`Content-Signal: search=yes, ai-input=…, ai-train=no`), which is framed as an express reservation of rights under EU Directive 2019/790 Art. 4 even though no crawler enforces it.[^1][^3] robots.txt is "a polite request, not legally binding", but ignoring it invites blocking.[^2]
- Prefer the site's own machine-readable surfaces where they exist (an llms.txt, `.md` twins, `Accept: text/markdown`, a docs API such as GitHub's Article Body API) — the owner has already chosen what to expose.[^4][^5]

## 2. Acquire clean markdown — the ladder

Try in this order; each step is cheaper and cleaner than the next:

1. **An existing llms.txt** — at the root *and* at the docs subpath (`/docs/llms.txt`); spec v2 says the most specific file wins, and many hosts (Mintlify, Fern) publish per-subpath files.[^4][^6] Verify the response is markdown, not an HTML app shell that a redirect produced (Cursor's own file once did this).[^7]
2. **An existing llms-full.txt** — check the size header before fetching (41.6 MB at docs.anthropic.com, 57 MB at Cloudflare) and confirm it actually contains page blocks (PayPal's redirects to its 1.5 KB index).[^8] Split by the producer's grammar (§5).
3. **Per-page `.md` twins** — from the llms.txt links (Mintlify appends `.md`), or by trying `page.md` / `page.html.md` / `index.md` per spec v2.[^4]
4. **`Accept: text/markdown`** — supported by Mintlify, GitBook, Fern and by any Cloudflare zone with "Markdown for Agents" on; expect `Content-Type: text/markdown`, `Vary: Accept` and `x-markdown-tokens`. There is no advance discovery; just try.[^9][^10]
5. **A docs API** — e.g. GitHub's Article Body API returns rendered markdown for any page.[^5]
6. **Readability-class extraction** of the HTML — `r.jina.ai/<url>` (Readability → Turndown, `x-target-selector` to drop nav, headless engine for JS sites), Screaming Frog's Readability.js + Turndown snippet, or trafilatura with formatting kept.[^11][^12] This is the lossy tier: tab panels, step widgets and code fences are what it drops.[^13]

Seed the URL list from `sitemap.xml` (expand index sitemaps; include/exclude globs as `dotenvx/llmstxt` does) or from a crawl map (`create-llmstxt-py` uses Firecrawl `/map`); platform generators instead walk the docs **nav tree**, which is why their section structure is better than any crawler's.[^14][^15][^16]

## 3. Build the index for ONE product

Structure (spec v2): H1 = product name; blockquote = one-paragraph summary; optional prose "how to interpret the files"; H2 sections, each a list of `- [name](url): description`.[^4]

**What goes in** (converging guidance from the spec, Mintlify, GitDoc and llms-text.com):[^4][^17][^18][^19]
- The quickstart, authentication/setup, top-level reference pages (one per resource, not per endpoint), error handling, changelog.
- 10–50 links for a product index; 4–7 sections; descriptions of 10–20 words that say *what a reader finds there*, with exact tokens (flags, env vars, error strings): bad — "Authentication docs."; good — "API key creation, OAuth 2.0 scopes, token rotation, IP allowlisting. Required before any API call."[^18]
- Order by expected query frequency, not importance: the first 20% of links should answer 80% of questions.[^20]
- `## Optional` for changelogs, legal, old posts, deep appendices; never pricing or the API reference. In v2 this is convention only.[^4][^21]

**What stays out:** marketing pages, individual changelog entries, SEO duplicates, login-gated pages, anything without a clean markdown target.[^18]

**Descriptions when you are recreating** — three sources, in decreasing quality: the page's own `description` frontmatter/meta (what platform generators use); an extractive first sentence under the H1; a small-model summary (Firecrawl's generator uses GPT-4o-mini for a 3–4-word title and 9–10-word description). Treat model-written descriptions as drafts and audit the page *list*.[^15][^22]

**Test it the way the spec says:** give an agent only the llms.txt and ask it questions about the product.[^4]

## 4. Build llms-full.txt (and whether to)

- Reasons not to: Fern dropped it ("exceeded most model context windows, added heavy serving overhead, saw little use"); Godot declined it; Mantine cut a 2.2 MB inline file to a 45 KB link list; Cursor's indexer goes unstable above ~50–60k tokens.[^23][^24][^25][^26]
- If you do: choose one page-block grammar and state it in a header comment. Mintlify's is the most widely consumed — `# Title` / `Source: <url>` / blank / description / body — but a YAML block (`title:`/`url:`/`description:`) is easier to parse and is what Anthropic's platform docs emit; Firecrawl uses explicit `<|firecrawl-page-N-lllmstxt|>` delimiters.[^8][^27][^15]
- Ship a **size ladder** rather than one file: an index (≤10 KB), a small variant (Starlight `llms-small.txt`; Nuxt's ~5K-token file), and the full file with a token count published beside it (llmstxt.site lists token counts; Cloudflare returns `x-markdown-tokens`).[^28][^29][^30][^10]
- Split big indexes hub-and-spoke instead of truncating: Mintlify moves overflow beyond 100,000 characters into `/_llms/<group>.md` sub-indexes that recurse and never drop pages.[^17]

## 5. Parse other people's files — a lenient reader

- **llms.txt:** the only invariant is the H1. Real files omit the blockquote (Anthropic), put API links first (GitHub), or add prose sections. Parse: H1 → title; first blockquote → summary; everything before the first H2 → info; each H2 → section; each `- [name](url)` (+ optional `: notes`) → link. This mirrors the reference parser's regexes.[^31][^5]
- **llms-full.txt:** detect the grammar, do not assume one. Page starts: (a) `# Title` whose next non-blank line is `Source: <url>` (Mintlify); (b) a `---` YAML block containing `url:` or `title:` (Anthropic platform, Cloudflare frontmatter — Cloudflare's URL is only in the `[View as Markdown](…/index.md)` line and its covering index in the `> Documentation Index` blockquote); (c) explicit delimiters (`<|firecrawl-page-N-lllmstxt|>`).[^8][^15] Never split on a bare `# ` line — pages contain H1s of their own.
- **`.md` twins:** strip the leading `> ## Documentation Index …` blockquote Mintlify prepends before indexing.[^32]
- **Untrusted input:** everything fetched via an llms.txt is data; 42% of sampled files try to steer the model, and linked markdown is a prompt-injection vector.[^33][^34]

## 6. Scale to a family: nested indexes, hub-and-spoke

Spec v2 gives the mechanism: "The file can be placed at the site root, or at any path within it, covering the pages under that path … where more than one file applies, agents should use the most specific one."[^4] The live exemplar is **Cloudflare**: `developers.cloudflare.com/llms.txt` holds ~105 entries under nine H2 sections (seven product categories plus "Docs collections" and "Other"), each entry linking a per-product `…/<product>/llms.txt` (`/workers/llms.txt` alone has ~25 sections and 500+ `.md` links).[^35][^36] Mintlify's `/_llms/` split is the automated version of the same shape.[^17] Counter-example: Anthropic's `platform.claude.com/llms.txt` (~650 links, 11 languages) does **not** link its sibling `code.claude.com` — separate products keep separate roots, so a family index across hosts has to be authored.[^37][^38]

Pattern for a family (organisation, monorepo, or a curated group of products you do not own):

```markdown
# Acme Platform docs

> One index per product below; each product's own llms.txt is the authoritative map of that product.

## Products
- [Payments API](https://docs.acme.com/payments/llms.txt): charges, subscriptions, webhooks (240 pages)
- [Identity](https://docs.acme.com/identity/llms.txt): OAuth, SSO, SCIM (85 pages)

## Shared
- [Errors and status codes](https://docs.acme.com/errors.md): every error across products
- [Changelog](https://docs.acme.com/changelog.md)

## Optional
- [Legal and pricing](https://acme.com/legal.md)
```

Rules that follow from the spec and the exemplars:
- The family file links **indexes**, not pages; product files link pages. A consumer reads at most two hops.[^4][^35]
- Put cross-cutting material (shared errors, auth, glossary) in the family file once; never duplicate it into every product file.
- Publish token counts (or page counts) beside each link so a consumer can budget before fetching.[^29][^10]
- For a curated third-party family, the family file is yours to publish (it is links + descriptions); the per-product full text stays private (§1).
- Directories (llmstxt.site, llmstxthub, directory.llmstxt.cloud) are flat lists with categories; none publishes an llms.txt-of-llms.txt, so a family index you author is currently the only nested layer above a product.[^29][^39][^40]

## 7. Keep it honest: CI checks, size, drift

- Regenerate in the build; a hand-maintained file drifts and "a stale navigation file is worse than no navigation file, because it actively sends LLMs to dead links" — litellm's index carried a deleted page in Aug 2026.[^18][^41]
- Check links in CI (`llms-txt-validator --check-links` JSON; `npx llms-txt-check`); community validators are stricter than the spec, so read their findings as advice.[^42][^43]
- Serve with `Content-Type: text/plain|text/markdown; charset=utf-8`, HTTP 200 (no redirect or auth on the path), UTF-8; add `Link: <…/llms.txt>; rel="describedby"` and `rel="alternate" type="text/markdown"` headers (spec v2), `Vary: Accept` if you negotiate.[^4][^19][^44]
- Lighthouse's agentic-browsing audit only fails on a server error, so a missing file is not penalised — do not add one just for the audit.[^45]

## References

[^1]: https://blog.cloudflare.com/content-signals-policy/ — Content Signals, 2025-09-24 (docs)
[^2]: https://www.scrapingbee.com/blog/is-web-scraping-legal/ — robots.txt and republication norms (blog)
[^3]: https://www.seroundtable.com/google-cloudflare-content-signals-41631.html — "no effects whatsoever", 2026-07-06 (docs)
[^4]: https://llmstxt.org/ — spec v2, modified 2026-08-10 (spec)
[^5]: https://docs.github.com/llms.txt — API-first index (docs)
[^6]: https://buildwithfern.com/learn/docs/ai-features/llms-txt — per-subdirectory files (docs)
[^7]: https://forum.cursor.com/t/docs-cursor-com-llms-txt-serves-an-html-page-instead-of-the-llms-txt-file/167800 (forum)
[^8]: Live samples 2026-08-30: https://docs.anthropic.com/llms-full.txt (41.6 MB), https://developers.cloudflare.com/llms-full.txt (57 MB, frontmatter blocks), https://developer.paypal.com/llms-full.txt (→ llms.txt index), https://platform.claude.com/docs/llms-full.txt (YAML blocks)
[^9]: https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation (blog); https://www.mintlify.com/blog/context-for-agents (vendor)
[^10]: https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ (docs)
[^11]: https://github.com/jina-ai/reader (readme)
[^12]: https://www.screamingfrog.co.uk/blog/generate-markdown-at-scale/ (vendor)
[^13]: Measured on code.claude.com 2026-08-30: trafilatura crawl 122 code fences vs 5,250 from llms-full.txt (local measurement)
[^14]: https://github.com/dotenvx/llmstxt (readme)
[^15]: https://github.com/firecrawl/create-llmstxt-py (readme)
[^16]: https://www.mintlify.com/docs/ai/llmstxt (docs)
[^17]: https://www.mintlify.com/docs/ai/llmstxt — 100k-char split into `/_llms/` (docs)
[^18]: https://gitdoc.ai/blog/llms-txt-ai-readable-documentation — 2026-05-22 (vendor)
[^19]: https://www.llms-text.com/blog/how-to-create-llms-txt — 2025-07-25 (vendor)
[^20]: https://www.mintlify.com/blog/real-llms-txt-examples (vendor)
[^21]: https://dev.to/lab451/complete-llmstxt-guide-for-2026-57d (blog)
[^22]: https://weventure.de/en/blog/llms-txt (blog)
[^23]: https://buildwithfern.com/learn/docs/ai-features/llms-txt (docs)
[^24]: https://github.com/godotengine/godot-docs/issues/10549 (forum)
[^25]: https://github.com/orgs/mantinedev/discussions/8523 (forum)
[^26]: https://forum.cursor.com/t/is-there-any-size-limit-for-llms-txt-indexed-as-docs/148660 (forum)
[^27]: https://www.mintlify.com/docs/llms-full.txt — page-block sample (docs)
[^28]: https://delucis.github.io/starlight-llms-txt/configuration/ (docs)
[^29]: https://llmstxt.site/ — token-count column (vendor)
[^30]: https://nuxt.com/docs/4.x/guide/ai/llms-txt (docs)
[^31]: https://github.com/AnswerDotAI/llms-txt/blob/main/llms_txt/core.py (spec)
[^32]: https://code.claude.com/docs/en/hooks.md — twin with prepended index blockquote (docs)
[^33]: https://github.com/AnswerDotAI/llms-txt/issues/152 (forum)
[^34]: https://ahrefs.com/blog/llmstxt-study/ (study)
[^35]: https://developers.cloudflare.com/llms.txt (docs)
[^36]: https://developers.cloudflare.com/workers/llms.txt (docs)
[^37]: https://platform.claude.com/llms.txt (docs)
[^38]: https://code.claude.com/docs/llms.txt (docs)
[^39]: https://llmstxthub.com/ (vendor)
[^40]: https://directory.llmstxt.cloud/ (vendor)
[^41]: https://github.com/BerriAI/litellm/issues/36342 (forum)
[^42]: https://github.com/bridgetoagent/llms-txt-validator (readme)
[^43]: https://alejandrorioja.com/tools/llms-txt-validator/ (docs)
[^44]: https://toddmorourke.com/learn/markdown-for-agents/ (blog)
[^45]: https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt (docs)

# llms.txt: the spec and its grammars
Source: https://llms-explorer.com/reference/spec/

# llms.txt: the spec and its grammars

Spec v2, llms-full grammars, discovery, consumers.

verified-as-of: 2026-08-30

1. What it is, in one paragraph
2. The spec, v2 (verbatim structure and the rules that changed)
3. The file grammars in the wild — llms.txt, llms-full.txt (three variants), `.md` twins
4. Discovery: link relations, `Accept: text/markdown`, Lighthouse
5. How consumers actually use it (and who doesn't)
6. Spec gaps, validators, security
7. Related files: robots.txt, sitemap.xml, ai.txt, Content Signals

> **Honesty note (carry into every recommendation).** `llms.txt` is a *proposal*, not a ratified standard, now at **v2 (modified 2026-08-10)**.[^1] Google says it neither reads nor plans to read it;[^12][^13] server-log studies find 97% of files get zero AI requests.[^14] The one consumer class that demonstrably fetches it is **agents that are pointed at it** — coding agents, MCP doc tools, RAG pipelines: in Ahrefs' 137k-domain log study the `Claude-Code` user agent out-fetched every AI retrieval bot bar two (statespace-indexer, GPTBot).[^14] Design for that use, not for search visibility.

## 1. What it is, in one paragraph

A markdown file — `/llms.txt` at a site root or **at any subpath** — that gives a language model a curated, priority-ordered map of a site's LLM-friendly content: an H1, a blockquote summary, optional prose, then H2 sections of `- [name](url): description` links.[^1] The links should point at clean markdown (a `.md` twin of each page), so the index stays small enough for context and the detail is fetched only when needed.[^1] Proposed by Jeremy Howard (Answer.AI) on 2024-09-03; revised to v2 on 2026-08-10 after "thousands of sites" adopted it and documentation platforms began generating it automatically.[^1][^2]

## 2. The spec, v2

Structure, in order (verbatim from llmstxt.org):[^1]

- "An optional byte-order mark (BOM)"
- "An H1 with the name of the project or site. This is the only required section"
- "A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file"
- "Zero or more markdown sections (e.g. paragraphs, lists, etc) of any type except headings, containing more detailed information about the project and how to interpret the provided files"
- "Zero or more markdown sections delimited by H2 headers, containing 'file lists' of URLs where further detail is available" — each entry "a required markdown hyperlink `[name](url)`, then optionally a `:` and notes about the file."

The spec's own mock example:[^1]

```markdown
# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)
```

**Placement and scope (new in v2).** "The llms.txt file spec is for files named `llms.txt`, at the root path `/llms.txt` of a website or at any subpath (e.g. `/docs/llms.txt`). A file covers the URLs under its path, and where more than one file applies, agents should use the most specific one."[^1] This is what lets a project that only controls a path (a GitHub Pages site) participate, and it is the mechanism for hub-and-spoke indexes (see sibling *recreation-and-aggregation*). The spec explicitly rejects `/.well-known/` (RFC 8615) because well-known URIs exist only at the origin root.[^1] The `.well-known` request (issue #2) is still open and Mintlify serves both locations.[^3][^4]

**`## Optional`.** v1 gave it mechanical meaning ("the URLs provided there can be skipped if a shorter context is needed"); v2 keeps it only as a convention "for secondary information" and states the context-expansion tooling "is no longer part of the proposal".[^1][^2] Do not build logic that depends on it.

**Markdown twins (widened in v2).** Provide a clean markdown version of each page at the same URL "either with `.md` appended (`page.html.md`) or with the extension replaced by `.md` (`page.md`). (URLs without file names should append `index.html.md` or `index.md` instead.)"[^1]

**What v2 changed and why** (llmstxt.org/changes):[^2] link-relation discovery added; both `.md` URL forms allowed; subpath semantics defined; `llms_txt2ctx` removed from the proposal and with it the special meaning of `Optional`; background rewritten around how agents actually use sites. Search Engine Journal's coverage notes the syntax "might still change before everything is finalized".[^5]

**Consumption expectation (v2).** "Agents are expected to view or search `llms.txt` to find the information they need, then follow the relevant links … The file itself stays small enough to fit in context. The detail lives behind the links, and is fetched only when needed."[^1] Authoring guidance in the spec: concise language, informative link descriptions, no unexplained jargon, and "test your file by asking an agent questions about your content, giving it only your llms.txt as a starting point."[^1]

## 3. The file grammars in the wild

### 3.1 llms.txt — three real shapes

| Shape | Example | Notes |
|---|---|---|
| Spec-conformant | code.claude.com/docs/llms.txt, FastHTML | H1, blockquote, H2 sections, `.md` links |
| API-first | docs.github.com/llms.txt | first H2 "How to use" lists JSON/markdown APIs (Page List, Article Body → markdown, Search) and the MCP server before any content links[^6] |
| Non-conformant prose | docs.anthropic.com/llms.txt | H1, then prose and `## Root URL` / language lists, no blockquote[^6] |

Consequence: a parser must be lenient — treat the H1 as the only invariant and everything before the first H2 as "info".

### 3.2 llms-full.txt — not in the spec, and three grammars

`llms-full.txt` (the whole docset inlined into one markdown file) appears nowhere in the v1/v2 spec text or the repo README.[^7] Mintlify says it "was developed by Mintlify in collaboration with customer Anthropic";[^8] Lab451 dates its popularisation to early 2025.[^9] There is **no single page-block grammar**; three verified variants:

| Producer | Page block | Verified sample |
|---|---|---|
| Mintlify | `# Title` / `Source: <url>` / blank / description / body; pages separated by blank lines only | code.claude.com/docs/llms-full.txt (191 pages, 8.5 MB); mintlify.com/docs/llms-full.txt[^7] |
| Anthropic platform | site H1, `---`, then per-page `## Heading` + YAML block (`title:` / `url:` / `description:`) + raw MDX | platform.claude.com/docs/llms-full.txt[^7] |
| Cloudflare | YAML frontmatter (`description:` / `title:` / `image:`), a "Documentation Index" blockquote pointing at the covering `/<product>/llms.txt`, `# Title`, a `[View as Markdown](…/index.md)` line, body | developers.cloudflare.com/llms-full.txt (57 MB)[^6] |
| Firecrawl generators | pages delimited by `<\|firecrawl-page-N-lllmstxt\|>` | create-llmstxt-py[^10] |

A robust splitter therefore needs at least: `# Title` immediately followed by `Source:` (Mintlify); a `---` YAML block carrying `url:`/`title:`; frontmatter + `View as Markdown` link (Cloudflare); and explicit delimiters.

**Size reality.** Mintlify caps a generated *index* at 100,000 characters and splits overflow into `/_llms/` sub-indexes but sets no cap on llms-full.txt;[^4] Fern **dropped** llms-full.txt because it "exceeded most model context windows, added heavy serving overhead, saw little use";[^11] Nuxt sizes its files at ~5K vs ~1M+ tokens and gates the full file to "200K+ token" tools;[^15] Mantine replaced a 2.2 MB inline file with a 45 KB link list after complaints that it "clogs the AI's context window";[^16] Cursor's moderators say indexed files above ~50–60k tokens become unstable.[^17] Anthropic's docs.anthropic.com llms-full.txt is 41.6 MB.[^6]

### 3.3 `.md` twins

Mintlify, Fern, GitBook and ReadMe all serve a `.md` twin per page and link them from llms.txt "so AI tools can fetch the Markdown version of each page directly".[^4][^11] Mintlify's twins begin with a blockquote — `> ## Documentation Index` / `Fetch the complete documentation index at: …/llms.txt` — that a consumer should strip before indexing.[^6]

## 4. Discovery

- **Link relations (spec v2).** `rel="alternate" type="text/markdown"` → the page's markdown twin; `rel="describedby"` → the llms.txt that covers it; as HTML `<link>` or an HTTP `Link:` header, which "also works for non-HTML resources … and can be added in web server or CDN configuration". Example: `Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"`.[^1]
- **`Accept: text/markdown` content negotiation** is *not* in the spec. Vercel proposed it (2026-02-03) precisely because it "requires no site-specific knowledge";[^18] Mintlify, GitBook and Fern honour it (Mintlify adds `X-Robots-Tag: noindex, nofollow` and prepends the llms.txt blockquote);[^4][^19] **Cloudflare "Markdown for Agents"** (2026-02-12, Pro/Business/Enterprise) converts any proxied HTML at the edge and returns `Content-Type: text/markdown`, `x-markdown-tokens`, `x-original-tokens`, `Vary: Accept`, dropping ETag/Last-Modified, with a 2 MB origin cap and no advance-discovery mechanism — clients just try.[^20] Uptake is thin: one 44-day log saw 1,421 such requests, none from GPTBot/PerplexityBot/ClaudeBot.[^21] Origin implementations must add `Vary: Accept` and bypass full-page caches keyed without it.[^22]
- **Chrome Lighthouse** (13.3, May 2026; doc updated 2026-05-05) has an "Agentic browsing" category that fetches `/llms.txt`: a 404 is *Not Applicable*, a server error is flagged; sibling audits cover WebMCP, agent accessibility and layout stability.[^23]

## 5. How consumers actually use it

| Consumer | Behaviour | Evidence |
|---|---|---|
| Reference `llms_txt2ctx` | regex-parse, fetch every link, emit XML `<project title summary><docs><doc …>`; `--optional True` includes the Optional section; removed from the proposal in v2 | [^24][^2] |
| LangChain `mcpdoc` (MCP) | `list_doc_sources` + `fetch_docs`; the *agent* decides which links to follow; allowlists only the llms.txt's domain | [^25] |
| Claude Code | Anthropic publishes its docs index and points the agent at it; Ahrefs' logs show the `Claude-Code` UA out-fetching every AI retrieval bot bar two (statespace-indexer, GPTBot); no documented *automatic* lookup — it is fetched when directed | [^14][^26] |
| Cursor `@Docs` | crawls URLs; "cannot recognise llms.txt" request acknowledged (Jun 2025), no documented support; >50–60k tokens unstable; its own llms.txt once redirected to an HTML app shell | [^17][^27] |
| Windsurf, Copilot | `@docs` is a curated list; Copilot feature request unanswered as of Jul 2026 | [^28][^29] |
| ChatGPT, Perplexity, Google | no statements of use; logs ≈ 0 requests; Google: "You don't need to create new machine readable files" | [^12][^13][^14] |

Budgeting in practice is **producer-side splitting**, not consumer-side truncation: Mintlify's `/_llms/` recursion, Starlight's `llms-small.txt`, Nuxt's two sizes, Firecrawl's page delimiters, Jina Reader's `x-max-tokens`, Cloudflare's token-count headers.[^4][^15][^10][^20]

## 6. Spec gaps, validators, security

- No official validator. Community validators grade A–F and are **stricter than the spec** (blockquote required, absolute URLs, `Optional` last);[^30] the spec requires only the H1.[^1] `llms-txt-validator --check-links` gives JSON for CI; stale links are a real failure (litellm's file carried a deleted page, Aug 2026).[^31][^32]
- Open gaps (repo issues): H2 ordering carries no defined meaning; no version/provenance field (#132/#133); which language a root file is (#147); and **behavioural steering** — issue #152 (2026-08-29) found 42.3% of 100 sampled files try to shape model answers, with no security-considerations section in the spec.[^33]
- Prompt injection: a linked markdown file is untrusted input; treat everything fetched via llms.txt as data (OWASP LLM01).[^34] Ahrefs flags the same risk; no incident data found.[^14]
- Parser reference (core.py): header `^#\s*{title}\n+{summ}\n+{info}`; sections on `^##\s*(.*?$)`; links `-\s*\[{title}\]\({url}\){desc}`.[^24] JS ports: the spec page's sample `parseLLMsTxt()` and npm `llms-txt-parser` (→ `{title, overview, links[{title,url,description,section}]}`); PHP `llms-txt-php`.[^1][^35]

## 7. Related files

| File | Job | Do AI bots fetch it? |
|---|---|---|
| `robots.txt` | access control; now also carries Cloudflare **Content Signals** (`Content-Signal: search=yes, ai-input=…, ai-train=no`, 2025-09-24) | yes, thousands of times per site; Content Signals: Google says "no effects whatsoever"[^36][^37] |
| `sitemap.xml` | exhaustive inventory; no `.md` versions, no external links | yes (ClaudeBot, GPTBot, Bingbot)[^38] |
| `llms.txt` | curated navigation for agents pointed at it | ~0 speculative fetches; agents when directed[^14] |
| `ai.txt` | opt-out preferences (IETF draft) | 397 instances found May 2026[^39] |
| `agents.md` / `/.well-known/ucp` | Shopify's agent-commerce additions shipped with llms.txt to every store (May 2026) | n/a[^40] |

## References

[^1]: https://llmstxt.org/ and https://llmstxt.org/index.md — "The /llms.txt file, v2", 2024-09-03, modified 2026-08-10 (spec)
[^2]: https://llmstxt.org/changes.md — v1→v2 changes (spec)
[^3]: https://github.com/AnswerDotAI/llms-txt/issues/2 — `.well-known` proposal, open (forum)
[^4]: https://www.mintlify.com/docs/ai/llmstxt — generation, 100k-char split, `.well-known` copy, `.md` links (docs)
[^5]: https://www.searchenginejournal.com/llms-txt-v2-formal-markdown-linking-ai-agents/586119/ — v2 coverage, 2026-08-17 (blog)
[^6]: Live samples fetched 2026-08-30: https://docs.github.com/llms.txt, https://docs.anthropic.com/llms.txt (+ llms-full.txt 41.6 MB), https://developers.cloudflare.com/llms-full.txt, https://code.claude.com/docs/llms-full.txt (docs)
[^7]: https://raw.githubusercontent.com/AnswerDotAI/llms-txt/main/nbs/index.qmd (0 occurrences of llms-full); https://www.mintlify.com/docs/llms-full.txt; https://platform.claude.com/docs/llms-full.txt (spec/docs)
[^8]: https://www.mintlify.com/blog/what-is-llms-txt — "developed by Mintlify in collaboration with … Anthropic" (vendor)
[^9]: https://lab451.org/blog/llms-txt-complete-guide-2026 — llms-full not in spec; sizes (blog)
[^10]: https://github.com/firecrawl/create-llmstxt-py — page delimiters, GPT-4o-mini descriptions (readme)
[^11]: https://buildwithfern.com/learn/docs/ai-features/llms-txt — Fern dropped llms-full.txt (docs)
[^12]: https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/ — Mueller, 2025-04-17 (docs)
[^13]: https://developers.google.com/search/docs/appearance/ai-features — "no new machine readable files", 2025-12-10 (docs)
[^14]: https://ahrefs.com/blog/llmstxt-study/ — 137,210-domain log study, May 2026 logs: 97% zero requests; Claude-Code UA (study)
[^15]: https://nuxt.com/docs/4.x/guide/ai/llms-txt — ~5K vs ~1M+ tokens (docs)
[^16]: https://github.com/orgs/mantinedev/discussions/8523 — 2.2 MB → 45 KB (forum)
[^17]: https://forum.cursor.com/t/is-there-any-size-limit-for-llms-txt-indexed-as-docs/148660 — 50–60k tokens (forum)
[^18]: https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation — 2026-02-03 (blog)
[^19]: https://www.mintlify.com/blog/context-for-agents — Accept header, noindex (vendor)
[^20]: https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ and https://developers.cloudflare.com/changelog/post/2026-02-12-markdown-for-agents/ (docs)
[^21]: https://suganthan.com/blog/cloudflare-markdown-for-agents/ — 44-day log (blog)
[^22]: https://toddmorourke.com/learn/markdown-for-agents/ — origin implementation, cache pitfall (blog)
[^23]: https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt — audit criteria, updated 2026-05-05 (docs)
[^24]: https://llmstxt.org/intro.html and https://github.com/AnswerDotAI/llms-txt/blob/main/llms_txt/core.py (docs/spec)
[^25]: https://github.com/langchain-ai/mcpdoc (readme)
[^26]: https://code.claude.com/docs/llms.txt (docs)
[^27]: https://forum.cursor.com/t/cursor-not-support-llms-txt-standard/108980 and https://forum.cursor.com/t/docs-cursor-com-llms-txt-serves-an-html-page-instead-of-the-llms-txt-file/167800 (forum)
[^28]: https://docs.devin.ai/windsurf/plugins/cascade/web-search (docs)
[^29]: https://github.com/orgs/community/discussions/162955 (forum)
[^30]: https://alejandrorioja.com/tools/llms-txt-validator/ and https://llmstxtvalidator.dev/ (docs)
[^31]: https://github.com/bridgetoagent/llms-txt-validator (readme)
[^32]: https://github.com/BerriAI/litellm/issues/36342 (forum)
[^33]: https://github.com/AnswerDotAI/llms-txt/issues/152 and https://github.com/AnswerDotAI/llms-txt/issues (forum)
[^34]: https://www.llms-text.com/blog/llms-txt — injection via linked markdown, OWASP LLM01 (vendor)
[^35]: https://libraries.io/npm/llms-txt-parser (docs)
[^36]: https://blog.cloudflare.com/content-signals-policy/ (docs)
[^37]: https://www.seroundtable.com/google-cloudflare-content-signals-41631.html — Mueller, 2026-07-06 (docs)
[^38]: https://www.wislr.com/articles/ai-bot-behavior-log-analysis/ (study)
[^39]: https://originality.ai/blog/llms-txt-tracking-study (study)
[^40]: https://www.shopifreaks.com/shopify-quietly-rolls-out-native-llms-txt-files-for-stores-adding-structured-data-layer-for-ai-agents/ (blog)

# Generation tooling
Source: https://llms-explorer.com/reference/tooling/

# Generation tooling

Generators compared; why extractive descriptions win.

verified-as-of: 2026-08-30 (tool versions, maintenance status and platform features are volatile — re-verify before recommending)

1. Pick by situation
2. Docs platforms (built-in)
3. Static-site-generator plugins
4. Crawl-based generators (sites you do not own)
5. CMS and site builders
6. Edge content negotiation
7. Quality practices that generators get wrong

## 1. Pick by situation

| You have… | Use | Emits |
|---|---|---|
| Docs on Mintlify / GitBook / ReadMe / Fern | nothing — it is automatic | llms.txt (+ full on Mintlify/GitBook) + `.md` twins |
| Docusaurus, MkDocs, VitePress, Starlight, Sphinx, Nuxt | the framework plugin (table §3) | llms.txt + llms-full.txt (+ `.md`, `llms-small.txt` on Starlight) |
| A live site you do not own | crawl-based generator (§4) — `create-llmstxt-py`, `dotenvx/llmstxt`, or your own sitemap→markdown pipeline | llms.txt (+ full) with **extracted or AI-written** descriptions |
| WordPress | Yoast ≥25.3 / Rank Math / AIOSEO (§5) | llms.txt only (AIOSEO Pro adds full + markdown posts) |
| Webflow / Framer | host a hand-written file | whatever you upload |
| Any Cloudflare-proxied HTML site | "Markdown for Agents" toggle (§6) | on-the-fly markdown on `Accept: text/markdown`, no llms.txt |

## 2. Docs platforms

| Platform | Emits | Descriptions from | Notes |
|---|---|---|---|
| **Mintlify** | llms.txt, llms-full.txt, `.md` per page, `/.well-known/` copies, `/_llms/` split indexes | frontmatter `description` (truncated at 300 chars), nav order from `docs.json`; optional `markdown.instructions` agent text | index capped at 100,000 chars → recursive `/_llms/<group>.md`; default language/version only; hidden/noindex pages excluded; hand-written root files override; auth sites list public pages or require auth[^1] |
| **Fern** | llms.txt (root **and per-subdirectory**), `.md` per page; **no llms-full.txt** | frontmatter `description`, fallback `subtitle`; adds OpenAPI/AsyncAPI links | dropped llms-full because it "exceeded most model context windows, added heavy serving overhead, saw little use"[^2] |
| **GitBook** | llms.txt (Jan 2025), llms-full.txt + `.md` per page (Jun 2025), `/sitemap.md`, `Accept: text/markdown` | auto from page structure | zero-config; no curation controls documented; full export "will be more expensive"[^3][^4] |
| **ReadMe** | llms.txt (default on, all plans), `.md` per page; **no llms-full** | project title + guide/API hierarchy | a custom file from the repo root disables auto-updates; hidden pages excluded[^5] |
| **GitDoc** (vendor claim) | llms.txt + llms-full.txt "for the pages you mark as priority", regenerated in the build | sidebar/nav | vendor blog, 2026-05-22[^6] |

## 3. Static-site-generator plugins

| Plugin | Emits | Input | Descriptions / ordering | Maturity & limits |
|---|---|---|---|---|
| `docusaurus-plugin-llms` (rachfop) | llms.txt, llms-full.txt, optional per-page `.md`, versioned + `customLLMFiles` | source tree at `postBuild` | frontmatter → first heading → site fallback; `includeOrder` globs | 144★, MIT; not run in `docusaurus start`; image rewrite only for bundled assets[^7] |
| `@signalwire/docusaurus-plugin-llms-txt` | llms.txt, `.md`, optional full | **built HTML** (rehype/remark) | manual `sections[].description`, `autoSectionDepth` | v1.2.2, ~10 months stale; ENOENT / "processed 0 documents" bug[^8][^9] |
| Docusaurus core | none | — | — | issue #10899 open since Feb 2025[^10] |
| `mkdocs-llmstxt` (pawamoy) | llms.txt, `.md`, optional `full_output` | built HTML → BeautifulSoup → Markdownify | `sections:` dict with per-file descriptions | 130★, v0.5.x, **maintenance mode, seeking maintainer**; needs `site_url`; mkdocstrings `show_source` mangles tables/code in the full file[^11][^12] |
| `vitepress-plugin-llms` (okineadev) | llms.txt, llms-full.txt, `.md` | VitePress source | frontmatter `description`; `<llm-only>` / `<llm-exclude>` tags | 394★; used by Vite, Vue, Vitest, Rolldown; relative URLs break under redirects/domain moves[^13] |
| `starlight-llms-txt` (delucis) | llms.txt, llms-full.txt, **llms-small.txt** | Astro Starlight | `projectName`, `description`, `details`, `optionalLinks`, `customSets`, `promote`/`demote`; `minify` strips asides | 110★, docs updated Aug 2026; needs `site`[^14] |
| `sphinx-llms-txt` (jdillard) | llms.txt (markdown), llms-full.txt (**reStructuredText**) | Sphinx build | toctree titles; `llms_txt_summary`, `llms_txt_exclude`, `llms_txt_full_max_size` | v0.7.1; full file is RST; points to NVIDIA `sphinx-llm`[^15] |
| `nuxt-llms` / Nuxt Content | llms.txt (~5K tokens), opt-in llms-full.txt (~1M+ tokens) | Nuxt Content, runtime hooks | `sections` in `nuxt.config` | first-party; full file explicitly for 200K+-context tools[^16] |
| Next.js / Nextra | hand-rolled `app/llms.txt/route.ts` (force-static or dynamic); `next-llms-txt` adds per-page `.md` endpoints | components | "reads and parses readable text" | discussion #80692 unresolved; no Nextra built-in found (tentative)[^17][^18] |
| `llms-txt-action` (demodrive-ai) | llms.txt, llms-full.txt, `.md` | built HTML dir + sitemap.xml | local/offline or cloud LLM summaries via LiteLLM (default GPT-4o) | 16★; needs `--dirty` with `mkdocs gh-deploy`[^19] |

## 4. Crawl-based generators (sites you do not own)

| Tool | What it does | Limits |
|---|---|---|
| Firecrawl `/llmstxt` API + llmstxt.firecrawl.dev | URL → async job → llms.txt (+ full); `maxUrls` 1–100 (default 10), 1 credit/URL, public pages only, 5,000-URL alpha cap | **deprecated in favour of the main endpoints** (page carries no date; still up); users pointed to the Python repo[^20][^21] |
| `create-llmstxt-py` (Firecrawl, 320★) | `/map` → scrape each page to markdown (batches of 10; failures skipped, no retry) → GPT-4o-mini writes a 3–4-word title + 9–10-word description → flat llms.txt; llms-full.txt concatenates under `<\|firecrawl-page-N-lllmstxt\|>` | default 20 URLs; memory issues on large sites; **sections are not inferred**; descriptions are AI-written and unreviewed[^22] |
| `dotenvx/llmstxt` (147★, BSD-3) | sitemap.xml → `- [Title](url): description` bullets; `--include-path` / `--exclude-path` globs; `--replace-title` regex | llms.txt only; titles extracted from HTML; description derivation undocumented[^23] |
| Jina Reader `r.jina.ai/<url>` | headless Chrome or curl engine → Readability → Turndown; headers `x-respond-with`, `x-target-selector`, `x-retain-links`, `x-max-tokens`, `x-markdown-chunking` | per-page cleaner, no site/llms.txt mode; anonymous traffic rate-limited[^24] |
| Screaming Frog v24.3 | per-page `.md` via a Readability.js + Turndown custom-JS snippet; llms.txt via n8n/CSV converters | no native llms.txt export; thin pages return nothing; JS rendering slow[^25][^26] |
| `plainsignal/llmstxt` Chrome extension | llms.txt + one `.md` per page + zip from sitemap or rendered DOM; meta description as blockquote | 10★, HTTPS only[^27] |
| SEO-tool generators (SEOmator etc.) | robots.txt → sitemap discovery, index-sitemap expansion, LLM-written title+description per URL | vendor-claimed mechanics only[^28] |
| llms-text.com generator/validator | crawls a domain and exports llms.txt + llms-full.txt ("deep-crawls up to 50 subpages"); validator checks syntax, links, UTF-8, headers | vendor; its guidance: 10–20 evergreen URLs, 4–7 H2s, 10–20-word descriptions, index under 10 KB, `Content-Type: text/plain|text/markdown; charset=utf-8`, HTTP 200 (no redirect/auth), `Link: <…/llms.txt>; rel="describedby"` header[^29][^30] |

## 5. CMS and site builders

| Platform | Emits | Descriptions | Limits |
|---|---|---|---|
| Yoast SEO ≥25.3 (2025-06-10) | llms.txt only, regenerated weekly | custom excerpt only — **no description otherwise**; 5 latest posts/pages/CPT (≤12 months, cornerstone first) + top-5 taxonomies | 5-item cap; markdown chars escaped; a static file wins over the dynamic one[^31][^32] |
| Rank Math | llms.txt only | "intro text"; post types/taxonomies, limit default 100; custom lines | no full[^33] |
| AIOSEO | llms.txt (free); llms-full.txt + markdown post conversion (Pro) | site title/tagline; per-post-type limits, exclusions | paywall[^34] |
| `website-llms-txt`, `llms-full-txt-generator` | llms.txt (+ full) | titles + SEO-plugin descriptions; honour noindex | one shipped a broken-access-control CVE fix[^35] |
| Joost de Valk "Markdown Alternate" | `<link rel="alternate" type="text/markdown">` + `.md` URLs per post | — | negotiation, not an index[^36] |
| Webflow / Framer | host an uploaded file (Framer: Pro/Enterprise "Hosting → Files"); a Framer marketplace plugin scans the CMS | manual | no generation[^37][^38] |
| Shopify (Apr–May 2026, silent) | auto `/llms.txt`, `/agents.md`, `/sitemap_agentic_discovery.xml`, `/.well-known/ucp` on every store | boilerplate: H1 store name, `/collections/all`, contact, UCP + MCP endpoints | `templates/llms.txt.liquid` **replaces, does not merge**; no changelog; 78.1% of top-10k Shopify hosts vs WordPress 8.7%[^39][^40][^41] |

## 6. Edge content negotiation

Cloudflare "Markdown for Agents" (2026-02-12; Pro/Business/Enterprise; zone toggle under AI Crawl Control): on `Accept: text/markdown` the edge converts HTML → markdown (body + meta-derived YAML frontmatter + JSON-LD, nav/header/footer/scripts dropped) and returns `Content-Type: text/markdown; charset=utf-8`, `x-markdown-tokens`, `x-original-tokens`, `Vary: Accept`; ETag/Last-Modified/Content-Encoding stripped; origin HTML ≤ 2 MB (raised from 1 MB); a chunked-encoding silent pass-through was fixed Jul 2026.[^42][^43][^44] It produces no llms.txt — pair it with a hand-written index. Checkly measured a 99.7% token reduction on its own docs (single site).[^45]

## 7. Quality practices that generators get wrong

- **Descriptions are the product.** Every platform generator draws the one-liner from frontmatter `description`; crawl tools scrape `<meta>` or have a small model invent it; WordPress generators are weakest (Yoast emits none without a custom excerpt; Yoast/Rank Math "list content but don't really prioritize it").[^1][^22][^31][^46] Treat AI-written descriptions as drafts to edit, and audit the *page list*, not just the output. The spec's own test: give an agent only the llms.txt and ask it questions.[^47]
- **Sections and order come from config or nav, never inferred by crawlers.** mkdocs `sections:`, signalwire `sections[]` + `autoSectionDepth`, Starlight `customSets` + `promote`/`demote`, docusaurus `includeOrder`; Mintlify uses `docs.json` order. Mintlify's editorial rule: order by "frequency, not importance" — the first 20% of links should answer 80% of questions.[^11][^14][^7][^1][^48]
- **`## Optional`** (convention only in v2): changelogs, legal, old posts, deep appendices; never pricing or the API reference.[^49]
- **Size budgets are producer-side.** Split large indexes (Mintlify 100k chars → `/_llms/`), ship a small variant (Starlight `llms-small.txt`, Nuxt's ~5K-token file), cap the full file (`llms_txt_full_max_size`), or drop it (Fern). No cross-vendor numeric budget exists; "index under 10 KB" is a vendor number.[^1][^14][^15][^2][^30]
- **Regenerate in the build; check links in CI.** Custom/static files freeze updates (ReadMe, Yoast); dead links happen (litellm's index carried a deleted page); `llms-txt-validator --check-links` and `npx llms-txt-check` exist for pipelines.[^5][^31][^50][^51]
- **Serve `.md` twins and honour `Accept: text/markdown`** where the platform allows; add `Vary: Accept` and bypass full-page caches keyed without it.[^1][^52]
- **Counter-evidence to weigh before spending effort:** 97% of files get zero AI requests; Google has no implementation. Generation pays off for developer docs consumed by coding agents, not for general SEO.[^53][^54]

## References

[^1]: https://www.mintlify.com/docs/ai/llmstxt (docs)
[^2]: https://buildwithfern.com/learn/docs/ai-features/llms-txt (docs)
[^3]: https://gitbook.com/docs/ai-and-search/llm-ready-docs (docs)
[^4]: https://gitbook.com/docs/changelog/june-2025/24-june-performance-upgrades-llms-full.txt-and-.md-support-text-alignment-and-more (docs)
[^5]: https://docs.readme.com/main/docs/LLMstxt (docs)
[^6]: https://gitdoc.ai/blog/llms-txt-ai-readable-documentation — 2026-05-22 (vendor)
[^7]: https://github.com/rachfop/docusaurus-plugin-llms (readme)
[^8]: https://github.com/signalwire/docusaurus-plugins/tree/main/packages/docusaurus-plugin-llms-txt (readme)
[^9]: https://github.com/signalwire/docusaurus-plugins/issues/5 (forum)
[^10]: https://github.com/facebook/docusaurus/issues/10899 (forum)
[^11]: https://github.com/pawamoy/mkdocs-llmstxt (readme)
[^12]: https://github.com/mkdocstrings/python/issues/299 (forum)
[^13]: https://github.com/okineadev/vitepress-plugin-llms (readme)
[^14]: https://delucis.github.io/starlight-llms-txt/configuration/ (docs)
[^15]: https://sphinx-llms-txt.readthedocs.io/en/latest/ (docs)
[^16]: https://nuxt.com/docs/4.x/guide/ai/llms-txt (docs)
[^17]: https://next-llms-txt.vercel.app/ (docs)
[^18]: https://github.com/vercel/next.js/discussions/80692 (forum)
[^19]: https://github.com/demodrive-ai/llms-txt-action (readme)
[^20]: https://docs.firecrawl.dev/features/alpha/llmstxt (docs)
[^21]: https://github.com/firecrawl/llmstxt-generator (readme)
[^22]: https://github.com/firecrawl/create-llmstxt-py (readme)
[^23]: https://github.com/dotenvx/llmstxt (readme)
[^24]: https://github.com/jina-ai/reader (readme)
[^25]: https://www.screamingfrog.co.uk/blog/generate-markdown-at-scale/ (vendor)
[^26]: https://n8n.io/workflows/3219-generate-ai-ready-llmstxt-files-from-screaming-frog-website-crawls/ (vendor)
[^27]: https://github.com/plainsignal/llmstxt (readme)
[^28]: https://seomator.com/free-llms-txt-generator (vendor)
[^29]: https://www.llms-text.com/blog/how-to-create-llms-txt — Michael Vereb, 2025-07-25 (vendor)
[^30]: https://www.llms-text.com/blog/llms-txt — 2025-07-25 (vendor)
[^31]: https://developer.yoast.com/features/llms-txt/functional-specification/ (docs)
[^32]: https://developer.yoast.com/changelog/yoast-seo/25.3/ (docs)
[^33]: https://rankmath.com/kb/llms-txt/ (docs)
[^34]: https://aioseo.com/docs/how-to-create-an-llms-txt-using-all-in-one-seo/ (docs)
[^35]: https://wordpress.org/plugins/website-llms-txt/ (vendor)
[^36]: https://joost.blog/markdown-alternate/ (blog)
[^37]: https://university.webflow.com/videos/optimize-your-site-for-llms-with-llms-txt (docs)
[^38]: https://www.framer.com/help/articles/llms-txt-framer/ (docs)
[^39]: https://honeybound.co/blog/shopify-llms-txt-agents-md (blog)
[^40]: https://imakemvps.com/blog/llms-txt-generator-for-shopify (blog)
[^41]: https://caseyrb.com/blog/state-of-llms-txt-adoption/ — HTTP Archive, Jun 2026 (study)
[^42]: https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ (docs)
[^43]: https://developers.cloudflare.com/changelog/post/2026-02-12-markdown-for-agents/ (docs)
[^44]: https://community.cloudflare.com/t/cloudflare-fundamentals-content-encoding-support-for-markdown-for-agents-and-other-improvements/893536 (forum)
[^45]: https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/ (blog)
[^46]: https://weventure.de/en/blog/llms-txt (blog)
[^47]: https://llmstxt.org/ (spec)
[^48]: https://www.mintlify.com/blog/real-llms-txt-examples (vendor)
[^49]: https://dev.to/lab451/complete-llmstxt-guide-for-2026-57d (blog)
[^50]: https://github.com/BerriAI/litellm/issues/36342 (forum)
[^51]: https://github.com/bridgetoagent/llms-txt-validator (readme)
[^52]: https://toddmorourke.com/learn/markdown-for-agents/ (blog)
[^53]: https://ahrefs.com/blog/llmstxt-study/ (study)
[^54]: https://www.searchenginejournal.com/google-says-llms-txt-is-purely-speculative-for-now/577576/ (blog)

# Usage: serving, discovering and reading llms files
Source: https://llms-explorer.com/reference/usage/

# Usage: serving, discovering and reading llms files

The headers to send, the .md twins to publish, how a reader discovers the family, how an agent reads an index, and how Claude Code and the hub MCP tools consume one.

Two halves: what a **publisher** serves, and what a **reader** does with it. Both are what this
site does to itself (principle 6, dog food), so every example below can be checked against
`/llms.txt` and any page's `.md` twin.

## 1. Serving

Every markdown file in the family is served with:

| Header | Value | Why |
|---|---|---|
| `Content-Type` | `text/markdown; charset=utf-8` | attribute H2; `text/plain` is tolerated, HTML is a High |
| `X-Markdown-Tokens` | `bytes // 4` — the same estimator `manifest.json` uses | H4: cost known before fetch |
| `Link` | `</llms.txt>; rel="describedby"` — the index that covers this file | H3, spec v2 discovery |

HTML pages carry the reverse links in `<head>` (this site's `Base.astro` does):
`<link rel="alternate" type="text/markdown" href="/reference/usage.md">` and
`<link rel="describedby" href="/llms.txt">`. Serve with HTTP 200, no redirect, no auth on the
path: Lighthouse's agentic-browsing audit treats a 404 as not applicable but flags a server
error (H7), and a redirect to an HTML app shell — as docs.cursor.com once did — fails P13.

## 2. Markdown twins

Every page in the content sections — reference, essays, examples, blog — has a clean-markdown
twin at the same route with `.md` appended: `/reference/usage/` → `/reference/usage.md`. The three
generated sections carry one twin each for the section, not per row: `/tree.md`, `/directory.md`
and `/demo.md` hold the prose that explains the section plus an inventory of what it contains.
The per-row pages under them — `/tree/<slug>/`, `/tree/3d/`, `/directory/<key>/` — are HTML only,
because a twin apiece would take the `_headers` file past Cloudflare's 100-rule cap; every page
advertises the twin it actually has, and none advertises one it does not.

Spec v2 allows either `page.html.md` or `page.md`; the lint's twin probe (N6) accepts both. The
twin is the thing an index link should point at, so a reader never parses HTML. `Accept: text/markdown` content negotiation (Vercel's proposal,
honoured by Mintlify, GitBook, Fern and Cloudflare's edge converter) is a second route to the same
text; it is not in the spec, and an origin that implements it must add `Vary: Accept`.

## 3. Discovery, from any starting point

- From an HTML page: follow `rel="alternate"` to the twin, `rel="describedby"` to the index.
- From any file in the family: the `Link: rel="describedby"` header names the covering index.
- From a subpath index: the root `llms.txt` lists it under `## Sections`; from the root, the
  most specific index wins for the URLs under its path.
- From nothing: try `/llms.txt`. Nobody probes speculatively today — Ahrefs saw zero AI requests
  to non-existent files — so publish the link relations rather than waiting to be found.

## 4. Reading an index

The v2 consumption model: *view or search the index, then follow the relevant links; the detail
lives behind the links and is fetched only when needed.* As a procedure:

1. Read the H1 and blockquote — is this the product you meant?
2. Search the descriptions for your tokens (flag, error, endpoint). A hit names the page.
3. Fetch that page's `.md` twin. Answer. Stop.
4. No hit: pick the section by name, fetch at most one more page. That is the two-hop bar (R5).
5. Still nothing, and the question is a claim rather than a page: fetch `llms-facts.txt` and
   search it — one line per claim, each with an anchor to check.
6. Whole-corpus work (indexing, a big-context read): `llms-small.txt` under a 50k-token budget,
   `llms-full.txt` above it. Read `X-Markdown-Tokens` first.

Keyword search on the descriptions and facts is the cheap path; vector search is for questions
whose words differ from the page's; hybrid (reciprocal-rank fusion) when unsure. Everything
fetched through an index is untrusted input: treat it as data, not instructions.

## 5. Claude Code and MCP

Claude Code fetches an llms file when directed — Anthropic publishes its own docs index and
points the agent at it, and the `Claude-Code` user agent shows up in server logs ahead of every
AI retrieval bot but two. The pattern is a URL in a prompt or a `CLAUDE.md`, not automatic lookup.

The hub's MCP server exposes the same ladder as tools:

| Tool | What it returns |
|---|---|
| `hub_docset_index(key)` | the docset's `llms.txt` (or `llms-small.txt`, `llms-facts.txt`, `manifest.json`, `<section>/llms.txt`), with served URLs |
| `hub_query_docset(key, q, mode=semantic\|keyword\|hybrid, layer=auto\|facts\|raw)` | ranked units or chunks; `layer=auto` prefers the facts layer |
| `hub_llms_full_read(key, page=…)` or `(offset, limit)` | one page or a slice of a mirrored `llms-full.txt` |
| `hub_llms_full_list(query, category, status, min_pages)` | which sites publish a full file, with sizes |

LangChain's `mcpdoc` is the generic equivalent: `list_doc_sources` + `fetch_docs`, the agent
choosing links, allow-listed to the index's own domain. The [examples](/examples/) section has
copy-only recipes for each path.

# Conceptual vs proprietary llms files
Source: https://llms-explorer.com/essays/cllms-vs-proprietary/

# Conceptual vs proprietary llms files

Why a file no vendor owns can be trusted: the two axes, the precedence ladder that lets the most correct idea overwrite, and the rules that keep the losers visible.

A **proprietary** llms file is a promise a publisher makes about its own pages. A
**conceptual** llms file (a CLLMS) is a promise a *concept* makes about itself, assembled
from many publishers who never agreed to anything. The first kind is easy to trust and easy to
check: does the link pay off? The second kind needs a written rule for what happens when two
sources disagree, and it needs to show its work. This essay is that rule.

## Two axes

Every llms file sits on one of two axes.

The **source axis** is the one the spec describes. `llms.txt` is the navigation of one site;
`llms-full.txt` is that site's pages concatenated; `llms-facts.txt` is the same site reduced to
one-line units, each anchored to the page and heading it came from. The publisher is the
authority. If code.claude.com says its admin guide has four decisions to make, that is what the
file says, and nobody else's opinion is in scope.

The **concept axis** regroups those same units by what they are *about*. `llms-concepts.txt`
is the navigation of a concept tree, not a site. A concept page ("prompt caching", "cookie
expiry", "the `Link: rel=describedby` header") pulls units from every docset that mentions it.
A topical file (`/t/<slug>/`) is a view over that regrouping for one subject; a concept pack
is the same view with facets and a vocabulary attached.

The axes share a grammar and a lint. What they do not share is an authority:

| | source axis | concept axis |
|---|---|---|
| file | `llms.txt`, `llms-full.txt`, `llms-facts.txt` | `llms-concepts.txt`, `/t/<slug>/llms-facts.txt`, concept packs |
| the authority | the publisher | the concept |
| the unit of truth | a page | a claim about the concept |
| what "correct" means | the link resolves and says what the description said | the claim survives comparison with every other source's claim |
| what a disagreement is | impossible: one publisher, one page | routine: two sources, two claims, one concept |

A proprietary file can be wrong only by being stale or by over-promising. A conceptual file
can be wrong in a third way: by picking the weaker of two competing claims. So it needs a
procedure for picking, and the procedure has to be public.

## The most correct idea overwrites

On the concept axis, units compete. When a new unit arrives (a contributor submits it, an
abstraction run extracts it, a re-fetch changes it), the `resolve` job computes a **claim
key** for it — the normalised text, or for numeric claims the pair (subject, number) — and
looks for an existing unit with the same key under the same concept. No match: write. Match
with the same claim: merge the corroboration (`also[]` grows by one). Match with a
*different* claim: score both on the ladder below. The higher score **overwrites**; the lower
one is not deleted.

"Overwrite" is a narrow word here. It changes which line appears in `llms-facts.txt` under the
concept. It preserves:

- **provenance** — the loser keeps its source URL and anchor, and gains `superseded_by: <winner id>`;
- **the record** — one line is appended to `conflicts.jsonl`: `{concept, claim_key,
  winner_id, loser_ids[], rung, scores{}, resolved_at, resolver: ladder|human, note,
  prior_winner_id}`;
- **the stamp history** — the winner's `verified-as-of` and the loser's are both kept, so a
  later re-fetch that flips the result can be explained.

Replaying `conflicts.jsonl` from an empty pack regenerates identical files. That is the
acceptance test for the whole mechanism: if the served files cannot be rebuilt from the
record, the record is not the truth and the mechanism is not honest.

A worked case. Two units under the concept *llms.txt adoption*, both on the

- `[fact] 5.07% of the top 1M sites publish an llms.txt — <HTTP Archive, 2026-06>`
- `[fact] 28% of 137,210 Ahrefs-Web-Analytics domains publish a valid file — <Ahrefs, 2026-05>`

Same subject, numbers five times apart, so the claim keys collide. Rung 1 (source grade) ties:
both are primary measurements, neither is a vendor page. Rung 3 (recency) hands it to HTTP
Archive by one month — a thin reason to bury a 28% figure. Rung 6 is the rung that actually
decides it: the populations differ (a crawl of the top million versus a self-selected panel of
sites that installed Ahrefs' analytics), so neither number answers the other's question. The
scoped claim wins for a scoped question, the loser stays readable, and the record's `note`
carries the reason rather than deleting the disagreement.

## The precedence ladder

Higher rungs win; a tie on a rung falls to the next. The rungs, in order:

1. **Source grade** — `grade`: spec/standard > vendor docs > primary measurement > reputable
   secondary > blog. The hierarchy is the one deep-research methodology uses; nothing here is
   invented for llms files.
2. **Corroboration** — the count of independent sources in `also[]` stating the same claim.
   Independent means it: a citation chain (B quotes A, C quotes B) collapses to one.
3. **Recency of verification** — `verified-as-of`, and only when the stamp came from an actual
   re-fetch. A date bump without a fetch is not evidence and the lint treats it as none.
4. **Agreement with the canonical definition** — the unit's vocabulary sense id matches the
   family's canonical sense (see [the vocabulary essay](/essays/vocabulary/)). A claim about
   the snack loses to a claim about the HTTP cookie inside a web family.
5. **Agent-test performance** — the unit answered questions in the P12 eval bank
   (`evals/*.eval.jsonl`) that the other did not.
6. **Scope precision** — when the question is scoped (a version, a platform), the narrower
   unit beats the broader. Most "disagreements" that reach this rung are apparent, not real,
   and resolve here by scoping rather than by winning.
7. **Tie → moderation queue** — both units stay in `## Disagreements`; a human accepts or
   rejects with a note, and the note becomes part of the record.

The ladder is total: every conflict resolves to a rung or to the queue. There is no rung that
says "drop silently", and the test corpus of synthetic conflicts checks that none of them
vanish. The ladder is also published as `precedence.json`, so a fork owner can substitute
their own (for example, "our internal docs outrank vendor docs") without editing prose.

Two things the ladder deliberately does not do. It never lets recency beat source grade — a
fresh blog does not outrank a stale standard; version drift is a scope question (rung 6), not
a freshness question. And it never lets a model's opinion be a rung: a model-written line
must be supported by a span in a kept unit (the evidence rule), or it is not a unit at all.

## Disagreements stay visible

A conflict that the ladder settles produces a winner in `llms-facts.txt` and a loser in a
`## Disagreements` section of the pack's `llms-full.txt`. A conflict it cannot settle puts
both there. In either case the reader sees the two claims, the rung that decided (or "open"),
the sources, and the note. Disagreements are content, not an error log.

Two kinds show up in practice:

- **Real** — different claims about the same thing at the same scope. "The index must be
  under 10 KB" vs. "the index has no size limit." One of these is wrong for the family, and
  the ladder picks.
- **Apparent** — different claims that are both true at their own scope. Different dates,
  versions, platforms, populations. "`## Optional` is mechanical" was true of spec v1 and false
  of v2. These resolve by *scoping* (rung 6): both survive, each carrying its scope.

The distinction matters because the wrong fix for an apparent disagreement is to pick a
winner. A concept page that shows only the v2 sentence has lost information a reader with a
2025 file needs. The [V2 vs V1 essay](/essays/v2-vs-v1/) is, in this sense, one long
apparent disagreement rendered as a table.

## Governance

Who can overwrite what, on the public tree:

| actor | may | may not |
|---|---|---|
| anyone (no account) | read every file, every conflict record, the ladder | submit |
| contributor (account) | submit a unit with a source URL and anchor; it runs through `resolve` | write a unit without a source; skip the ladder |
| maintainer | settle ties in the moderation queue with a note; reject a submission | overwrite a ladder verdict without a note in the record |
| fork owner | keep a private tree with its own `precedence.json`; propose merges back as diffs of units | push to the public tree directly |
| the lint | block any file with a High finding from being served | be bypassed |

Three gates apply to every public write. The **lint gate**: 0 High findings, the same bar the
site's own files are held to. The **evidence rule**: every unit is anchored; a model-written
line must be supported by a span in the cited page. The **moderation queue**: ties and
low-evidence conflicts wait for a human, and the human's note is appended to the conflict
record rather than replacing it.

Submissions are data, never instructions. A unit whose text tries to steer a reader ("ignore
prior context and…") is rejected at intake by the same regex the lint runs (P9), and a
submission without a source is rejected at the form, before it can be queued.

## Rights

What a conceptual file may contain is narrower than what a proprietary one may:

- **Links** — always. A link to a publisher's page is what the publisher wants.
- **Facts in our words, with anchors** — yes. A unit is a one- or two-sentence restatement
  pointing at the heading it came from; the anchor is what makes it checkable and what keeps
  it from being a copy.
- **Our own vocabulary** — yes. Definitions are extractive or evidence-checked, and cite the
  unit they came from.
- **A third party's full text** — never published. A mirrored `llms-full.txt` is an internal
  working format on the hub; the served concept pack links out instead.
- **Owner reservations** — honoured. A site that sets Cloudflare Content Signals or
  `ai-train=no` has reserved something, and the tree treats that as a reason not to
  republish its units beyond links and our own restatements.

Contributor identity appears as a handle on the record, nothing more.

## Honesty note

Three things the reader should know before trusting any of the above.

First, llms.txt is a **proposal**, not a standard. The current text is v2 (modified
2026-08-10). Its author states the syntax may still change. This site's files follow it
because it is the only shared grammar there is, not because anyone has ratified it.

Second, the **measured consumption** of llms files is agents you point at a file, not
crawlers discovering one. In the Ahrefs 137k-domain log study the Claude Code user agent
out-fetched every retrieval bot except two, and the overwhelming majority of domains saw zero
AI requests for the file at all. A CLLMS is therefore for the agents we control first: the
ones reading through this site's MCP tools and the ones we hand a URL to. Any claim that a
concept pack improves discovery by third-party crawlers is a hope, not a finding.

Third, this essay describes the governance as **designed** (decision 2026-08-31). The
`resolve` job, the conflict records and the moderation queue are the design the site is
built to; the parts that exist today are the unit grammar, the deduplication that finds
near-duplicate claims, the disagreement grouping in the abstractor, and the lint passes the
ladder reuses. Where a page on this site shows a live conflict record, it will say which
rung decided and when; until then the worked example above is a worked example.

# Semantic indexing: two legs and a fusion
Source: https://llms-explorer.com/essays/semantic-indexing/

# Semantic indexing: two legs and a fusion

A docset can be asked a question by token or by meaning, and the two ways fail in opposite directions. What each leg costs, where each breaks, why reciprocal-rank fusion is the default, and how to read the recorded run at /demo/.

Retrieval arguments usually run on assertion: someone says embeddings understand meaning,
someone else says keyword search is fine, and neither side shows a query. The demo at
[`/demo/`](/demo/) is the same eleven questions run three ways against one indexed docset,
with the hits and the timings kept. This essay is what to look for in it.

## The two legs

A docset in the hub is indexed twice over the same units.

The **keyword leg** is SQLite FTS5 with BM25 ranking. A question becomes an OR of its terms,
the index returns the rows that contain them, and BM25 ranks by term rarity against document
length. It is exact: `CLAUDE_CODE_SYNC_SKILLS` matches the row that literally contains
`CLAUDE_CODE_SYNC_SKILLS`, and no row that does not. It costs no model call — sub-millisecond
on a single rare token in the recorded run, and tens of milliseconds on a six-word question,
because cost scales with how many terms have to be unioned.

The **vector leg** embeds the question with the same model the docset was indexed with and
ranks units by cosine similarity. It matches on meaning, so a question that shares no words
with its answer still finds it. Its cost is one embedding call and is almost independent of
the question: in the recording every vector query *after the first* lands in the same narrow band
whether the question is one token or nine words. The first pays the one-off cost of opening the
connection to the embedding host — 124 ms against a median of 15 — and is marked as such on the
demo page; read it as a connection cost, not a retrieval cost.

Their failure modes are mirror images. BM25 cannot find a page that phrases the answer
differently. Embeddings dilute a lone identifier into a low-signal vector, and a docset full
of near-identical config rows gives it many almost-equally-good answers. This is why the
exact-token probes in `/ldo` pass P11 exist at all: they are the class that a vector index
quietly gets wrong.

## Fusing them

The hub's default is neither leg alone. `hub_query_docset(mode="hybrid")` runs both and fuses
them with **reciprocal-rank fusion**: each hit scores `1 / (60 + rank)` in each list it
appears in, and the scores add, keyed by `(url, seq)`. No score from either leg survives the
fusion — only the ranks — which is the point. BM25 scores and cosine similarities are not on
a comparable scale, and any attempt to weight one against the other is a knob nobody can
tune honestly. Rank agreement needs no scale: a unit that both legs put near the top beats
a unit that only one leg loves.

The recording shows the mechanism working in the least glamorous way. The two legs agreed on
the top hit for only two of the eleven questions; for two others the fused winner was the top
hit of *neither* leg — a unit each leg ranked second or third, promoted past two disagreeing
favourites because both legs voted for it.

## What the recording shows

Read [`/demo/`](/demo/) with three questions in mind.

- **Does the cheap leg already answer it?** For the exact-token probes it usually does, first
  hit, with the value in the snippet — and it does so without touching a model. An agent that
  reaches for an embedding call to look up an environment variable is paying for nothing.
- **Where does the paraphrase leg earn its cost?** Look at the hook questions. BM25 ranks a
  *Debug hooks* prose row first because it contains every word in the question; the vector leg
  puts the exit-code table row first, which is the answer. That is the whole case for
  embeddings in one comparison.
- **What did fusion change?** The `legs` count on a hybrid hit says how many legs found it.
  Hits with two legs are the ones the fusion is built to promote; a hybrid list of all
  one-leg hits means the legs never agreed, and the fusion is doing nothing but interleaving.

Two honest caveats. The timings are a single run on a single laptop with one local embedding
host, so the ratios between the legs are the finding and the absolute milliseconds are not (and
the first query pays the connection cost noted above). And this is a
recording, dated on the page — it is not a live endpoint, and it will not be one until the
query API arrives.

## Run it yourself

Everything on the demo page comes from one command against an indexed docset, so the same
comparison can be run over yours. The keyword index is built on first use; the facts layer is
preferred automatically when a docset has one.

```
# the cheap leg — FTS5/BM25, no model call
docset_indexer.py keyword <docset> "CLAUDE_CODE_SYNC_SKILLS"

# the meaning leg — one embedding call, then cosine
docset_indexer.py query <docset> "which hook events fire once per turn" --layer auto

# both, fused — what hub_query_docset serves by default
hub_query_docset(docset=..., question=..., mode="hybrid")
```

The recording itself is regenerated with `site/tools/gen_demo.py`, which is the one generator
in this site that reads a live hub and therefore the one that never runs in CI: it is run by
hand on a box that has the indexes, and its output is committed. See
[the reference](/reference/) for the surrounding commands.

# V2 vs V1
Source: https://llms-explorer.com/essays/v2-vs-v1/

# V2 vs V1

Two versioned things share a name: the llms.txt spec (v1 → v2, 2026-08-10) and the hub pipeline (V1 site dumps → V2 acquire, refine, dual index, gate). Both diffs, a migration guide, and what breaks.

Two different things are called "v2" on this site, and they confuse everyone in the same
way. The **llms.txt spec** went from v1 to v2 on 2026-08-10, and a file written against v1 is
still valid. The **hub's pipeline** went from V1 to V2 on 2026-08-30, and the V1 artifacts are
retired. One is a proposal that loosened; the other is a toolchain that was replaced. They are
laid out side by side below because a reader with a 2025 file usually needs both answers:
"is my file still valid?" (yes) and "why did the folder layout change?" (because the old one
produced output nobody consumed).

## The spec: v1 → v2

The spec's structure did not change: an optional BOM, an H1, a blockquote, free markdown
without headings, then H2 sections holding `- [name](url): notes` lines. What changed is
which parts are required, where the file may live, and how a consumer is expected to use it.

| Rule | v1 | v2 (2026-08-10) | Effect on an existing file |
|---|---|---|---|
| Required elements | H1 + blockquote + sections implied | **H1 only** is required; blockquote, prose and sections are optional | none is invalidated; the lint still scores a missing blockquote as Medium (I2) — a quality finding, not a validity one |
| Placement | `/llms.txt` at the site root | root **or any subpath** (`/docs/llms.txt`); a file covers the URLs under its path; where several apply, **the most specific wins** | enables families and split roots (`<section>/llms.txt`) |
| Discovery | none | `Link: <…>; rel="describedby"` on the files; `rel="alternate" type="text/markdown"` on HTML pages | add two headers (see the [serving reference](/reference/usage/#1-serving)) |
| Markdown twins | `page.html.md` | `page.html.md` **or** `page.md`; directories append `index.html.md` or `index.md` | either form passes the twin probe |
| `## Optional` | mechanical: "can be skipped if a shorter context is needed", consumed by `llms_txt2ctx` | a **convention** for secondary information; `llms_txt2ctx` and its context-expansion mechanics are no longer part of the proposal | keep it last; build nothing that depends on it |
| BOM | — | an optional BOM is tolerated | the lint strips it as hygiene (P14) |
| Consumption expectation | expand the file into context | "view or search the index, then follow the relevant links"; the index stays small; detail lives behind links | the size ladder (small / full) becomes the producer's job |
| `/.well-known/` | — | explicitly rejected: well-known URIs exist only at the origin root, which defeats subpath scoping | serve at the root or the subpath, not under `.well-known` |

The one-sentence summary: v2 made the file *smaller in obligation and larger in reach*. Less
is required, more places may hold one, and the reader is now told to search-and-follow rather
than to inhale. The consumption sentence is the important one for producers. If an agent is
expected to read the index and then fetch two pages, the index has to be small enough to read
and descriptive enough to choose from — which is why the lint measures both.

## The pipeline: V1 → V2

The hub's V1 pipeline produced site dumps. It crawled with trafilatura, wrote one banner
mirror per site, distilled that mirror with a zero-LLM bulk pass, and indexed the raw text in
one vector layer. The distilled output was never consumed: the working notes describe it as
"hundreds of pages of repeating internal links". V2 keeps the banner mirror as the internal
format and replaces everything around it.

| Stage | V1 (to 2026-08-29) | V2 (from 2026-08-30) |
|---|---|---|
| Acquire | trafilatura BFS crawl → banner mirror | a ladder in `llms_acquire.py`: the site's `llms-full.txt` → its `llms.txt` + `.md` twins → `Accept: text/markdown` → a docs API → a structured crawl; the banner mirror stays the internal format |
| Clean | none (raw HTML → text) | `docset_refine clean`: boilerplate lines, MDX → markdown, page classes (reference / guide / changelog / marketing / index) |
| Extract | `distill_offline.py bulk` — zero-LLM, output never consumed | `extract` (code snippets, table rows → `parameter`, definitions, changelog `change` units; anchors to real source headings) + `units` (local LLM under the evidence rule) + `polish` (Claude) |
| Export | none | `export_llms`: index (split above 10 KB) / full (Mintlify grammar) / small (≤ ~50k tokens) / facts / `manifest.json`; `topical`; `vocabulary` |
| Index | one raw vector layer (`nomic-embed-text` in `hub.db` for files; `mxbai-embed-large` for docsets) | raw **and** facts vector layers, plus an FTS5 keyword layer beside each (`docset_indexer keyword-index`) |
| Serve | `web-text-mirror --serve` (HTML) | `llms_serve.py`: `/llms.txt`, `/d/<stem>/…` (with sections), `/m/<key>/…`, `/t/<slug>/…`, markdown headers on every response |
| Gate | none | `llms_lint.py` (the deterministic passes P0–P3, P5–P7, P9, P14) inside `docset_rollout cleanup`; `/ldo` for the model, live and family passes |
| Artifacts | `<stem>.pages/`, `_master.md`, `._distill_index.json` | `<stem>.reference/{pages.json, structured.jsonl, units.jsonl, all_units.jsonl}` and `<stem>.llms/` |

"V2" for the pipeline is a naming choice made on this site; the code carries no version
constant. Dating it (2026-08-30) is more honest than numbering it, and the tables here do
that.

The measured shape of one V2 export, from `outputs/exports/code.claude.com.llms/manifest.json`
(191 pages, acquired from the publisher's own `llms-full.txt`): the root index is 1,136 bytes
(~280 tokens) and splits into three section indexes; `llms-small.txt` is 199,155 chars
(~49,785 tokens, just under the budget); `llms-full.txt` is ~2.1M tokens; `llms-facts.txt`
holds 14,031 units (~845k tokens). The V1 pipeline had no equivalent numbers to print, which
is its own summary.

## Migration

For a **publisher** with a v1 file:

1. Run the migrate check (`llmsx migrate <url|file>`; today, `llms_lint.py check <file>`
   with `--check-links`). It is the lint with a V1→V2 lens: findings are mapped to the steps
   below.
2. If the report says *full file wearing the wrong name* — page bodies inside `llms.txt`, a
   file over 100 KB (I6) — split it into `llms.txt` + `llms-full.txt`. `docset_refine export`
   does this from a mirror; by hand, the index keeps the link lines and the full file takes
   the bodies in the Mintlify grammar (`# Title` / `Source: <url>` / blank / body).
3. Add `.md` twins for every linked page (either form) and the two `Link` headers (H3).
4. Move skippable material — changelog, legal, old posts — to a trailing `## Optional` (N4).
5. If the index is over 10 KB: hub-and-spoke split. `## Sections` in the root, one
   `<slug>/llms.txt` per section, counts on every section line.
6. Re-lint. The bar is 0 High.

For a **hub user** with V1 folders: run `extract → render → export` on each mirror, then
`docset_rollout cleanup` to retire the V1 artifacts. Cleanup only removes a site's `.pages/`,
`_master.md` and `_distill_index.json` once a fact layer exists for it, so nothing is lost
before its replacement is in place.

For most 2025 files the report's first line is "nothing required". The recommendations that
follow are the twins and the headers, because those are what v2 added for consumers to find
the file at all.

## Compatibility matrix

Rows are producer choices; columns are consumers. The cells are dated evidence (verified
2026-08-30) and need re-checking every 90 days, because consumer behaviour is the part of
this table nobody controls.

| producer choice | Claude Code (`WebFetch` / hub MCP) | Cursor | generic MCP client | `llms_acquire` | lint | Lighthouse agentic audit |
|---|---|---|---|---|---|---|
| v1 file at root | works | works | works | works | works (I2 Medium if no blockquote) | works |
| v2 file at root | works | works | works | works | works | works |
| v2 file at a subpath only | works if given the URL | degraded — no root discovery | works if given the URL | works — the ladder probes the given path | works | degraded — expects the root |
| split root (`## Sections`) | works — follows section links | works — one extra hop | works | works — recurses by path then `part-N` | works — `check DIR` walks sections | works |
| family file (links only indexes) | works | works | works | works | works (F1 requires index targets) | not evaluated |
| `llms-full.txt`, Mintlify grammar | works via `hub_llms_full_read(page=…)` | degraded above ~50k tokens (the consumer ceiling) | works | works — `split_llms_full` round-trips | works | not evaluated |
| `llms-full.txt`, YAML-block or Cloudflare frontmatter grammar | works | degraded above ~50k tokens | works | works — grammar detected from the header comment | works | not evaluated |
| `llms-full.txt` served **as** `llms.txt` | degraded — the index is unreadable at that size | breaks | degraded | works — detected and split | **High** (I6) | breaks |
| no `.md` twins | works — fetches HTML | works | works | degraded — falls to the `Accept` probe or the crawl | High (N6, with `--check-links`) | degraded |
| no `Link` headers | works | works | works | works | Low (H3) | degraded |

Read down a column to see what a given consumer needs; read across a row to see what a given
choice costs. The only cell that breaks *everything* is the full file served under the
index's name.

## What breaks

Honest list of what does not survive the two transitions.

On the **spec** side, nothing a v1 file relied on is invalidated, but two things stop
meaning what they meant:

- **`## Optional` is no longer mechanical.** A consumer that skipped it "when context is
  short" was implementing v1's `llms_txt2ctx`; that tool is out of the proposal. Keep the
  section, keep it last, and do not put reference material in it — but expect nothing to
  skip it for you.
- **"Expand into context" is no longer the reading model.** A file designed to be inhaled
  whole (long descriptions, page bodies, every URL on the site) is now a full file wearing an
  index's name. Under v2 it is expected to be searched and followed, so it has to be small.

On the **pipeline** side:

- **The V1 artifacts are gone once a fact layer exists.** `<stem>.pages/`, `_master.md` and
  `._distill_index.json` are removed by `docset_rollout cleanup`. Anything that read them
  reads `<stem>.reference/` and `<stem>.llms/` now.
- **The raw vector layer is no longer the default answer.** `query --layer auto` prefers the
  facts layer when one exists. A query that used to return a text chunk now returns a
  source-anchored unit; callers that parsed the chunk shape need the unit shape.
- **The two embedding models do not mix.** `hub.db` file vectors are `nomic-embed-text`
  (768d); every docset and semantic-ops store is `mxbai-embed-large` (1024d). This was true
  in V1 too, but V2 added stores, so there are more places to get it wrong. Mixing them
  returns nothing, silently.
- **Serving moved from HTML to markdown-with-headers.** `web-text-mirror --serve` returned
  HTML; `llms_serve.py` returns `text/markdown; charset=utf-8` with `X-Markdown-Tokens` and
  `Link: rel="describedby"`. A client that scraped the HTML view has to read markdown.

What does *not* break, and is worth saying plainly: every v1 `llms.txt` still parses, still
lints, and still answers questions for an agent handed its URL. The migration guide above is
a list of recommendations for one, not repairs.

# The vocabulary file
Source: https://llms-explorer.com/essays/vocabulary/

# The vocabulary file

llms-vocabulary.txt is the lexical layer of a family: the words a field uses, what each means here, how it differs from its neighbours, and what people say instead. The line grammar, the sense model, where it feeds, and how to build one.

"Cookie" is an HTTP state token in a web docset, a monster in one children's canon, and a
snack in a recipe corpus. An index does not settle which one a family means; a facts file
mentions all three without noticing. The vocabulary file is the layer that pins the sense,
lists the surface forms people actually type, and says what the term is *not*. It is neither
index nor facts. It is what makes both findable and unambiguous.

## What a vocabulary file is

`llms-vocabulary.txt` is one line per term of a family, each line carrying: the canonical
name, a definition taken from a kept unit, the neighbours it is easy to confuse it with
(`not:`) and how it differs, the words people say instead (`aka:`), and the URL of the unit
the definition came from. The file is shaped like a spec-v2 llms file — H1, blockquote,
generator banner, H2 sections — so any reader that opens an index can open this.

Three properties make it different from a glossary someone typed:

- **Every definition is extractive.** It comes from a `definition` unit or an "X is/are …"
  sentence in the pool, and the line ends with that unit's anchor. A term the pool names but
  never defines goes to `## Named, not yet defined` with its hit count, and never receives a
  model-invented definition.
- **Every alias appeared.** `aka:` entries are surface forms found in the pool — backticked
  tokens clustered by normalised spelling, with the most frequent surface as the canonical
  name and the rest as aliases. No synonym list is imported from outside.
- **Contrast is first-class.** `not:` comes from contrast cues in the units themselves —
  *not*, *unlike*, *vs*, *rather than*, *instead of*, *not to be confused with* — so the file
  says "small is not full" because a source said it, and can point at where.

Sources are ranked by trust, deterministic first and a model last:

1. concept-tree node names and their existing `aliases` (the subject and its children);
2. backticked tokens the pool keeps (file names, headers, paths, tools), seen at least
   twice, clustered by spelling;
3. `definition` units and "X is/are …" sentences → definitions; contrast cues → `not:`;
4. `--llm`: the local model writes a missing definition or differentiator from at most six
   units that mention the term — and every name it returns must appear in those units, or the
   line is dropped. Lines that reach the file this way are marked `origin: llm` with a
   grounding score, and below the floor they carry *verify before citing*.

## The line grammar

The full grammar, with every optional field shown:

```
# <Family> — vocabulary
> <n> terms of <family>; canonical name, definition, how it differs (not:), what people say instead (aka:). Each line anchored to the unit it came from.

## Terms
- **<term>** [<sense-id>] (<pos>): <definition> — <url#anchor> · aka: a, b · not: <neighbour> — <how it differs> · ant: <antonym> · broader: <term> · narrower: <term>, <term> · related: <term> · measure: <unit> · field: <family-slug> · verified-as-of: <date>

## Homonyms
- **cookie** [web.cookie] · [folklore.cookie-monster] · [food.cookie]: <one line per sense with its family> — the sense picker's data

## Named, not yet defined
- <term> — seen in <n> units, no definition unit found (evidence rule)
```

Field by field:

| field | required | comes from | rule |
|---|---|---|---|
| `**term**` | yes | tree node or canonical token | one line per term per sense |
| `[sense-id]` | in a multi-family file | `<family-slug>.<term-slug>` | disambiguates the pair (term × family) |
| `(pos)` | no | part of speech | noun unless stated |
| `definition` | for a `## Terms` line | a kept unit | must be extractive; its anchor is the line's source |
| `— url#anchor` | with a definition | the unit's source | resolves to a heading on the page (P7) |
| `aka:` | no | surface forms in the pool | never imported; the FTS5 layer expands through these |
| `not:` … `— how` | no | contrast cues | the neighbour and one clause on the difference |
| `ant:` | no | explicit antonyms | proposed extension |
| `broader:` / `narrower:` / `related:` | no | the abstractor's relation taxonomy | proposed extension |
| `measure:` | no | the unit a quantity is stated in | proposed extension |
| `field:` | no | the family slug | redundant with the sense id; kept for grep |
| `verified-as-of:` | no | an actual re-fetch | a date bump without a fetch is not evidence |

Two honest notes on the grammar. The builder that exists today writes `definition`, `aka:`,
`not:` (with `differs:` for the how-clause), the source anchor, and the `origin: llm` marker;
`ant:`, `broader:`, `narrower:`, `related:` and `measure:` are proposed extensions that the
concept abstractor's relation taxonomy is expected to supply. And **contranyms** — a word whose
senses oppose each other (*sanction*, *cleave*, *oversight*) — are two sense lines under
`## Homonyms` marked `contranym`, because a sense picker that offered only one would be
wrong half the time.

A line in the **target** grammar, with the proposed fields shown, for the term
`llms-small.txt`:

```
- **llms-small.txt** [llms.small] (noun): the budgeted variant of a full file — reference-class pages first, within about 50k tokens — /reference/formatting/#3-the-budgeted-file--llms-smalltxt · aka: small, llms-small · not: llms-full.txt — full is every page with no budget; small is a selection that fits a consumer's stable window · broader: llms-full.txt · related: manifest.json · measure: tokens (chars/4) · field: llms-txt
```

And the line the builder actually wrote for the same term in the llms.txt family's own
`llms-vocabulary.txt`, abridged — no sense id, no `broader:`/`related:`/`measure:`, and the
how-clause under `differs:` rather than after a dash:

```
- **llms-small.txt** — llms-small.txt is a small variant of a tokenized text file used to enforce size budgets on the producer-side. · not: /_llms/, x-markdown-tokens, llms.txt, x-max-tokens · differs: not consumer-side truncation … — https://www.mintlify.com/docs/ai/llmstxt · evidence: hub estate · origin: llm (grounded 0.64)
```

The gap between the two is the honest state of the builder: the required fields ship, the
relation fields do not yet. Either line tells an agent that "small" in a query is this file
and that "full" is the neighbour it is contrasted with; only the first tells it that size is
counted in tokens at four characters each.

## Senses across fields

A sense id is `<family-slug>.<term-slug>`. A term is disambiguated by the pair (term ×
family): *cookie* in the `web` family is `web.cookie`, in a folklore family
`folklore.cookie-monster`, in a recipe family `food.cookie`. The three vocabularies are built
independently; the cross-family pass finds homonyms by matching term slugs across them and
writes a `## Homonyms` line listing every sense with its family.

What a consumer does with that depends on its scope:

- **scoped to a family** — the query gets that family's sense and the others are invisible;
  "cookie expiry" inside a web docset never sees the snack;
- **unscoped** — the query gets the sense picker: the homonym line, one sense per row, and
  the consumer (or the reader) chooses;
- **crossing families on purpose** — an agent that switches from the web docset to a recipe
  corpus reads the new family's vocabulary first and stops guessing.

The model has a known seam. Keying senses by family may split a term that is really one
sense across two families — `Link` header in an HTTP family and in an llms-txt family are the
same header. A "same-as" link between senses is the obvious fix and is left open; until it
exists, the homonym line simply lists both.

## Where it feeds

The vocabulary was built because three consumers were weak without it:

| consumer | what it takes | what changes |
|---|---|---|
| **assignment** — the topical builder's keyword pass | `aka:` lists, merged into the concept-tree node's `aliases` by `--register` (add-only) | a fact that says "session cookie" is filed under the node named "cookie" instead of falling to `## Shared` |
| **keyword** — the FTS5 layer | `aka:` surfaces of a matched term, OR-ed into the query (**designed**: an `expand` flag on `hub_query_docset`, which today takes only `docset, question, top, layer, mode`) | an exact-token search for `X-Markdown-Tokens` would also find lines that wrote "the tokens header" |
| **descriptions** — the index exporter | the canonical definition | the one-liner after a link in `llms.txt` is the definition the pool agreed on, not a generated paraphrase |

A fourth consumer is the concept abstractor, which seeds its lexicon — synonyms, parts,
sub-types, contrasts — from the family's vocabulary before it harvests, and a fifth is the
precedence ladder in the [CLLMS essay](/essays/cllms-vs-proprietary/), whose rung 4 is
"agreement with the canonical definition" — which is a lookup in this file.

The acceptance bar for the keyword consumer is written down but not yet measured, because
server-side expansion is not shipped: when it lands, on the P12 question bank an expanded query
must raise exact-token recall by at least one hit per family without losing any. Until then a
client can do the expansion itself — read the term's `aka:` list out of the vocabulary file and
send the surfaces as one `mode="keyword"` query.

## Build one

The walkthrough below builds the llms.txt family's own vocabulary — the terms are *index,
full, small, facts, twin, describedby, family, split root, unit, anchor* and their
neighbours. It is the same procedure for any field.

1. **Assemble the pool.** Any mix of `units.jsonl`, `llms-facts.txt` and hub spokes. The
   pool is what the file may cite; nothing outside it can become a definition.

   ```
   PYTHONPATH=scripts .venv/bin/python -m docset_refine vocabulary \
     --from outputs/exports/site.llms/llms-facts.txt \
     --from skills/llms-deep-optimizer/references/attributes.md \
     --subject "llms.txt" \
     --out llms-topical/llms-txt.llms/
   ```

2. **Read the candidates.** The builder proposes terms from the tree, then from backticked
   tokens seen at least twice, then from definitions and contrast cues. A term with no
   definition lands in *Named, not yet defined* — that list is the research gap, not an
   error.

3. **Decide about `--llm`.** Without it, every line is deterministic. With it, the local
   model writes the missing definitions from at most six evidence units each, and anything
   it names that the units do not contain is dropped. Lines it wrote are marked, and the
   floor (`--floor`) decides which ones say *verify before citing*.

4. **Lint.** `llms_lint.py check llms-vocabulary.txt --kind vocabulary` parses every line
   against the grammar and checks that every definition's anchor resolves. The site's CI
   runs this on its own file; 0 High is the bar.

5. **Register.** `--register` merges each term's `aka:` into the matching concept-tree
   node's `aliases`. From then on the topical builder's keyword pass matches the synonyms,
   and the FTS5 layer can expand through them.

6. **Serve.** The file lands beside the family's other files and is served at
   `/t/<slug>/llms-vocabulary.txt` with the same markdown headers as everything else.

The pilot bar for the llms.txt family is at least 40 terms, at least 5 `not:` contrasts, and
the cookie-style homonym demo across at least two families. The [glossary
page](/reference/glossary/) is a hand page — the terms in the sense this site uses them — and
the site's generated `llms-vocabulary.txt` is its machine twin; the two are checked against each
other, not derived one from the other.

# Which layer answers which question
Source: https://llms-explorer.com/examples/decision-table/

# Which layer answers which question

The decision table for the cookbook: match the shape of your question to the cheapest llms layer that answers it, then open the recipe.

Every llms family has a ladder of layers — index, split root, small, full, facts, topical,
vocabulary — and beside them two retrieval modes over the facts, keyword and vector. The
cheapest layer that answers your question is the right one. This table matches the shape of
a question to that layer and to the recipe that shows it. Recipes are copy-only in this step:
the code illustrates, the cost line states what was measured or estimated, and each recipe
closes with the note that it becomes runnable in the playground step.

## The table

| question shape | layer | cost class | recipe |
|---|---|---|---|
| Orientation before any retrieval: what does this site cover, where do I start | `llms.txt` (≤ 10 KB) then ≤ 2 hops to a `.md` twin | ~3k tokens, 3 requests, 0 embeddings | recipe-01 |
| Orientation on a site whose index split into sections (`## Sections` present) | split root: root index → `<slug>/llms.txt` → page | ~3–5k tokens, 3–4 requests | recipe-02 |
| An exact token: an env var, a flag, a header name, an error string | keyword layer (`mode="keyword"`, FTS5/BM25) over `llms-facts.txt` | microseconds, 0 model tokens, 0 embeddings | recipe-03 |
| A paraphrased question, or mixed / unsure whether the words match the source | hybrid (`mode="hybrid"`, RRF over keyword + vector), or vector alone (`layer="facts"`) | 1 embedding, 0 generation tokens | recipe-04 |
| An agent that must find the right page from an MCP client without a search index | index-first via `hub_docset_index` → `sections` → section index → page | ~2k tokens read per hop, 0 embeddings | recipe-05 |
| A scripted check or query from a shell or a CI step | the `llmsx` CLI (today: the hub scripts it wraps) | seconds; 0 model tokens for lint / keyword | recipe-06 |
| Citation-grade answers inside your own RAG store | `llms-facts.txt` units, one document each, `url#anchor` as metadata | 1 embedding per unit at ingest; ~845k tokens for a 191-page site | recipe-07 |
| Keeping a published file honest on every push | the lint as a GitHub Action gate (exit 1 on High) | ~10 s per file; network only with `--check-links` | recipe-08 |
| Serving the files so agents and the lint can find them | headers: `text/markdown`, `X-Markdown-Tokens`, `Link: rel="describedby"`, `rel="alternate"` on HTML | one config block; verify with `curl -I` | recipe-09 |
| Whole-corpus reasoning, offline and private, within a token budget | a local hub: Ollama + indexer + keyword layer + `llms_serve.py`; `llms-small.txt` for budgeted reads | one machine; ~50k tokens per small read, 0 API spend | recipe-10 |
| One concept across many sources, disagreements visible | a topical file (`/t/<slug>/`) built from a fact pool | minutes to build; `--no-embed` for 0 embeddings | recipe-11 |
| Disambiguation: which sense of a word this family means, and its aliases | `llms-vocabulary.txt` senses and `aka:` expansion before FTS5 | free: string match, 0 model tokens | recipe-12 |

## How to read it

- **Start at the top.** The first three rows are almost always enough. An index read plus
  two hops answers "what is here"; a keyword lookup answers "what is the exact flag". Only
  when the words in your question may not be the words in the source do you pay for an
  embedding.
- **Cost class is honest, not precise.** Token counts are chars/4, the same estimator the
  hub writes into `manifest.json`. Where a recipe has a measured figure it says *measured*;
  where it does not, it says *estimated*. The CI in a later step replaces every estimate with
  a run.
- **Two shapes have no recipe of their own.** *Whole-corpus reasoning* against a hosted
  family is `hub_docset_index(docset, file="llms-small.txt")` (recipe-05 shows the call);
  a *full-file read* is `hub_llms_full_read(key, page=…)` for one page at a time (recipe-03
  shows it). Both are one call, not a recipe.
- **Every layer's numbers, for one real family.** From
  `outputs/exports/code.claude.com.llms/manifest.json` (191 pages, acquired from the
  publisher's `llms-full.txt`): root index ~280 tokens; the three section indexes ~167,
  ~1,615 and ~1,388 tokens; `llms-small.txt` ~49,785 tokens; `llms-full.txt` ~2,097,403
  tokens; `llms-facts.txt` 14,031 units, ~844,553 tokens. The gap between the first line and
  the last is the whole argument for reading the index first.

## When the table is the wrong tool

If the question is "is this file any good", none of these rows apply — that is the
[lint](/reference/passes/), not a retrieval. If the question is "what do many sites say
about X", the source-axis rows do not apply either; recipe-11 and the [CLLMS
essay](/essays/cllms-vs-proprietary/) cover the concept axis. And if the corpus is not
published as an llms family at all, the first step is to make one (recipe-10 in miniature,
`docset_refine export` at scale), after which every row above starts to work.

# Recipe 01 — Two hops with requests
Source: https://llms-explorer.com/examples/recipe-01/

# Recipe 01 — Two hops with requests

Read a site's llms.txt, pick a page by its description, fetch the .md twin, answer. The baseline every other recipe is measured against.

## Goal

Answer a question about a documentation site using only its `llms.txt` and one page, with
nothing but `requests`. This is the reading model spec v2 describes — "view or search the
index, then follow the relevant links" — and it is the floor for cost: if a question can be
answered this way, no retrieval layer beats it.

## When not to use it

- The question is an exact token (an env var, a flag, a header). Descriptions rarely contain
  them; use the keyword layer ([recipe-03](/examples/recipe-03/)).
- The index has a `## Sections` block instead of page links. That is a split root; add the
  section hop ([recipe-02](/examples/recipe-02/)).
- You need more than about two pages. Past that, read `llms-small.txt` or query the facts
  layer instead of hopping.

## Steps

1. GET `/llms.txt`. Parse the link lines with the same regex the lint uses (`LINK_RE` in
   `llms_lint.py`), so anything that lints as a link parses as one here.
2. Score each line's name and notes against the question's tokens; take the best.
3. GET that URL with `.md` appended — after stripping any trailing slash, because a v2 twin
   is `/reference/usage.md`, not `/reference/usage/.md`.
4. Hand the page to whatever answers — a model, a grep, a human.

```python
import re, requests

LINK_RE = re.compile(r"^\s*[-*]\s+\[([^\]]*)\]\(([^)\s]+)\)\s*(?::\s*(.*))?$")

def two_hop(root: str, question: str) -> tuple[str, str, str]:
    index = requests.get(f"{root}/llms.txt", timeout=10).text
    links = [m.groups() for m in map(LINK_RE.match, index.splitlines()) if m]
    q = set(re.findall(r"\w+", question.lower()))
    name, url, notes = max(
        links,
        key=lambda l: len(q & set(re.findall(r"\w+", f"{l[0]} {l[2] or ''}".lower()))),
    )
    if not url.startswith("http"):
        url = f"{root}/{url.lstrip('/')}"
    twin = url if url.endswith(".md") else url.rstrip("/") + ".md"
    page = requests.get(twin, headers={"Accept": "text/markdown"}, timeout=10).text
    return url, twin, page

url, twin, page = two_hop(
    "https://llms-explorer.pages.dev",
    "how do I serve a markdown twin with the right headers?",
)
print(url, twin)
```

The scorer is deliberately naive — a bag-of-words overlap. It is enough when the index's
descriptions are extractive (the lint's D-attributes exist to make them so), and it fails
loudly when they are not, which is a finding about the file rather than the code.

## Expected output

The two URLs and the page text. Against this site's own index the question above scores
`/examples/recipe-09/` highest — its title and description share more tokens with the question
than any other line — and the twin is that route with the slash traded for `.md`:

```
https://llms-explorer.pages.dev/examples/recipe-09/ https://llms-explorer.pages.dev/examples/recipe-09.md
```

The page comes back as markdown, not HTML: frontmatter, headings the facts file anchors to,
a few thousand characters. If the second request returns HTML, the site has no twins and the
`Accept` header was ignored — a serving finding (`N6` for a dead target, `H2` for the wrong
content type), not a parsing one.

Pointed at a **split** root the same code lands on a section index rather than a page —
`code.claude.com`'s root is `## Sections` with `overview/llms.txt` under it, so `two_hop`
returns the section file and you need the extra hop of

## Cost

Estimated: about 3.5k tokens — this site's index is 9,325 bytes (~2.3k tokens at chars/4;
the rubric's bar is 10 KB) plus one page (~1.1k tokens for the twin above). Two HTTP
requests, zero embeddings, zero model tokens until you hand the page to something.

# Recipe 02 — Split root: follow a section index
Source: https://llms-explorer.com/examples/recipe-02/

# Recipe 02 — Split root: follow a section index

When the root llms.txt has a ## Sections block, let the counts on each section line decide which section index to fetch before touching a page.

## Goal

Navigate a hub-and-spoke family: a root `llms.txt` whose links point at other indexes
(`<slug>/llms.txt`), each carrying its page and token counts, and only the section index
links pages. The extra hop costs one small request and saves reading an index that would
have been over 10 KB — the size at which `export_llms` splits (`INDEX_SPLIT_BYTES =
10_000`).

## When not to use it

- The root already links pages. Then it is a plain index; [recipe-01](/examples/recipe-01/)
  is one hop shorter.
- The root is a *family* file (its links are other sites' indexes, `## Shared` once). That
  is a different shape: pick the site first, then apply this recipe to it.
- You already know the page URL. Fetch the twin; the index is for choosing, not for
  confirming.

## Steps

1. GET the root. Detect `## Sections`. Each line reads
   `- [Name](slug/llms.txt): N pages, ~T tokens — first titles…`.
2. Choose a section by title overlap; prefer the smaller token count on a tie — the counts
   exist on the line so you can decide before fetching.
3. GET `<root>/<slug>/llms.txt`. If it is itself split (`part-N/llms.txt`), recurse — the
   exporter splits by path first, then by parts of 60 pages.
4. Pick the page from the section index exactly as in recipe-01 and fetch its twin.

```python
import re, requests

LINK_RE = re.compile(r"^\s*[-*]\s+\[([^\]]*)\]\(([^)\s]+)\)\s*(?::\s*(.*))?$")
COUNT_RE = re.compile(r"(\d[\d,]*)\s+pages?,\s*~?(\d[\d,]*)\s+tokens?")

def links_under(text, heading):
    block, on = [], False
    for line in text.splitlines():
        if line.startswith("## "):
            on = line[3:].strip() == heading
            continue
        if on and (m := LINK_RE.match(line)):
            block.append(m.groups())
    return block

def pick_section(root, question):
    idx = requests.get(f"{root}/llms.txt", timeout=10).text
    sections = links_under(idx, "Sections")
    if not sections:
        return None  # plain index: use recipe-01
    q = set(re.findall(r"\w+", question.lower()))
    def score(l):
        name, url, notes = l
        m = COUNT_RE.search(notes or "")
        tokens = int(m.group(2).replace(",", "")) if m else 10**9
        overlap = len(q & set(re.findall(r"\w+", f"{name} {notes or ''}".lower())))
        return (overlap, -tokens)
    name, url, notes = max(sections, key=score)
    section_index = requests.get(f"{root}/{url}", timeout=10).text
    return name, url, section_index

print(pick_section("http://127.0.0.1:8788/d/code.claude.com", "track cost and usage in the SDK")[:2])
```

## Expected output

For the `code.claude.com` export the root has three sections. The question above overlaps
the *Agent Sdk* line ("How the agent loop works, Use Claude Code features in the SDK, Track
cost and usage and 28 more"), so the second hop is `agent-sdk/llms.txt`:

```
('Agent Sdk', 'agent-sdk/llms.txt')
```

That section index is 31 pages, ~1,615 tokens, and links the page directly. Compare the
*Overview* section: 137 pages, ~7,460 tokens, itself split into three `part-N` indexes —
the counts on the root line are what let you avoid it.

## Cost

Measured from the family's `manifest.json`: root ~280 tokens, `agent-sdk/llms.txt` ~1,615
tokens, then one page. About 3–5k tokens total, four requests, zero embeddings. The
section hop is cheaper than the alternative in every case the split exists for, because the
split only happens when the unsplit index would exceed the spec-sized 10 KB.

# Recipe 03 — Keyword layer from Claude Code
Source: https://llms-explorer.com/examples/recipe-03/

# Recipe 03 — Keyword layer from Claude Code

Find an exact token — an env var, a flag, an error string — with hub_query_docset(mode=\"keyword\"), then open the page it came from. Zero model tokens.

## Goal

Answer "where is `CLAUDE_CODE_SYNC_SKILLS` documented?" without an embedding call. The
keyword layer is an FTS5 index (BM25 ranking) built beside each docset's vector layer; a query
in `mode="keyword"` is a sub-token match over the facts layer, returns the unit's snippet and
its `url#anchor`, and costs microseconds.

## When not to use it

- The question is a paraphrase. "The variable that pulls my claude.ai skills down" contains
  none of the tokens in `CLAUDE_CODE_SYNC_SKILLS`; use hybrid
  ([recipe-04](/examples/recipe-04/)).
- The docset has no facts layer yet. Keyword still works over the raw chunks, but the hits
  are chunks, not anchored units — `layer` in the reply says which answered.
- You want the whole page. The keyword hit tells you *which* page; the read is the second
  call below.

## Steps

1. Call `hub_query_docset` with the token and `mode="keyword"`. `docset` is the store key,
   `<host-slug>__<mirror-stem-slug>` — for this mirror `codeclaudecom__codeclaudecom`, not the
   host name; a friendly name returns `no such docset`. The index is built on first use
   (`docset_indexer keyword-index` under the hood), so the first call on a docset is slower
   once.
2. Read `url` and `snippet` off the top hit — those are the fields a keyword hit carries
   (`score`, `url`, `seq`, `snippet`); `text`, `unit_type` and `origin` come back from the
   semantic and hybrid legs instead. The URL carries the anchor.
3. Open the page with `hub_llms_full_read(key, page=<url>)`. That `key` is the llms-full
   catalog key from `hub_llms_full_list` (`code.claude.com__docs`), a different namespace from
   the docset key; `page` matches by exact source URL or a case-insensitive title substring.

From a Claude Code session with the `global_ai_hub` server connected:

```
hub_query_docset(docset="codeclaudecom__codeclaudecom", question="CLAUDE_CODE_SYNC_SKILLS", mode="keyword", top=3)
```

The reply is JSON. Verbatim, run against this hub on 2026-08-31 — nothing dropped:

```json
{
  "docset": "codeclaudecom__codeclaudecom",
  "layer": "facts",
  "queried": "codeclaudecom__codeclaudecom__facts",
  "mode": "keyword",
  "results": [
    {
      "score": 15.2427,
      "url": "https://code.claude.com/docs/en/env-vars#variables",
      "seq": 8391,
      "snippet": " … Timeout in milliseconds for a mid-session skills resync when `[CLAUDE_CODE_SYNC_SKILLS]` is set (default: 30000). Bounds the download triggered when the … "
    },
    {
      "score": 14.651,
      "url": "https://code.claude.com/docs/en/env-vars#variables",
      "seq": 8392,
      "snippet": " … Timeout in milliseconds for the first query to wait for the initial skill list when `[CLAUDE_CODE_SYNC_SKILLS]` is set (default: 5000). When … "
    },
    {
      "score": 14.3472,
      "url": "https://code.claude.com/docs/en/env-vars#variables",
      "seq": 8390,
      "snippet": " … Before it runs the first query, Claude Code waits up to `[CLAUDE_CODE_SYNC_SKILLS]_WAIT_TIMEOUT_MS` for the list of your skills … "
    }
  ]
}
```

The square brackets are FTS5's `snippet()` match markers, not part of the text; the leading
and trailing ` … ` are its elision markers, so a snippet is a window around the match rather
than the whole unit.

Read the ranking honestly: all three hits are on the same page, and the variable you asked
for is *third*. Units 8391 and 8392 are `CLAUDE_CODE_SYNC_SKILLS_INSTALL_TIMEOUT_MS` and
`…_WAIT_TIMEOUT_MS`; unit 8390 is `CLAUDE_CODE_SYNC_SKILLS` itself. BM25 rewards the token in
a shorter field, so the two timeout units outscore the variable they refer to. That costs
nothing here, because all three carry the same `url#anchor` — keyword tells you *which page*,
and the anchor is identical whichever of the three you take.

Then, to open the page behind the top hit:

```
hub_llms_full_read(key="code.claude.com__docs", page="https://code.claude.com/docs/en/env-vars")
```

That returns an envelope, not raw markdown. The real fields on this call, with `text` cut
here (it is 20,000 characters):

```json
{
 "key": "code.claude.com__docs",
 "url": "https://code.claude.com/docs/llms-full.txt",
 "page_title": "Environment variables",
 "page_url": "https://code.claude.com/docs/en/env-vars",
 "total_chars": 475588,
 "truncated": true,
 "text": "Reference for environment variables that control Claude Code behavior.\n\nEnvironment variables can control Claude Code behavior such as model selection, authentication, request routing, and feature toggles. …  ← truncated here for the page; the tool returned 20,000 characters"
}
```

The `# Title` / `Source:` lines are the grammar the tool *matches* on inside `llms-full.txt`;
they are lifted into `page_title` and `page_url` rather than left in `text`, so `text` starts
at the page body. `total_chars` is the page's full length and `truncated` says whether you got
all of it — continue with `offset`, or raise `limit`.

The same two calls from the shell, for a script:

```
.venv/bin/python scripts/docset_indexer.py keyword codeclaudecom__codeclaudecom "CLAUDE_CODE_SYNC_SKILLS" --layer facts --mode phrase --top 3
```

`--mode` is `any | all | phrase | raw`; `phrase` is what an exact token wants.

## Expected output

Every hit's URL ends in the heading that documents the variable
(`/docs/en/env-vars#variables`), and the reply's `layer` names which layer answered (`facts`)
while `queried` names the collection it actually read (`…__facts`). The corresponding line in
the export's `llms-facts.txt` is typed `[parameter]` — line 11,378 of that file begins
``- [parameter] `CLAUDE_CODE_SYNC_SKILLS`: Set to `1` …`` — but keyword hits do not carry the
type, so ask for `mode="hybrid"` or `"semantic"` when you want `unit_type` on the hit. What a
keyword hit does carry is exactly `score`, `url`, `seq`, `snippet`, as above.

The full-read then lands you on that page: `page_title` comes back as `"Environment
variables"`, `page_url` as the URL you asked for, so the anchor from the hit resolves to a
heading you can see in `text`.

If `layer` says `raw`, the docset has no facts layer; the hit is still correct but is a text
chunk without an anchor. That is the signal to run `extract → export` on it.

## Cost

Measured on the hub: the FTS5 lookup is sub-millisecond after the index exists; the first
call on a docset builds the index (seconds for this family's 14,031 units). Zero model
tokens, zero embeddings. The page read is bounded by `limit`, which defaults to 20,000
characters — about 5k tokens — and is capped at 200k by the tool. The env-vars page above is
475,588 characters, so the default read came back `"truncated": true`; a shorter reference
page returns whole.

# Your account
Source: https://llms-explorer.com/account/

# Your account

Who you are signed in as, which plan you are on, and the sign-in methods and private tree forks attached to the account — all fetched in the browser.

The address, the plan and the attached sign-in methods are one visitor's, so they are requested from the API after the page loads rather than built into it.

## What the account holds

Three things the public site has no place for: the plan and its quotas, the API keys that authenticate the hosted MCP endpoint, and the private tree forks whose changes are proposed back rather than published. Deleting the account revokes every key with it.

# Semantic indexing, recorded
Source: https://llms-explorer.com/demo/

# Semantic indexing, recorded

One question set run three ways against one indexed docset — keyword (BM25), vector, and the fusion of both — hits and timings as recorded.

Data recorded 2026-08-31; twin built 2026-09-02.

What this section is and how it is built: [Semantic indexing: two legs and a fusion](https://llms-explorer.com/essays/semantic-indexing/).

## What this section holds (11)

- [CLAUDE_CODE_SYNC_SKILLS](https://llms-explorer.com/demo/)
- [allowUnsandboxedCommands](https://llms-explorer.com/demo/)
- [--append-system-prompt](https://llms-explorer.com/demo/)
- [extraKnownMarketplaces](https://llms-explorer.com/demo/)
- [--output-format stream-json](https://llms-explorer.com/demo/)
- [Install Claude Code on Windows with PowerShell](https://llms-explorer.com/demo/)
- [PreToolUse hook exit codes and meanings](https://llms-explorer.com/demo/)
- [Which hook events fire once per turn](https://llms-explorer.com/demo/)
- [Run headless in CI and get JSON output](https://llms-explorer.com/demo/)
- [SessionStart versus UserPromptSubmit](https://llms-explorer.com/demo/)
- [Check the installed version and update](https://llms-explorer.com/demo/)

# This site's llms family
Source: https://llms-explorer.com/family/

# This site's llms family

The five files an agent reads, what each one is for, and the index rendered as clickable links rather than the raw text/markdown a browser cannot follow.

The family is served as `text/markdown`, which is right for the agents it is written for and unreadable in a browser: the links inside arrive as text. This route is the reader for it, and it renders the file it links rather than a second copy of the index — there is only one llms.txt, and it is the one the lint gates.

## What is on it

A table of the five members and what each is for, the index fetched and rendered with its links clickable, and a note on the `.md` twin every content page publishes beside itself.