# The vocabulary file llms-vocabulary.txt is the lexical layer of a family: the words a field uses, what each means here, how it differs from its neighbours, and what people say instead. The line grammar, the sense model, where it feeds, and how to build one. "Cookie" is an HTTP state token in a web docset, a monster in one children's canon, and a snack in a recipe corpus. An index does not settle which one a family means; a facts file mentions all three without noticing. The vocabulary file is the layer that pins the sense, lists the surface forms people actually type, and says what the term is *not*. It is neither index nor facts. It is what makes both findable and unambiguous. ## What a vocabulary file is `llms-vocabulary.txt` is one line per term of a family, each line carrying: the canonical name, a definition taken from a kept unit, the neighbours it is easy to confuse it with (`not:`) and how it differs, the words people say instead (`aka:`), and the URL of the unit the definition came from. The file is shaped like a spec-v2 llms file — H1, blockquote, generator banner, H2 sections — so any reader that opens an index can open this. Three properties make it different from a glossary someone typed: - **Every definition is extractive.** It comes from a `definition` unit or an "X is/are …" sentence in the pool, and the line ends with that unit's anchor. A term the pool names but never defines goes to `## Named, not yet defined` with its hit count, and never receives a model-invented definition. - **Every alias appeared.** `aka:` entries are surface forms found in the pool — backticked tokens clustered by normalised spelling, with the most frequent surface as the canonical name and the rest as aliases. No synonym list is imported from outside. - **Contrast is first-class.** `not:` comes from contrast cues in the units themselves — *not*, *unlike*, *vs*, *rather than*, *instead of*, *not to be confused with* — so the file says "small is not full" because a source said it, and can point at where. Sources are ranked by trust, deterministic first and a model last: 1. concept-tree node names and their existing `aliases` (the subject and its children); 2. backticked tokens the pool keeps (file names, headers, paths, tools), seen at least twice, clustered by spelling; 3. `definition` units and "X is/are …" sentences → definitions; contrast cues → `not:`; 4. `--llm`: the local model writes a missing definition or differentiator from at most six units that mention the term — and every name it returns must appear in those units, or the line is dropped. Lines that reach the file this way are marked `origin: llm` with a grounding score, and below the floor they carry *verify before citing*. ## The line grammar The full grammar, with every optional field shown: ``` # — vocabulary > terms of ; canonical name, definition, how it differs (not:), what people say instead (aka:). Each line anchored to the unit it came from. ## Terms - **** [] (): — · aka: a, b · not: — · ant: · broader: · narrower: , · related: · measure: · field: · verified-as-of: ## Homonyms - **cookie** [web.cookie] · [folklore.cookie-monster] · [food.cookie]: — the sense picker's data ## Named, not yet defined - — seen in units, no definition unit found (evidence rule) ``` Field by field: | field | required | comes from | rule | |---|---|---|---| | `**term**` | yes | tree node or canonical token | one line per term per sense | | `[sense-id]` | in a multi-family file | `.` | disambiguates the pair (term × family) | | `(pos)` | no | part of speech | noun unless stated | | `definition` | for a `## Terms` line | a kept unit | must be extractive; its anchor is the line's source | | `— url#anchor` | with a definition | the unit's source | resolves to a heading on the page (P7) | | `aka:` | no | surface forms in the pool | never imported; the FTS5 layer expands through these | | `not:` … `— how` | no | contrast cues | the neighbour and one clause on the difference | | `ant:` | no | explicit antonyms | proposed extension | | `broader:` / `narrower:` / `related:` | no | the abstractor's relation taxonomy | proposed extension | | `measure:` | no | the unit a quantity is stated in | proposed extension | | `field:` | no | the family slug | redundant with the sense id; kept for grep | | `verified-as-of:` | no | an actual re-fetch | a date bump without a fetch is not evidence | Two honest notes on the grammar. The builder that exists today writes `definition`, `aka:`, `not:` (with `differs:` for the how-clause), the source anchor, and the `origin: llm` marker; `ant:`, `broader:`, `narrower:`, `related:` and `measure:` are proposed extensions that the concept abstractor's relation taxonomy is expected to supply. And **contranyms** — a word whose senses oppose each other (*sanction*, *cleave*, *oversight*) — are two sense lines under `## Homonyms` marked `contranym`, because a sense picker that offered only one would be wrong half the time. A line in the **target** grammar, with the proposed fields shown, for the term `llms-small.txt`: ``` - **llms-small.txt** [llms.small] (noun): the budgeted variant of a full file — reference-class pages first, within about 50k tokens — /reference/formatting/#3-the-budgeted-file--llms-smalltxt · aka: small, llms-small · not: llms-full.txt — full is every page with no budget; small is a selection that fits a consumer's stable window · broader: llms-full.txt · related: manifest.json · measure: tokens (chars/4) · field: llms-txt ``` And the line the builder actually wrote for the same term in the llms.txt family's own `llms-vocabulary.txt`, abridged — no sense id, no `broader:`/`related:`/`measure:`, and the how-clause under `differs:` rather than after a dash: ``` - **llms-small.txt** — llms-small.txt is a small variant of a tokenized text file used to enforce size budgets on the producer-side. · not: /_llms/, x-markdown-tokens, llms.txt, x-max-tokens · differs: not consumer-side truncation … — https://www.mintlify.com/docs/ai/llmstxt · evidence: hub estate · origin: llm (grounded 0.64) ``` The gap between the two is the honest state of the builder: the required fields ship, the relation fields do not yet. Either line tells an agent that "small" in a query is this file and that "full" is the neighbour it is contrasted with; only the first tells it that size is counted in tokens at four characters each. ## Senses across fields A sense id is `.`. A term is disambiguated by the pair (term × family): *cookie* in the `web` family is `web.cookie`, in a folklore family `folklore.cookie-monster`, in a recipe family `food.cookie`. The three vocabularies are built independently; the cross-family pass finds homonyms by matching term slugs across them and writes a `## Homonyms` line listing every sense with its family. What a consumer does with that depends on its scope: - **scoped to a family** — the query gets that family's sense and the others are invisible; "cookie expiry" inside a web docset never sees the snack; - **unscoped** — the query gets the sense picker: the homonym line, one sense per row, and the consumer (or the reader) chooses; - **crossing families on purpose** — an agent that switches from the web docset to a recipe corpus reads the new family's vocabulary first and stops guessing. The model has a known seam. Keying senses by family may split a term that is really one sense across two families — `Link` header in an HTTP family and in an llms-txt family are the same header. A "same-as" link between senses is the obvious fix and is left open; until it exists, the homonym line simply lists both. ## Where it feeds The vocabulary was built because three consumers were weak without it: | consumer | what it takes | what changes | |---|---|---| | **assignment** — the topical builder's keyword pass | `aka:` lists, merged into the concept-tree node's `aliases` by `--register` (add-only) | a fact that says "session cookie" is filed under the node named "cookie" instead of falling to `## Shared` | | **keyword** — the FTS5 layer | `aka:` surfaces of a matched term, OR-ed into the query (**designed**: an `expand` flag on `hub_query_docset`, which today takes only `docset, question, top, layer, mode`) | an exact-token search for `X-Markdown-Tokens` would also find lines that wrote "the tokens header" | | **descriptions** — the index exporter | the canonical definition | the one-liner after a link in `llms.txt` is the definition the pool agreed on, not a generated paraphrase | A fourth consumer is the concept abstractor, which seeds its lexicon — synonyms, parts, sub-types, contrasts — from the family's vocabulary before it harvests, and a fifth is the precedence ladder in the [CLLMS essay](/essays/cllms-vs-proprietary/), whose rung 4 is "agreement with the canonical definition" — which is a lookup in this file. The acceptance bar for the keyword consumer is written down but not yet measured, because server-side expansion is not shipped: when it lands, on the P12 question bank an expanded query must raise exact-token recall by at least one hit per family without losing any. Until then a client can do the expansion itself — read the term's `aka:` list out of the vocabulary file and send the surfaces as one `mode="keyword"` query. ## Build one The walkthrough below builds the llms.txt family's own vocabulary — the terms are *index, full, small, facts, twin, describedby, family, split root, unit, anchor* and their neighbours. It is the same procedure for any field. 1. **Assemble the pool.** Any mix of `units.jsonl`, `llms-facts.txt` and hub spokes. The pool is what the file may cite; nothing outside it can become a definition. ``` PYTHONPATH=scripts .venv/bin/python -m docset_refine vocabulary \ --from outputs/exports/site.llms/llms-facts.txt \ --from skills/llms-deep-optimizer/references/attributes.md \ --subject "llms.txt" \ --out llms-topical/llms-txt.llms/ ``` 2. **Read the candidates.** The builder proposes terms from the tree, then from backticked tokens seen at least twice, then from definitions and contrast cues. A term with no definition lands in *Named, not yet defined* — that list is the research gap, not an error. 3. **Decide about `--llm`.** Without it, every line is deterministic. With it, the local model writes the missing definitions from at most six evidence units each, and anything it names that the units do not contain is dropped. Lines it wrote are marked, and the floor (`--floor`) decides which ones say *verify before citing*. 4. **Lint.** `llms_lint.py check llms-vocabulary.txt --kind vocabulary` parses every line against the grammar and checks that every definition's anchor resolves. The site's CI runs this on its own file; 0 High is the bar. 5. **Register.** `--register` merges each term's `aka:` into the matching concept-tree node's `aliases`. From then on the topical builder's keyword pass matches the synonyms, and the FTS5 layer can expand through them. 6. **Serve.** The file lands beside the family's other files and is served at `/t//llms-vocabulary.txt` with the same markdown headers as everything else. The pilot bar for the llms.txt family is at least 40 terms, at least 5 `not:` contrasts, and the cookie-style homonym demo across at least two families. The [glossary page](/reference/glossary/) is a hand page — the terms in the sense this site uses them — and the site's generated `llms-vocabulary.txt` is its machine twin; the two are checked against each other, not derived one from the other.