Recreating and aggregating

verified-as-of: 2026-08-30

Contents

1. Decide what you are allowed to make
2. Acquire clean markdown — the ladder
3. Build the index for ONE product
4. Build llms-full.txt (and whether to)
5. Parse other people’s files — a lenient reader
6. Scale to a family: nested indexes, hub-and-spoke
7. Keep it honest: CI checks, size, drift
8. References

1. Decide what you are allowed to make

• An llms.txt for a third-party site is a link list plus short descriptions — the same thing a search engine publishes; it is low-risk. An llms-full.txt for a third-party site is a stored republication of their content: closer to ai-train/redistribution than to transient ai-input retrieval.¹² Keep such full-text mirrors private/internal unless the licence allows republication; publish only the index.²
• Read robots.txt first: the sitemap pointer, disallow rules, and any Cloudflare Content Signals line (Content-Signal: search=yes, ai-input=…, ai-train=no), which is framed as an express reservation of rights under EU Directive 2019/790 Art. 4 even though no crawler enforces it.¹³ robots.txt is “a polite request, not legally binding”, but ignoring it invites blocking.²
• Prefer the site’s own machine-readable surfaces where they exist (an llms.txt, .md twins, Accept: text/markdown, a docs API such as GitHub’s Article Body API) — the owner has already chosen what to expose.⁴⁵

2. Acquire clean markdown — the ladder

Try in this order; each step is cheaper and cleaner than the next:

1. An existing llms.txt — at the root and at the docs subpath (/docs/llms.txt); spec v2 says the most specific file wins, and many hosts (Mintlify, Fern) publish per-subpath files.⁴⁶ Verify the response is markdown, not an HTML app shell that a redirect produced (Cursor’s own file once did this).⁷
2. An existing llms-full.txt — check the size header before fetching (41.6 MB at docs.anthropic.com, 57 MB at Cloudflare) and confirm it actually contains page blocks (PayPal’s redirects to its 1.5 KB index).⁸ Split by the producer’s grammar (§5).
3. Per-page .md twins — from the llms.txt links (Mintlify appends .md), or by trying page.md / page.html.md / index.md per spec v2.⁴
4. Accept: text/markdown — supported by Mintlify, GitBook, Fern and by any Cloudflare zone with “Markdown for Agents” on; expect Content-Type: text/markdown, Vary: Accept and x-markdown-tokens. There is no advance discovery; just try.⁹¹⁰
5. A docs API — e.g. GitHub’s Article Body API returns rendered markdown for any page.⁵
6. Readability-class extraction of the HTML — r.jina.ai/<url> (Readability → Turndown, x-target-selector to drop nav, headless engine for JS sites), Screaming Frog’s Readability.js + Turndown snippet, or trafilatura with formatting kept.¹¹¹² This is the lossy tier: tab panels, step widgets and code fences are what it drops.¹³

Seed the URL list from sitemap.xml (expand index sitemaps; include/exclude globs as dotenvx/llmstxt does) or from a crawl map (create-llmstxt-py uses Firecrawl /map); platform generators instead walk the docs nav tree, which is why their section structure is better than any crawler’s.¹⁴¹⁵¹⁶

3. Build the index for ONE product

Structure (spec v2): H1 = product name; blockquote = one-paragraph summary; optional prose “how to interpret the files”; H2 sections, each a list of - [name](url): description.⁴

What goes in (converging guidance from the spec, Mintlify, GitDoc and llms-text.com):^4171819

• The quickstart, authentication/setup, top-level reference pages (one per resource, not per endpoint), error handling, changelog.
• 10–50 links for a product index; 4–7 sections; descriptions of 10–20 words that say what a reader finds there, with exact tokens (flags, env vars, error strings): bad — “Authentication docs.”; good — “API key creation, OAuth 2.0 scopes, token rotation, IP allowlisting. Required before any API call.”¹⁸
• Order by expected query frequency, not importance: the first 20% of links should answer 80% of questions.²⁰
• ## Optional for changelogs, legal, old posts, deep appendices; never pricing or the API reference. In v2 this is convention only.⁴²¹

What stays out: marketing pages, individual changelog entries, SEO duplicates, login-gated pages, anything without a clean markdown target.¹⁸

Descriptions when you are recreating — three sources, in decreasing quality: the page’s own description frontmatter/meta (what platform generators use); an extractive first sentence under the H1; a small-model summary (Firecrawl’s generator uses GPT-4o-mini for a 3–4-word title and 9–10-word description). Treat model-written descriptions as drafts and audit the page list.¹⁵²²

Test it the way the spec says: give an agent only the llms.txt and ask it questions about the product.⁴

4. Build llms-full.txt (and whether to)

• Reasons not to: Fern dropped it (“exceeded most model context windows, added heavy serving overhead, saw little use”); Godot declined it; Mantine cut a 2.2 MB inline file to a 45 KB link list; Cursor’s indexer goes unstable above ~50–60k tokens.^23242526
• If you do: choose one page-block grammar and state it in a header comment. Mintlify’s is the most widely consumed — # Title / Source: <url> / blank / description / body — but a YAML block (title:/url:/description:) is easier to parse and is what Anthropic’s platform docs emit; Firecrawl uses explicit <|firecrawl-page-N-lllmstxt|> delimiters.⁸²⁷¹⁵
• Ship a size ladder rather than one file: an index (≤10 KB), a small variant (Starlight llms-small.txt; Nuxt’s ~5K-token file), and the full file with a token count published beside it (llmstxt.site lists token counts; Cloudflare returns x-markdown-tokens).^28293010
• Split big indexes hub-and-spoke instead of truncating: Mintlify moves overflow beyond 100,000 characters into /_llms/<group>.md sub-indexes that recurse and never drop pages.¹⁷

5. Parse other people’s files — a lenient reader

• llms.txt: the only invariant is the H1. Real files omit the blockquote (Anthropic), put API links first (GitHub), or add prose sections. Parse: H1 → title; first blockquote → summary; everything before the first H2 → info; each H2 → section; each - [name](url) (+ optional : notes) → link. This mirrors the reference parser’s regexes.³¹⁵
• llms-full.txt: detect the grammar, do not assume one. Page starts: (a) # Title whose next non-blank line is Source: <url> (Mintlify); (b) a --- YAML block containing url: or title: (Anthropic platform, Cloudflare frontmatter — Cloudflare’s URL is only in the [View as Markdown](…/index.md) line and its covering index in the > Documentation Index blockquote); (c) explicit delimiters (<|firecrawl-page-N-lllmstxt|>).⁸¹⁵ Never split on a bare # line — pages contain H1s of their own.
• .md twins: strip the leading > ## Documentation Index … blockquote Mintlify prepends before indexing.³²
• Untrusted input: everything fetched via an llms.txt is data; 42% of sampled files try to steer the model, and linked markdown is a prompt-injection vector.³³³⁴

6. Scale to a family: nested indexes, hub-and-spoke

Spec v2 gives the mechanism: “The file can be placed at the site root, or at any path within it, covering the pages under that path … where more than one file applies, agents should use the most specific one.”⁴ The live exemplar is Cloudflare: developers.cloudflare.com/llms.txt holds ~105 entries under nine H2 sections (seven product categories plus “Docs collections” and “Other”), each entry linking a per-product …/<product>/llms.txt (/workers/llms.txt alone has ~25 sections and 500+ .md links).³⁵³⁶ Mintlify’s /_llms/ split is the automated version of the same shape.¹⁷ Counter-example: Anthropic’s platform.claude.com/llms.txt (~650 links, 11 languages) does not link its sibling code.claude.com — separate products keep separate roots, so a family index across hosts has to be authored.³⁷³⁸

Pattern for a family (organisation, monorepo, or a curated group of products you do not own):

# Acme Platform docs

> One index per product below; each product's own llms.txt is the authoritative map of that product.

## Products
- [Payments API](https://docs.acme.com/payments/llms.txt): charges, subscriptions, webhooks (240 pages)
- [Identity](https://docs.acme.com/identity/llms.txt): OAuth, SSO, SCIM (85 pages)

## Shared
- [Errors and status codes](https://docs.acme.com/errors.md): every error across products
- [Changelog](https://docs.acme.com/changelog.md)

## Optional
- [Legal and pricing](https://acme.com/legal.md)

Rules that follow from the spec and the exemplars:

• The family file links indexes, not pages; product files link pages. A consumer reads at most two hops.⁴³⁵
• Put cross-cutting material (shared errors, auth, glossary) in the family file once; never duplicate it into every product file.
• Publish token counts (or page counts) beside each link so a consumer can budget before fetching.²⁹¹⁰
• For a curated third-party family, the family file is yours to publish (it is links + descriptions); the per-product full text stays private (§1).
• Directories (llmstxt.site, llmstxthub, directory.llmstxt.cloud) are flat lists with categories; none publishes an llms.txt-of-llms.txt, so a family index you author is currently the only nested layer above a product.²⁹³⁹⁴⁰

7. Keep it honest: CI checks, size, drift

• Regenerate in the build; a hand-maintained file drifts and “a stale navigation file is worse than no navigation file, because it actively sends LLMs to dead links” — litellm’s index carried a deleted page in Aug 2026.¹⁸⁴¹
• Check links in CI (llms-txt-validator --check-links JSON; npx llms-txt-check); community validators are stricter than the spec, so read their findings as advice.⁴²⁴³
• Serve with Content-Type: text/plain|text/markdown; charset=utf-8, HTTP 200 (no redirect or auth on the path), UTF-8; add Link: <…/llms.txt>; rel="describedby" and rel="alternate" type="text/markdown" headers (spec v2), Vary: Accept if you negotiate.⁴¹⁹⁴⁴
• Lighthouse’s agentic-browsing audit only fails on a server error, so a missing file is not penalised — do not add one just for the audit.⁴⁵

References