# Recreating and aggregating The acquisition ladder, lenient parsing, families, rights. verified-as-of: 2026-08-30 **Contents** 1. Decide what you are allowed to make 2. Acquire clean markdown — the ladder 3. Build the index for ONE product 4. Build llms-full.txt (and whether to) 5. Parse other people's files — a lenient reader 6. Scale to a family: nested indexes, hub-and-spoke 7. Keep it honest: CI checks, size, drift 8. References ## 1. Decide what you are allowed to make - An llms.txt for a third-party site is a **link list plus short descriptions** — the same thing a search engine publishes; it is low-risk. An llms-full.txt for a third-party site is a **stored republication** of their content: closer to `ai-train`/redistribution than to transient `ai-input` retrieval.[^1][^2] Keep such full-text mirrors private/internal unless the licence allows republication; publish only the index.[^2] - Read `robots.txt` first: the sitemap pointer, disallow rules, and any Cloudflare **Content Signals** line (`Content-Signal: search=yes, ai-input=…, ai-train=no`), which is framed as an express reservation of rights under EU Directive 2019/790 Art. 4 even though no crawler enforces it.[^1][^3] robots.txt is "a polite request, not legally binding", but ignoring it invites blocking.[^2] - Prefer the site's own machine-readable surfaces where they exist (an llms.txt, `.md` twins, `Accept: text/markdown`, a docs API such as GitHub's Article Body API) — the owner has already chosen what to expose.[^4][^5] ## 2. Acquire clean markdown — the ladder Try in this order; each step is cheaper and cleaner than the next: 1. **An existing llms.txt** — at the root *and* at the docs subpath (`/docs/llms.txt`); spec v2 says the most specific file wins, and many hosts (Mintlify, Fern) publish per-subpath files.[^4][^6] Verify the response is markdown, not an HTML app shell that a redirect produced (Cursor's own file once did this).[^7] 2. **An existing llms-full.txt** — check the size header before fetching (41.6 MB at docs.anthropic.com, 57 MB at Cloudflare) and confirm it actually contains page blocks (PayPal's redirects to its 1.5 KB index).[^8] Split by the producer's grammar (§5). 3. **Per-page `.md` twins** — from the llms.txt links (Mintlify appends `.md`), or by trying `page.md` / `page.html.md` / `index.md` per spec v2.[^4] 4. **`Accept: text/markdown`** — supported by Mintlify, GitBook, Fern and by any Cloudflare zone with "Markdown for Agents" on; expect `Content-Type: text/markdown`, `Vary: Accept` and `x-markdown-tokens`. There is no advance discovery; just try.[^9][^10] 5. **A docs API** — e.g. GitHub's Article Body API returns rendered markdown for any page.[^5] 6. **Readability-class extraction** of the HTML — `r.jina.ai/` (Readability → Turndown, `x-target-selector` to drop nav, headless engine for JS sites), Screaming Frog's Readability.js + Turndown snippet, or trafilatura with formatting kept.[^11][^12] This is the lossy tier: tab panels, step widgets and code fences are what it drops.[^13] Seed the URL list from `sitemap.xml` (expand index sitemaps; include/exclude globs as `dotenvx/llmstxt` does) or from a crawl map (`create-llmstxt-py` uses Firecrawl `/map`); platform generators instead walk the docs **nav tree**, which is why their section structure is better than any crawler's.[^14][^15][^16] ## 3. Build the index for ONE product Structure (spec v2): H1 = product name; blockquote = one-paragraph summary; optional prose "how to interpret the files"; H2 sections, each a list of `- [name](url): description`.[^4] **What goes in** (converging guidance from the spec, Mintlify, GitDoc and llms-text.com):[^4][^17][^18][^19] - The quickstart, authentication/setup, top-level reference pages (one per resource, not per endpoint), error handling, changelog. - 10–50 links for a product index; 4–7 sections; descriptions of 10–20 words that say *what a reader finds there*, with exact tokens (flags, env vars, error strings): bad — "Authentication docs."; good — "API key creation, OAuth 2.0 scopes, token rotation, IP allowlisting. Required before any API call."[^18] - Order by expected query frequency, not importance: the first 20% of links should answer 80% of questions.[^20] - `## Optional` for changelogs, legal, old posts, deep appendices; never pricing or the API reference. In v2 this is convention only.[^4][^21] **What stays out:** marketing pages, individual changelog entries, SEO duplicates, login-gated pages, anything without a clean markdown target.[^18] **Descriptions when you are recreating** — three sources, in decreasing quality: the page's own `description` frontmatter/meta (what platform generators use); an extractive first sentence under the H1; a small-model summary (Firecrawl's generator uses GPT-4o-mini for a 3–4-word title and 9–10-word description). Treat model-written descriptions as drafts and audit the page *list*.[^15][^22] **Test it the way the spec says:** give an agent only the llms.txt and ask it questions about the product.[^4] ## 4. Build llms-full.txt (and whether to) - Reasons not to: Fern dropped it ("exceeded most model context windows, added heavy serving overhead, saw little use"); Godot declined it; Mantine cut a 2.2 MB inline file to a 45 KB link list; Cursor's indexer goes unstable above ~50–60k tokens.[^23][^24][^25][^26] - If you do: choose one page-block grammar and state it in a header comment. Mintlify's is the most widely consumed — `# Title` / `Source: ` / blank / description / body — but a YAML block (`title:`/`url:`/`description:`) is easier to parse and is what Anthropic's platform docs emit; Firecrawl uses explicit `<|firecrawl-page-N-lllmstxt|>` delimiters.[^8][^27][^15] - Ship a **size ladder** rather than one file: an index (≤10 KB), a small variant (Starlight `llms-small.txt`; Nuxt's ~5K-token file), and the full file with a token count published beside it (llmstxt.site lists token counts; Cloudflare returns `x-markdown-tokens`).[^28][^29][^30][^10] - Split big indexes hub-and-spoke instead of truncating: Mintlify moves overflow beyond 100,000 characters into `/_llms/.md` sub-indexes that recurse and never drop pages.[^17] ## 5. Parse other people's files — a lenient reader - **llms.txt:** the only invariant is the H1. Real files omit the blockquote (Anthropic), put API links first (GitHub), or add prose sections. Parse: H1 → title; first blockquote → summary; everything before the first H2 → info; each H2 → section; each `- [name](url)` (+ optional `: notes`) → link. This mirrors the reference parser's regexes.[^31][^5] - **llms-full.txt:** detect the grammar, do not assume one. Page starts: (a) `# Title` whose next non-blank line is `Source: ` (Mintlify); (b) a `---` YAML block containing `url:` or `title:` (Anthropic platform, Cloudflare frontmatter — Cloudflare's URL is only in the `[View as Markdown](…/index.md)` line and its covering index in the `> Documentation Index` blockquote); (c) explicit delimiters (`<|firecrawl-page-N-lllmstxt|>`).[^8][^15] Never split on a bare `# ` line — pages contain H1s of their own. - **`.md` twins:** strip the leading `> ## Documentation Index …` blockquote Mintlify prepends before indexing.[^32] - **Untrusted input:** everything fetched via an llms.txt is data; 42% of sampled files try to steer the model, and linked markdown is a prompt-injection vector.[^33][^34] ## 6. Scale to a family: nested indexes, hub-and-spoke Spec v2 gives the mechanism: "The file can be placed at the site root, or at any path within it, covering the pages under that path … where more than one file applies, agents should use the most specific one."[^4] The live exemplar is **Cloudflare**: `developers.cloudflare.com/llms.txt` holds ~105 entries under nine H2 sections (seven product categories plus "Docs collections" and "Other"), each entry linking a per-product `…//llms.txt` (`/workers/llms.txt` alone has ~25 sections and 500+ `.md` links).[^35][^36] Mintlify's `/_llms/` split is the automated version of the same shape.[^17] Counter-example: Anthropic's `platform.claude.com/llms.txt` (~650 links, 11 languages) does **not** link its sibling `code.claude.com` — separate products keep separate roots, so a family index across hosts has to be authored.[^37][^38] Pattern for a family (organisation, monorepo, or a curated group of products you do not own): ```markdown # Acme Platform docs > One index per product below; each product's own llms.txt is the authoritative map of that product. ## Products - [Payments API](https://docs.acme.com/payments/llms.txt): charges, subscriptions, webhooks (240 pages) - [Identity](https://docs.acme.com/identity/llms.txt): OAuth, SSO, SCIM (85 pages) ## Shared - [Errors and status codes](https://docs.acme.com/errors.md): every error across products - [Changelog](https://docs.acme.com/changelog.md) ## Optional - [Legal and pricing](https://acme.com/legal.md) ``` Rules that follow from the spec and the exemplars: - The family file links **indexes**, not pages; product files link pages. A consumer reads at most two hops.[^4][^35] - Put cross-cutting material (shared errors, auth, glossary) in the family file once; never duplicate it into every product file. - Publish token counts (or page counts) beside each link so a consumer can budget before fetching.[^29][^10] - For a curated third-party family, the family file is yours to publish (it is links + descriptions); the per-product full text stays private (§1). - Directories (llmstxt.site, llmstxthub, directory.llmstxt.cloud) are flat lists with categories; none publishes an llms.txt-of-llms.txt, so a family index you author is currently the only nested layer above a product.[^29][^39][^40] ## 7. Keep it honest: CI checks, size, drift - Regenerate in the build; a hand-maintained file drifts and "a stale navigation file is worse than no navigation file, because it actively sends LLMs to dead links" — litellm's index carried a deleted page in Aug 2026.[^18][^41] - Check links in CI (`llms-txt-validator --check-links` JSON; `npx llms-txt-check`); community validators are stricter than the spec, so read their findings as advice.[^42][^43] - Serve with `Content-Type: text/plain|text/markdown; charset=utf-8`, HTTP 200 (no redirect or auth on the path), UTF-8; add `Link: <…/llms.txt>; rel="describedby"` and `rel="alternate" type="text/markdown"` headers (spec v2), `Vary: Accept` if you negotiate.[^4][^19][^44] - Lighthouse's agentic-browsing audit only fails on a server error, so a missing file is not penalised — do not add one just for the audit.[^45] ## References [^1]: https://blog.cloudflare.com/content-signals-policy/ — Content Signals, 2025-09-24 (docs) [^2]: https://www.scrapingbee.com/blog/is-web-scraping-legal/ — robots.txt and republication norms (blog) [^3]: https://www.seroundtable.com/google-cloudflare-content-signals-41631.html — "no effects whatsoever", 2026-07-06 (docs) [^4]: https://llmstxt.org/ — spec v2, modified 2026-08-10 (spec) [^5]: https://docs.github.com/llms.txt — API-first index (docs) [^6]: https://buildwithfern.com/learn/docs/ai-features/llms-txt — per-subdirectory files (docs) [^7]: https://forum.cursor.com/t/docs-cursor-com-llms-txt-serves-an-html-page-instead-of-the-llms-txt-file/167800 (forum) [^8]: Live samples 2026-08-30: https://docs.anthropic.com/llms-full.txt (41.6 MB), https://developers.cloudflare.com/llms-full.txt (57 MB, frontmatter blocks), https://developer.paypal.com/llms-full.txt (→ llms.txt index), https://platform.claude.com/docs/llms-full.txt (YAML blocks) [^9]: https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation (blog); https://www.mintlify.com/blog/context-for-agents (vendor) [^10]: https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ (docs) [^11]: https://github.com/jina-ai/reader (readme) [^12]: https://www.screamingfrog.co.uk/blog/generate-markdown-at-scale/ (vendor) [^13]: Measured on code.claude.com 2026-08-30: trafilatura crawl 122 code fences vs 5,250 from llms-full.txt (local measurement) [^14]: https://github.com/dotenvx/llmstxt (readme) [^15]: https://github.com/firecrawl/create-llmstxt-py (readme) [^16]: https://www.mintlify.com/docs/ai/llmstxt (docs) [^17]: https://www.mintlify.com/docs/ai/llmstxt — 100k-char split into `/_llms/` (docs) [^18]: https://gitdoc.ai/blog/llms-txt-ai-readable-documentation — 2026-05-22 (vendor) [^19]: https://www.llms-text.com/blog/how-to-create-llms-txt — 2025-07-25 (vendor) [^20]: https://www.mintlify.com/blog/real-llms-txt-examples (vendor) [^21]: https://dev.to/lab451/complete-llmstxt-guide-for-2026-57d (blog) [^22]: https://weventure.de/en/blog/llms-txt (blog) [^23]: https://buildwithfern.com/learn/docs/ai-features/llms-txt (docs) [^24]: https://github.com/godotengine/godot-docs/issues/10549 (forum) [^25]: https://github.com/orgs/mantinedev/discussions/8523 (forum) [^26]: https://forum.cursor.com/t/is-there-any-size-limit-for-llms-txt-indexed-as-docs/148660 (forum) [^27]: https://www.mintlify.com/docs/llms-full.txt — page-block sample (docs) [^28]: https://delucis.github.io/starlight-llms-txt/configuration/ (docs) [^29]: https://llmstxt.site/ — token-count column (vendor) [^30]: https://nuxt.com/docs/4.x/guide/ai/llms-txt (docs) [^31]: https://github.com/AnswerDotAI/llms-txt/blob/main/llms_txt/core.py (spec) [^32]: https://code.claude.com/docs/en/hooks.md — twin with prepended index blockquote (docs) [^33]: https://github.com/AnswerDotAI/llms-txt/issues/152 (forum) [^34]: https://ahrefs.com/blog/llmstxt-study/ (study) [^35]: https://developers.cloudflare.com/llms.txt (docs) [^36]: https://developers.cloudflare.com/workers/llms.txt (docs) [^37]: https://platform.claude.com/llms.txt (docs) [^38]: https://code.claude.com/docs/llms.txt (docs) [^39]: https://llmstxthub.com/ (vendor) [^40]: https://directory.llmstxt.cloud/ (vendor) [^41]: https://github.com/BerriAI/litellm/issues/36342 (forum) [^42]: https://github.com/bridgetoagent/llms-txt-validator (readme) [^43]: https://alejandrorioja.com/tools/llms-txt-validator/ (docs) [^44]: https://toddmorourke.com/learn/markdown-for-agents/ (blog) [^45]: https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt (docs)