Generation tooling

verified-as-of: 2026-08-30 (tool versions, maintenance status and platform features are volatile — re-verify before recommending)

Contents

  1. Pick by situation
  2. Docs platforms (built-in)
  3. Static-site-generator plugins
  4. Crawl-based generators (sites you do not own)
  5. CMS and site builders
  6. Edge content negotiation
  7. Quality practices that generators get wrong
  8. References

1. Pick by situation

You have…UseEmits
Docs on Mintlify / GitBook / ReadMe / Fernnothing — it is automaticllms.txt (+ full on Mintlify/GitBook) + .md twins
Docusaurus, MkDocs, VitePress, Starlight, Sphinx, Nuxtthe framework plugin (table §3)llms.txt + llms-full.txt (+ .md, llms-small.txt on Starlight)
A live site you do not owncrawl-based generator (§4) — create-llmstxt-py, dotenvx/llmstxt, or your own sitemap→markdown pipelinellms.txt (+ full) with extracted or AI-written descriptions
WordPressYoast ≥25.3 / Rank Math / AIOSEO (§5)llms.txt only (AIOSEO Pro adds full + markdown posts)
Webflow / Framerhost a hand-written filewhatever you upload
Any Cloudflare-proxied HTML site“Markdown for Agents” toggle (§6)on-the-fly markdown on Accept: text/markdown, no llms.txt

2. Docs platforms

PlatformEmitsDescriptions fromNotes
Mintlifyllms.txt, llms-full.txt, .md per page, /.well-known/ copies, /_llms/ split indexesfrontmatter description (truncated at 300 chars), nav order from docs.json; optional markdown.instructions agent textindex capped at 100,000 chars → recursive /_llms/<group>.md; default language/version only; hidden/noindex pages excluded; hand-written root files override; auth sites list public pages or require auth1
Fernllms.txt (root and per-subdirectory), .md per page; no llms-full.txtfrontmatter description, fallback subtitle; adds OpenAPI/AsyncAPI linksdropped llms-full because it “exceeded most model context windows, added heavy serving overhead, saw little use”2
GitBookllms.txt (Jan 2025), llms-full.txt + .md per page (Jun 2025), /sitemap.md, Accept: text/markdownauto from page structurezero-config; no curation controls documented; full export “will be more expensive”34
ReadMellms.txt (default on, all plans), .md per page; no llms-fullproject title + guide/API hierarchya custom file from the repo root disables auto-updates; hidden pages excluded5
GitDoc (vendor claim)llms.txt + llms-full.txt “for the pages you mark as priority”, regenerated in the buildsidebar/navvendor blog, 2026-05-226

3. Static-site-generator plugins

PluginEmitsInputDescriptions / orderingMaturity & limits
docusaurus-plugin-llms (rachfop)llms.txt, llms-full.txt, optional per-page .md, versioned + customLLMFilessource tree at postBuildfrontmatter → first heading → site fallback; includeOrder globs144★, MIT; not run in docusaurus start; image rewrite only for bundled assets7
@signalwire/docusaurus-plugin-llms-txtllms.txt, .md, optional fullbuilt HTML (rehype/remark)manual sections[].description, autoSectionDepthv1.2.2, ~10 months stale; ENOENT / “processed 0 documents” bug89
Docusaurus corenone——issue #10899 open since Feb 202510
mkdocs-llmstxt (pawamoy)llms.txt, .md, optional full_outputbuilt HTML → BeautifulSoup → Markdownifysections: dict with per-file descriptions130★, v0.5.x, maintenance mode, seeking maintainer; needs site_url; mkdocstrings show_source mangles tables/code in the full file1112
vitepress-plugin-llms (okineadev)llms.txt, llms-full.txt, .mdVitePress sourcefrontmatter description; <llm-only> / <llm-exclude> tags394★; used by Vite, Vue, Vitest, Rolldown; relative URLs break under redirects/domain moves13
starlight-llms-txt (delucis)llms.txt, llms-full.txt, llms-small.txtAstro StarlightprojectName, description, details, optionalLinks, customSets, promote/demote; minify strips asides110★, docs updated Aug 2026; needs site14
sphinx-llms-txt (jdillard)llms.txt (markdown), llms-full.txt (reStructuredText)Sphinx buildtoctree titles; llms_txt_summary, llms_txt_exclude, llms_txt_full_max_sizev0.7.1; full file is RST; points to NVIDIA sphinx-llm15
nuxt-llms / Nuxt Contentllms.txt (~5K tokens), opt-in llms-full.txt (~1M+ tokens)Nuxt Content, runtime hookssections in nuxt.configfirst-party; full file explicitly for 200K+-context tools16
Next.js / Nextrahand-rolled app/llms.txt/route.ts (force-static or dynamic); next-llms-txt adds per-page .md endpointscomponents“reads and parses readable text”discussion #80692 unresolved; no Nextra built-in found (tentative)1718
llms-txt-action (demodrive-ai)llms.txt, llms-full.txt, .mdbuilt HTML dir + sitemap.xmllocal/offline or cloud LLM summaries via LiteLLM (default GPT-4o)16★; needs --dirty with mkdocs gh-deploy19

4. Crawl-based generators (sites you do not own)

ToolWhat it doesLimits
Firecrawl /llmstxt API + llmstxt.firecrawl.devURL → async job → llms.txt (+ full); maxUrls 1–100 (default 10), 1 credit/URL, public pages only, 5,000-URL alpha capdeprecated in favour of the main endpoints (page carries no date; still up); users pointed to the Python repo2021
create-llmstxt-py (Firecrawl, 320★)/map → scrape each page to markdown (batches of 10; failures skipped, no retry) → GPT-4o-mini writes a 3–4-word title + 9–10-word description → flat llms.txt; llms-full.txt concatenates under <|firecrawl-page-N-lllmstxt|>default 20 URLs; memory issues on large sites; sections are not inferred; descriptions are AI-written and unreviewed22
dotenvx/llmstxt (147★, BSD-3)sitemap.xml → - [Title](url): description bullets; --include-path / --exclude-path globs; --replace-title regexllms.txt only; titles extracted from HTML; description derivation undocumented23
Jina Reader r.jina.ai/<url>headless Chrome or curl engine → Readability → Turndown; headers x-respond-with, x-target-selector, x-retain-links, x-max-tokens, x-markdown-chunkingper-page cleaner, no site/llms.txt mode; anonymous traffic rate-limited24
Screaming Frog v24.3per-page .md via a Readability.js + Turndown custom-JS snippet; llms.txt via n8n/CSV convertersno native llms.txt export; thin pages return nothing; JS rendering slow2526
plainsignal/llmstxt Chrome extensionllms.txt + one .md per page + zip from sitemap or rendered DOM; meta description as blockquote10★, HTTPS only27
SEO-tool generators (SEOmator etc.)robots.txt → sitemap discovery, index-sitemap expansion, LLM-written title+description per URLvendor-claimed mechanics only28
llms-text.com generator/validatorcrawls a domain and exports llms.txt + llms-full.txt (“deep-crawls up to 50 subpages”); validator checks syntax, links, UTF-8, headersvendor; its guidance: 10–20 evergreen URLs, 4–7 H2s, 10–20-word descriptions, index under 10 KB, `Content-Type: text/plain

5. CMS and site builders

PlatformEmitsDescriptionsLimits
Yoast SEO ≥25.3 (2025-06-10)llms.txt only, regenerated weeklycustom excerpt only — no description otherwise; 5 latest posts/pages/CPT (≤12 months, cornerstone first) + top-5 taxonomies5-item cap; markdown chars escaped; a static file wins over the dynamic one2930
Rank Mathllms.txt only“intro text”; post types/taxonomies, limit default 100; custom linesno full31
AIOSEOllms.txt (free); llms-full.txt + markdown post conversion (Pro)site title/tagline; per-post-type limits, exclusionspaywall32
website-llms-txt, llms-full-txt-generatorllms.txt (+ full)titles + SEO-plugin descriptions; honour noindexone shipped a broken-access-control CVE fix33
Joost de Valk “Markdown Alternate”<link rel="alternate" type="text/markdown"> + .md URLs per post—negotiation, not an index34
Webflow / Framerhost an uploaded file (Framer: Pro/Enterprise “Hosting → Files”); a Framer marketplace plugin scans the CMSmanualno generation3536
Shopify (Apr–May 2026, silent)auto /llms.txt, /agents.md, /sitemap_agentic_discovery.xml, /.well-known/ucp on every storeboilerplate: H1 store name, /collections/all, contact, UCP + MCP endpointstemplates/llms.txt.liquid replaces, does not merge; no changelog; 78.1% of top-10k Shopify hosts vs WordPress 8.7%373839

6. Edge content negotiation

Cloudflare “Markdown for Agents” (2026-02-12; Pro/Business/Enterprise; zone toggle under AI Crawl Control): on Accept: text/markdown the edge converts HTML → markdown (body + meta-derived YAML frontmatter + JSON-LD, nav/header/footer/scripts dropped) and returns Content-Type: text/markdown; charset=utf-8, x-markdown-tokens, x-original-tokens, Vary: Accept; ETag/Last-Modified/Content-Encoding stripped; origin HTML ≤ 2 MB (raised from 1 MB); a chunked-encoding silent pass-through was fixed Jul 2026.404142 It produces no llms.txt — pair it with a hand-written index. Checkly measured a 99.7% token reduction on its own docs (single site).43

7. Quality practices that generators get wrong

References

Footnotes

  1. https://www.mintlify.com/docs/ai/llmstxt (docs) ↩ ↩2 ↩3 ↩4 ↩5

  2. https://buildwithfern.com/learn/docs/ai-features/llms-txt (docs) ↩ ↩2

  3. https://gitbook.com/docs/ai-and-search/llm-ready-docs (docs) ↩

  4. https://gitbook.com/docs/changelog/june-2025/24-june-performance-upgrades-llms-full.txt-and-.md-support-text-alignment-and-more (docs) ↩

  5. https://docs.readme.com/main/docs/LLMstxt (docs) ↩ ↩2

  6. https://gitdoc.ai/blog/llms-txt-ai-readable-documentation — 2026-05-22 (vendor) ↩

  7. https://github.com/rachfop/docusaurus-plugin-llms (readme) ↩ ↩2

  8. https://github.com/signalwire/docusaurus-plugins/tree/main/packages/docusaurus-plugin-llms-txt (readme) ↩

  9. https://github.com/signalwire/docusaurus-plugins/issues/5 (forum) ↩

  10. https://github.com/facebook/docusaurus/issues/10899 (forum) ↩

  11. https://github.com/pawamoy/mkdocs-llmstxt (readme) ↩ ↩2

  12. https://github.com/mkdocstrings/python/issues/299 (forum) ↩

  13. https://github.com/okineadev/vitepress-plugin-llms (readme) ↩

  14. https://delucis.github.io/starlight-llms-txt/configuration/ (docs) ↩ ↩2 ↩3

  15. https://sphinx-llms-txt.readthedocs.io/en/latest/ (docs) ↩ ↩2

  16. https://nuxt.com/docs/4.x/guide/ai/llms-txt (docs) ↩

  17. https://next-llms-txt.vercel.app/ (docs) ↩

  18. https://github.com/vercel/next.js/discussions/80692 (forum) ↩

  19. https://github.com/demodrive-ai/llms-txt-action (readme) ↩

  20. https://docs.firecrawl.dev/features/alpha/llmstxt (docs) ↩

  21. https://github.com/firecrawl/llmstxt-generator (readme) ↩

  22. https://github.com/firecrawl/create-llmstxt-py (readme) ↩ ↩2

  23. https://github.com/dotenvx/llmstxt (readme) ↩

  24. https://github.com/jina-ai/reader (readme) ↩

  25. https://www.screamingfrog.co.uk/blog/generate-markdown-at-scale/ (vendor) ↩

  26. https://n8n.io/workflows/3219-generate-ai-ready-llmstxt-files-from-screaming-frog-website-crawls/ (vendor) ↩

  27. https://github.com/plainsignal/llmstxt (readme) ↩

  28. https://seomator.com/free-llms-txt-generator (vendor) ↩

  29. https://developer.yoast.com/features/llms-txt/functional-specification/ (docs) ↩ ↩2 ↩3

  30. https://developer.yoast.com/changelog/yoast-seo/25.3/ (docs) ↩

  31. https://rankmath.com/kb/llms-txt/ (docs) ↩

  32. https://aioseo.com/docs/how-to-create-an-llms-txt-using-all-in-one-seo/ (docs) ↩

  33. https://wordpress.org/plugins/website-llms-txt/ (vendor) ↩

  34. https://joost.blog/markdown-alternate/ (blog) ↩

  35. https://university.webflow.com/videos/optimize-your-site-for-llms-with-llms-txt (docs) ↩

  36. https://www.framer.com/help/articles/llms-txt-framer/ (docs) ↩

  37. https://honeybound.co/blog/shopify-llms-txt-agents-md (blog) ↩

  38. https://imakemvps.com/blog/llms-txt-generator-for-shopify (blog) ↩

  39. https://caseyrb.com/blog/state-of-llms-txt-adoption/ — HTTP Archive, Jun 2026 (study) ↩

  40. https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ (docs) ↩

  41. https://developers.cloudflare.com/changelog/post/2026-02-12-markdown-for-agents/ (docs) ↩

  42. https://community.cloudflare.com/t/cloudflare-fundamentals-content-encoding-support-for-markdown-for-agents-and-other-improvements/893536 (forum) ↩

  43. https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/ (blog) ↩

  44. https://weventure.de/en/blog/llms-txt (blog) ↩

  45. https://llmstxt.org/ (spec) ↩

  46. https://www.mintlify.com/blog/real-llms-txt-examples (vendor) ↩

  47. https://dev.to/lab451/complete-llmstxt-guide-for-2026-57d (blog) ↩

  48. https://www.llms-text.com/blog/llms-txt — 2025-07-25 (vendor) ↩

  49. https://github.com/BerriAI/litellm/issues/36342 (forum) ↩

  50. https://github.com/bridgetoagent/llms-txt-validator (readme) ↩

  51. https://toddmorourke.com/learn/markdown-for-agents/ (blog) ↩

  52. https://ahrefs.com/blog/llmstxt-study/ (study) ↩

  53. https://www.searchenginejournal.com/google-says-llms-txt-is-purely-speculative-for-now/577576/ (blog) ↩