NLWeb and MCP as agentic-discovery alternatives to llms.txt — researched

NLWeb is R.V. Guha's schema.org/RSS query layer, launched by Microsoft in 2025-05 and transferred to the independent nlweb-ai org in 2025-07-30; every instance also exposes a version-pinned (2024-11-05), stateless, undiscoverable MCP /mcp route whose auth fails open, and the repo's main branch has been feature-dormant since 2026-06-10. Sourced from one reference document (14 footnoted sources, 90 scanned units, 74 kept) that frames NLWeb, MCP, and WebMCP not as competitors to a static llms.txt but as complementary layers, and closes with a default-plus-triggers decision rule: publish llms.txt always, and add a live endpoint only against six named triggers a file structurally cannot satisfy (auth/entitlements, actions, an unenumerable corpus, minute-scale volatility, metering, or wanting query-intent signal).

Definitions

**Auth is a presence check that fails open.** `/ask` is in `PUBLIC_ENDPOINTS`; `/mcp` is not — but the JWT branch ends with a comment that on `InvalidTokenError` it allows the token through "for backward compatibility", so any non-empty `Bearer` string passes, and development mode bypasses the check entirely.[^mcp-9] [source]
Note `site` is an **array** over MCP but a string on `/ask` — the two bindings are not argument-identical despite the doc's parity claim.[^mcp-5][^mcp-3] [source]
**But it is not a clean spin-out.** On current `main`, the README still lists `[email protected]` for support, retains a Microsoft Trademarks section, and ships `SECURITY.md` as verbatim Microsoft MSRC boilerplate routing vulnerability reports to MSRC; `RAI_TRANSPARENCY.md` is a Microsoft Responsible-AI artifact; and a Microsoft engineer is the #3 contributor.[^ad-4][^ad-3] Meanwhile the `nlweb-ai` org is unverified with no company, email, or location set, and **no legal entity named "NLWeb AI" surfaced in any search**.[^ad-1] The honest characterization: *a Guha-led project in a project- [source]

Structure and components

| Mechanism | Layer | Backer | Standards status (precise) | Adoption (Sept 2026) | |---|---|---|---|---| | `robots.txt` | Access control | IETF | **Ratified** — RFC 9309 | Universal | | schema.org JSON-LD | Content description | schema.org / Google et al. | De-facto standard, community vocabulary | Very wide; rewarded by Google rich results | | RSS / Atom | Content description | IETF (Atom RFC 4287) | Ratified / de-facto | Wide, mature | | `llms.txt` | Content description (curated index) | Jeremy Howard / Answer.AI | **Community proposal — no standards track** | ~5–10% of the general web; **97 [source]
- `llms.txt` **describes content**; MCP **transports tool calls**; WebMCP **exposes in-page actions**; A2A is **agent-to-agent entirely** and is routinely conflated into this debate when it belongs to a different conversation. - **NLWeb is not a publishing format at all.** It ingests schema.org and RSS — formats you already publish — and adds a *query interface* over them. It therefore does not compete with llms.txt for the "what should I put in my site root" slot; it competes for the "should I run a service" slot. - Microsoft's own agent-ready guidance stacks robots.txt + sitemaps + schema.or [source]
1. **"Competing" is mostly wrong.** These sit at different layers, the same vendors ship both, and Microsoft's own agent-ready guidance stacks robots.txt + sitemaps + schema.org + llms.txt + MCP + NLWeb as complementary. 2. **NLWeb is no longer a Microsoft repo.** `github.com/microsoft/NLWeb` 301-redirects to `nlweb-ai/NLWeb`; the transfer was ~2025-07-30 and was never widely announced. Wikipedia and most secondary write-ups are still wrong about this. 3. **"Every NLWeb instance is an MCP server" is weaker than it sounds** — a version-pinned, stateless, undiscoverable JSON-RPC route whose auth [source]
**Module layout** (after a 2026-03 restructure): `AskAgent` (query core), `AgentFinder` (discovery/routing), `DataFinder` (NL→SQL over HubSpot/Dynamics/Jira via schema.org ontology mappings), `ModelRouter` (cost-aware model selection), `NLWebScorer` (neural ranking).[^nw-3] [source] — module/component layout list
> **The framing correction.** "Static llms.txt *or* a live endpoint" is the wrong question in almost every real case. They sit > at different layers, the same vendors ship both, and Microsoft's own agent-ready guidance stacks robots.txt + sitemaps + > schema.org + llms.txt + MCP + NLWeb as complementary layers.[^df-17] The real scarce resource is engineering attention, and > the evidence says spend it on the file and the structured data first, and on an endpoint only against a named trigger. [source]
**What it does.** NLWeb turns a site's *existing* structured data into a natural-language query endpoint. It is not a new publishing format — it ingests **schema.org JSON-LD markup and RSS/Atom feeds** that sites already emit, on the premise that this markup is a de-facto semantic layer and that LLMs already parse it well.[^nw-4][^nw-5] Content is embedded into a vector store, and queries are answered by a server-side pipeline. [source]
These mechanisms are commonly discussed as rivals. Most of them are not: they occupy **different layers**, and several are designed to compose. Sorting them by layer is the fastest way to stop mis-scoping a decision. [source]
**Tool surface: three tools.** `handle_tools_list` returns exactly `ask`, `list_sites`, and — only when the who-endpoint is enabled — `who`.[^mcp-3] `ask_nlw`, which appears in some write-ups, is a *server nickname* in a `claude_desktop_config.json` `mcpServers` key, not a tool name.[^mcp-6] The `ask` input schema: [source] — lists the three MCP tool names

How it works

If a trigger fires, prefer a **managed path over self-hosting NLWeb's reference implementation**. Cloudflare's AutoRAG ships NLWeb as a one-click quick-deploy that crawls and indexes your domain and serves both `/ask` and `/mcp`, up to 100k pages.[^df-6] Self-hosting means Python 3.10+, three YAML configs, an LLM key, and a vector DB you keep reindexed,[^df-14] against a codebase whose `main` branch has had no feature commits since 2026-06-10, has never cut a release, and whose own `mcp_wrapper.py` warns "Backwards compatibility is not guaranteed at this time."[^df-15] [source]
**The claim, and what it actually means.** NLWeb's README and Microsoft's launch material both assert that every NLWeb instance is also an MCP server.[^mcp-1][^mcp-2] At the protocol level this is one aiohttp application with a second HTTP route: `setup_mcp_routes()` registers `/mcp` on the same router as `/ask`, and the MCP `tools/call` handler builds the *same* `NLWebHandler` object that `/ask` uses.[^mcp-3][^mcp-4] The REST API doc states it plainly: "NLWeb supports 2 APIs at the endpoints /ask and /mcp. The arguments are the same for both, as is most of the functionality."[^mcp-5] There is [source]

How-to and procedures

1. Ship `llms.txt` (linked, size-disciplined, CI-regenerated). Non-negotiable. 2. Emit/repair **schema.org JSON-LD and RSS**. These are the substrate NLWeb consumes and they pay off independently — they are the one part of this stack with a proven incentive behind it (Google rich results).[^df-17] 3. Walk the trigger table. **No trigger → stop.** You are done, and you have spent almost nothing. 4. One or more triggers → decide *what* you're exposing: - **Answers over your content** → NLWeb, via a managed host if available. - **Actions/tools** → your own MCP server on the current spec revision. [source] — numbered decision-procedure steps
1. **Link, don't inline, and keep it small.** The same benchmark's fourth arm — markdown with llms.txt *inlined* — "cost more tokens every time" than linking it.[^df-1] And unbounded `llms-full.txt` forfeits the advantage: Cloudflare's is ~3.7M tokens and Anthropic's 481k,[^df-2] against a mainstream client that becomes unstable indexing above ~50–60k.[^df-3] 2. **Regenerate in CI.** Staleness is the static path's defining failure and it is silent, because downstream clients cache your file for days.[^df-4] [source] — numbered discipline steps (link don't inline; regenerate in CI)

Measurements and reference values

Two numbers set the budget. A modeled 100k-document RAG pipeline lands at roughly **$0.0013–$0.0033 per query all-in** (~$98/month at 1k queries/day, ~$4,000/month at 100k), with reranking the dominant line item at scale.[^df-10] And NLWeb specifically reports its pipeline "might involve **over 50 LLM API calls**" for a single query.[^df-11] Multiply by expected agent traffic — which is not hypothetical: agents are 66% of traffic across Mintlify's docs network, and automated requests are 57.5% of all HTML web traffic, arriving in bursts 10–20× normal.[^df-12] **The publisher pays per query; th [source]
The pattern is consistent: **structured-web standards stick when a consumer with market power rewards publishing.** Guha built three of the entries in that table. For NLWeb, the powerful consumer does not yet exist — no major agent vendor grants ranking, traffic, or payment for running an NLWeb endpoint. That, not the technology, is the binding constraint, and it applies to llms.txt equally (97% of published files got zero requests[^ad-15]). MCP is the exception that proves the rule: its consumers (Claude, ChatGPT, Copilot) *do* call MCP servers, which is why MCP adoption outran both. [source]
**Static files get large fast.** Measured: Anthropic's `llms.txt` is 8,364 tokens but its `llms-full.txt` is **481,349 tokens**; Cloudflare's is **~3.7M tokens**; NVIDIA's main-site file is 252,607 tokens (its technical-docs file only 1,259).[^sd-8] Cursor users report `@Docs` indexing becoming unstable above roughly 50–60k tokens[^sd-15] — Cloudflare's file is ~60× that. And fitting is not the same as working: Chroma's Context Rot study across 18 frontier models finds reliability declines with input length even on simple retrieval, non-uniformly and sensitive to distractors.[^sd-16] [source]
**But the measured benefit of the cheap option is navigation, not accuracy.** Mintlify's benchmark (2,400 runs, 20 docs sites, 5 questions each, 3 repetitions) found 404s per task fell from 2.23 (HTML) to 1.42 (plain markdown) to **0.11** with an llms.txt pointer — ~90% fewer dead-URL fetches, replicated across four models. **Accuracy stayed in the mid-to-high 90s across every format.**[^sd-17] That null result is the single most important datapoint in this comparison: the static file removes wasted fetches, it does not make answers more correct. Vendor-run and not independently replicated — t [source]
**Demand for either surface is weak.** In a 137,210-domain study, 28% published an llms.txt and **97% of those files received zero requests** in the measurement month.[^sd-13] If agents are not fetching a free file, the prior that they will discover and authenticate against a bespoke endpoint needs its own evidence. [source]
**The project's own domain is broken.** `https://nlweb.ai` fails TLS verification with **"certificate has expired"** (independently reproduced twice, 2026-09-02), and the README's link to `nlweb.ai/spec` 404s (the live spec is at `/docs/intro`).[^ad-6] For a project whose entire pitch is "run this endpoint on your website," an expired certificate on its own apex domain is a meaningful signal about maintenance capacity. [source]

Problems, failure modes and limitations

Evidence against a tidy convergence story: three mutually incompatible proposed well-known URIs for MCP discovery with zero IANA registrations;[^lm-3] a WebMCP API that renamed its root object twice in under a year;[^lm-9] NLWeb pinned to an MCP revision five releases stale while the spec underwent its largest-ever rewrite;[^lm-2] and NLWeb's own spec and reference implementation exposing *different* endpoint sets. Governance is consolidating (MCP, AGENTS.md and goose all under the Linux Foundation's Agentic AI Foundation since Dec 2025; A2A under the LF since Jun 2025)[^lm-7] — but consolidat [source] — corroborates the same spec-vs-implementation divergence as s01u000042
1. **MCP went stateless.** The current MCP revision `2026-07-28` — the largest since launch — removes the `initialize`/`initialized` handshake and `Mcp-Session-Id`, making the protocol stateless by default so it scales on ordinary HTTP infrastructure behind load balancers.[^sd-20][^sd-21] Version negotiation moved into a per-request `_meta` key plus the `MCP-Protocol-Version` header, with a **mandatory `server/discover` RPC** returning supported versions, capabilities and identity in one call.[^sd-21] This weakens the old "a static site can't run a server" objection for the *discovery* half. ( [source]
| Trigger | Why a file cannot do it | |---|---| | **Auth / entitlements** — content differs per user, or is private | A file is anonymous and identical for everyone; MCP specifies OAuth 2.1 + PKCE + RFC 9728[^df-5] | | **Actions / transactions** — booking, ordering, mutating state | A file is read-only by construction | | **Corpus too large to enumerate** — inventory, listings, catalogues | You cannot flatten a million SKUs into a context window; this is precisely NLWeb's `schema.org`-list sweet spot[^df-6] | | **Volatility measured in minutes** — pricing, availability, live status | Any file, [source]
**The security problem is structural, not incidental.** WebMCP lets untrusted page content define agent-callable actions. A 2026 arXiv paper, *WebMCP Tool Surface Poisoning* (Lee, Chang, Yu, Yeh; submitted 2026-06-04), names the class **Mid-Session Tool Injection (MSTI)** — malicious tools injected during an active session via third-party scripts — and splits it into **Tool Hijacking** (changing the visible tool set via the `AbortSignal` API or registration race conditions) and **Tool Framing** (steering agent perception through `name`, `description`, `readOnlyHint`, `inputSchema`). Verbatim c [source]
**Cost and exposure.** Every anonymous `tools/call` puts vector retrieval plus at least one LLM call on the hot path (the API doc notes each result's `description` is "generated by an llm", and `generate` mode is full RAG).[^mcp-5] The only guard in the MCP path is a 30-second `asyncio.wait_for` timeout — there is no rate limiting, per-caller quota, circuit breaker, or cost accounting.[^mcp-3][^mcp-4] This matters because MCP has no protocol-level rate-limit primitives at all (no `Retry-After` convention, no standard `429` semantics).[^mcp-11] A Censys scan found 12,520 internet-facing MCP ser [source]
| Property | Verdict | |---|---| | Server-side retrieval + ranking | **Yes** — the real value; agent sends a string, site returns scored top matches[^mcp-5] | | Typed argument contract (`generate_mode`, `site[]`) | **Yes** — enumerated in JSON Schema[^mcp-3] | | Session / statefulness | **No** — "there is no server side state… the context of the conversation thus far has to be passed back as part of the request"[^mcp-5] | | Pagination | **No** — no cursor/offset/limit in the schema[^mcp-3] | | Version negotiation | **No** — returns its pinned version regardless of request[^mcp-3] | | Structure [source]
A live endpoint (NLWeb, or your own MCP server) earns its cost when the requirement is something a file **structurally cannot do** — not when it would merely be nicer: [source]
"Competing" is mostly the wrong frame — they sit at different points in the interaction lifecycle and the same vendors ship both. Where they *genuinely* compete is narrow: **when a site has already decided to invest in one agent-facing surface and must choose where the marginal engineering hour goes.** There, the evidence favours the static file first (measured navigation win, near-zero cost, no new attack surface) and the endpoint only when a requirement appears that a file structurally cannot meet — auth, entitlements, personalization, freshness guarantees, actions, or metering. [source]

Comparisons and alternatives

Every substantive comparison source found concludes the two are **complementary, not competing**.[^sd-2][^sd-3][^sd-4] That is structurally corroborated: every NLWeb instance is also an MCP server,[^sd-1] and platforms like Mintlify auto-generate llms.txt *and* run an MCP server over the same docs.[^sd-5][^sd-6] Microsoft's own "agent-ready" guidance stacks robots.txt + sitemaps + schema.org + llms.txt + MCP + NLWeb as layers of one strategy rather than alternatives.[^sd-7] [source]
| Axis | Static index (llms.txt / llms-full.txt) | Dynamic endpoint (NLWeb / site MCP server) | |---|---|---| | Who works | Agent pulls, selects, reasons — client-side | Site retrieves, ranks, synthesizes — server-side[^sd-1] | | Context economics | Whole cost lands on the client; real files run 250k–3.7M tokens[^sd-8] | Client pays only for the answer; site pays per query[^sd-9] | | Freshness | Stale by construction; needs CI regeneration; client caches extend staleness[^sd-10] | Current at query time; failure shifts to index drift and downtime | | State / auth / actions | Anonymous, identica [source] — static-vs-dynamic axis table, was heuristically bucketed as problems
And if the goal is simply *"be callable by agents"* rather than *"answer natural-language questions about my content"*, a plain MCP server on the **current** spec revision (`2026-07-28` — stateless, load-balancer-friendly, with a mandatory `server/discover` RPC[^df-16]) is the better-supported choice than NLWeb's binding, which is pinned to `2024-11-05`.[^df-15] [source]
WebMCP is **not** Anthropic's MCP. MCP is a server-side protocol connecting a model to remote tools. WebMCP is a **browser API**: a page registers typed JavaScript tools that an agent running *in the user's browser* can call, instead of scraping the DOM or driving the UI with vision. [source]
**The short answer.** Publish the static `llms.txt` — always, first, linked rather than inlined, size-disciplined and CI-regenerated. Add a live endpoint (NLWeb or your own MCP server) **only** against a named trigger that a file structurally cannot satisfy: auth/entitlements, actions, a corpus too large to enumerate, minute-scale volatility, metering, or wanting query intent. Jump to [the decision framework](#concept-the-decision--when-a-static-file-suffices-vs-when-a-live-endpoint-earns-its-keep-complete) if that is all you need. [source]
- **Security, and the disclosure handling.** An unauthenticated **path-traversal** vulnerability (misuse of `os.path.normpath`) allowed remote reads of `/etc/passwd` and `.env` files containing cloud credentials. Reported 2025-05-28, patched 2025-06-30, disclosed 2025-08-06/07 — and **MSRC declined to issue a CVE**, criticized as a transparency failure.[^ad-10][^ad-11] - **No native payments.** Ben Thompson argues NLWeb has no payment layer, so it worsens rather than resolves the agentic web's monetization "original sin" — sites do work for agents and receive nothing.[^ad-12] - **Cost asymmetr [source]
The `schema_object` design is genuinely strong: answers are assembled from datastore items, so the *items* cannot be fabricated,[^ad-13] which is a real correctness property that no static-file approach provides. Reusing schema.org and RSS rather than inventing a format means the publishing cost is near-zero for sites that already emit markup. Cloudflare's one-click AutoRAG path removes almost all operational burden.[^ad-7] If a large agent vendor ever created a *demand* signal, the supply side is cheap to switch on. [source]
**Source-quality note.** This topic was expected to be commentary-heavy, and for the *comparative* claims it is: the static-vs-dynamic trade-off rests largely on vendor blogs, practitioner posts and one self-labelled cost model, all tagged as opinion or estimate below. The *factual* spine, however, turned out to be strongly primary — GitHub's API, NLWeb's own source files and docs, the W3C WebMCP draft, the MCP specification, and an arXiv security paper. Where a claim is single-source, vendor-interested, or an estimate rather than a measurement, its footnote says so; weigh those differently fr [source]
**Response shape — the genuinely good idea.** Results are returned as schema.org-typed JSON: each carries `url`, `name`, `site`, `score`, an LLM-generated `description`, and **`schema_object`** (the source item encoded as JSON).[^nw-8] Because items come from the datastore rather than from generation, the docs note **a result "will not be 'made up'"** — the retrieval layer structurally bounds hallucination for the *items*, though the generated descriptions are still model output.[^nw-8] [source]
**Agent traffic is now the dominant load, which cuts both ways.** Mintlify measures agents at 66% of traffic across its docs network (213M agent requests vs 105M human page loads in one July month, up from 15.2% at the start of 2026);[^sd-5] Cloudflare's CEO put automated requests at 57.5% of all HTML web traffic (2026-06-03).[^sd-14] Some open-source projects report up to 97% bot traffic and real outages from crawler load.[^sd-19] A CDN-cached file absorbs that; a per-query inference endpoint bills for it. [source]
**The freshness admission.** The README advises production deployments to "connect NLWeb to live databases instead of duplicating content (to avoid freshness issues)" — an explicit concession that the default ingest-and-embed path goes stale and needs reindexing.[^nw-4] This matters: freshness is often cited as the dynamic architecture's advantage over a static file, but NLWeb's default configuration has the same staleness problem, just moved into a vector index. [source]

Changes and history

| Prior standard | Outcome | What decided it | |---|---|---| | RSS | Survived, niche | Useful without a gatekeeper, but no monetization | | RDF / Semantic Web | Failed at web scale | High authoring cost, no consumer paying for it | | schema.org | **Succeeded** | Google rewarded it with rich results — a powerful consumer created the incentive | | Open Graph | **Succeeded** | Facebook rewarded it with link previews — same mechanism | | AMP | Adopted then abandoned | Incentive was coercive; withdrawn when Google stopped ranking on it | [source] — table of prior web-standard outcomes used as analogy

Facts and statements

[^df-1]: https://www.mintlify.com/blog/llms-txt-agent-benchmark — measurement (vendor), 2,400 runs / 20 sites / 5 questions / 3 reps. 404s per task 2.23 → 1.42 → 0.11; verbatim: "Accuracy stayed in the mid-to-high 90s across every format, because agents are good at eventually finding the right page no matter how you serve it."; inlining "cost more tokens every time" than linking. Re-verified 2026-09-02. No independent replication found — treat directionally. [^df-2]: https://getpublii.com/blog/llms-txt-complete-guide.html — measurement, 2026-01-10. Cloudflare ~3.7M tokens; Anthropic llms-full. [source] — footnote block
[^lm-1]: https://ahrefs.com/blog/llmstxt-study/ — measurement, 2026-05. 97% zero requests. Adoption percentages and the Shopify `/agents.md` + `/.well-known/ucp` rollout (first week of May 2026) are documented in this hub's `llms-txt-ecosystem-evidence.md`. [^lm-2]: https://modelcontextprotocol.io/specification/versioning — spec (primary, re-verified 2026-09-02). Current revision `2026-07-28`. Plus https://blog.modelcontextprotocol.io/posts/2026-07-28/ — largest revision since launch; stateless by default. [^lm-3]: https://github.com/modelcontextprotocol/modelcontextprotocol/issues/2127 (SEP-2 [source] — footnote block
[^sd-1]: https://github.com/nlweb-ai/NLWeb — repo. Reference implementation requires vector DB + embeddings + LLM + scorer. [^sd-2]: https://www.agentready.it.com/blog/llms-txt-vs-mcp-which-do-you-need — opinion, 2026-07-02. "Not competing standards." [^sd-3]: https://cloudnsite.com/blog/webmcp-vs-llms-txt-vs-mcp-server — opinion, 2026-06-19. Three layers that reinforce each other. [^sd-4]: https://www.helpsite.com/blog/llms-txt-help-center — vendor-blog, 2026-08-08. Publish llms.txt with calibrated expectations; MCP as load-bearing. [^sd-5]: https://www.mintlify.com/blog/state-of-docs-traffic [source] — footnote block
[^mcp-1]: https://github.com/nlweb-ai/NLWeb/blob/main/README.md — repo. "Every NLWeb instance also acts as an MCP server… `ask`"; "NLWeb is to MCP/A2A what HTML is to HTTP." [^mcp-2]: https://news.microsoft.com/source/features/company-news/introducing-nlweb-bringing-conversational-interfaces-directly-to-the-web/ — vendor-blog, 2025-05-19. Launch claim. *Not independent of [^mcp-1].* [^mcp-3]: https://raw.githubusercontent.com/nlweb-ai/NLWeb/main/AskAgent/python/webserver/mcp_wrapper.py — repo (primary; re-verified 2026-09-02). `MCP_PROTOCOL_VERSION = "2024-11-05"` at line 27; tool schemas; cap [source] — footnote block carrying source facts, not a problem/limitation itself
[^nw-1]: https://news.microsoft.com/source/features/company-news/introducing-nlweb-bringing-conversational-interfaces-directly-to-the-web/ — vendor-blog, 2025-05-19. Announcement; Guha's role; "similar role to HTML"; launch partners. [^nw-2]: https://techcrunch.com/2025/05/19/nlweb-is-microsofts-project-to-bring-more-chatbots-to-webpages/ — news, 2025-05-19. Independent date confirmation. Corroborated by https://siliconangle.com/2025/05/19/microsofts-nlweb-new-open-source-tool-integrates-generative-ai-search-website/. [^nw-3]: GitHub API `repos/nlweb-ai/NLWeb` (contributors, commits) — repo (d [source] — footnote block
**Disconfirming evidence — the wrapper was not sufficient in practice.** For ChatGPT, the project built a *second, separate* MCP server (Node/TypeScript, different tool name `nlweb-list`, via an AppSDK adapter to `/ask`).[^mcp-13] If the built-in `/mcp` were genuinely agent-ready that server would be unnecessary. Issue #255 records three independent clients (OpenAI Playground, MCP Inspector, then Copilot/Theia/LangGraph) failing to connect while other MCP servers worked from the same machine.[^mcp-8] The file itself warns: "This code is under development and may undergo changes in future relea [source]
**Transport is bespoke, not spec Streamable HTTP.** The handler comment states MCP "always uses regular JSON-RPC responses, not SSE."[^mcp-4] There is no `Mcp-Session-Id`, no `MCP-Protocol-Version` header handling, and no `Origin` validation. A full SSE implementation (`handle_mcp_streaming()`) exists in the file but is **never registered on a route** — dead code.[^mcp-4] [source]
The API hangs off **`document.modelContext`** — `registerTool()`, `getTools()`, `executeTool()`:[^lm-5] [source]
A **declarative** form also exists — annotations on standard HTML forms produce a tool.[^lm-6] The namespace has churned (`window.agent` → `navigator.modelContext` → `document.modelContext`, with `provideContext()` removed around March 2026); treat any code sample older than mid-2026 as wrong.[^lm-9] Access is gated by a permissions-policy feature `"tools"` with a default allowlist of `['self']`, and the API requires a SecureContext.[^lm-5] [source]
**Pinned to the original MCP revision.** `mcp_wrapper.py:27` sets `MCP_PROTOCOL_VERSION = "2024-11-05"` and `handle_initialize` echoes it unconditionally, ignoring the version the client requested.[^mcp-3] The MCP spec's own versioning page lists the current revision as `2026-07-28`.[^mcp-7] NLWeb therefore uses none of `structuredContent` / `outputSchema` (2025-06-18) or the Streamable HTTP transport (2025-03-26). A 2025 bug report shows a client sending `"protocolVersion": "2025-03-26"` and getting HTTP 400.[^mcp-8] [source]
**Origin.** Microsoft announced NLWeb on **2025-05-19**, timed to Build 2025.[^nw-1][^nw-2] It was conceived and built by **R.V. Guha**, who had recently joined Microsoft as CVP and Technical Fellow after ~two decades as a Google Fellow; Microsoft's own post calls him "the creator of widely used web standards such as RSS, RDF and Schema.org."[^nw-1] Guha is the repo's top contributor (456 commits).[^nw-3] The project's self-description is an analogy: **"NLWeb is to MCP/A2A what HTML is to HTTP."**[^nw-4] [source]
**Two divergent interfaces.** The shipped Python implementation serves `/ask` and `/mcp` with flat parameters (`query`, `site`, `prev`, `mode` ∈ {list, summarize, generate}, `streaming`, `query_id`) and **no server-side state** — conversation context must be resent.[^nw-8] The **spec v0.55** on nlweb.ai defines a *different* surface: `POST /ask` with a nested body plus `POST /await` for long-running promise results and SSE streaming, with `/mcp` relegated to "Appendix A: MCP Binding", and a *separate* `POST /who` agent-discovery spec.[^nw-9][^nw-10] **Spec and implementation have diverged — ch [source] — spec v0.55 vs shipped Python implementation diverge on endpoint shape/statefulness
**MCP as a general publication surface (as of 2026-09).** The MCP Registry (`registry.modelcontextprotocol.io`) is real and running but in preview; it is a *centralized metadata registry*, not a per-site convention.[^mcp-14] Three **competing and unratified** well-known paths are proposed — SEP-2127/SEP-1649 (`/.well-known/mcp/server-card.json`, IANA registration explicitly deferred until approval), IETF `draft-serra-mcp-discovery-uri-04` (`mcp://`, `/.well-known/mcp-server`, `_mcp.{host}` DNS TXT), and SEP #1960 (`/.well-known/mcp`, closed as duplicative).[^mcp-15][^mcp-16][^mcp-17] Zero IANA [source]
> **Takeaway.** "Every NLWeb instance is an MCP server" is literally true and materially weaker than it sounds: a > version-pinned, stateless, unpaginated, undiscoverable JSON-RPC route whose auth fails open. The genuine architectural gain > is server-side retrieval, not the MCP label. [source]
1. **Budget and attention.** One team, one quarter, one agent-readiness initiative. (Resolved by the decision framework above.) 2. **The answer surface.** If an agent can get a synthesized answer from your `/ask` endpoint, it will not crawl the pages your `llms.txt` points at — which changes what your logs, your analytics, and your monetization see. That is a real conflict, and it is a *business* conflict, not a technical one. [source]
Status, precisely: a **Draft Community Group Report** of the **W3C Web Machine Learning Community Group**, published **2026-08-26**, edited by Brandon Walderman (Microsoft) and Khushal Sagar + Dominic Farolino (Google).[^lm-5] A Community Group draft is **not** a W3C Recommendation and carries no standards-track authority. Chrome describes it as "a proposed web standard" available from **Chrome 149** via origin trial, with `chrome://flags/#enable-webmcp-testing` for local development.[^lm-6] [source]
**No discovery surface.** A recursive tree listing of the repo returns zero `.well-known/*` paths.[^mcp-10] An NLWeb `/mcp` is discoverable only out-of-band — you must already know the URL. [source]
> **`agents.md` disambiguation.** At least four distinct things share this name (the repo-root `AGENTS.md` coding > convention, Shopify's web-root `/agents.md`, shop.app's, and coincidental `.md` page renditions). This table's row is the > *web-root, agent-facing* sense. See this hub's `agents-md.md` spoke before citing any `agents.md` claim. [source]
**Deployment reality.** Python 3.10+, clone, venv, `pip install`, an LLM API key in `.env`, and **three YAML configs** (`config_llm.yaml`, `config_embedding.yaml`, `config_retrieval.yaml`), then load data with `db_load <RSS URL> <site-name>` and run `app-aiohttp.py`.[^nw-11] Hosting: local, Azure, Docker, or Cloudflare AutoRAG's managed one-click path.[^nw-6] **GCP and AWS are still marked "coming soon" in the README** ~13 months after launch.[^nw-4] [source]
[^ad-1]: GitHub API `repos/microsoft/NLWeb` (HTTP 301), `repos/nlweb-ai/NLWeb`, `orgs/nlweb-ai`, `/releases` — repo (direct observation, 2026-09-02). Transfer not fork; 6,253/699/65; 0 releases; MIT; org unverified. [^ad-2]: https://github.com/nlweb-ai/NLWeb/wiki/We've-moved-to-our-new-home! — repo wiki, edited 2025-07-30. Migration statement and stated rationale; silent on Microsoft's continuing role. [^ad-3]: GitHub API `/commits`, `/search/commits`, `/contributors` — repo (direct observation). Branch-prefix bracket 2025-07-25→2025-08-11; `main` HEAD 2026-06-10; all-Dependabot recent commits [source] — footnote block
**Dynamic costs are real and linear.** A modeled 100k-document RAG pipeline puts all-in cost per query at $0.0033 (1K queries/day) / $0.0017 (10K) / $0.0013 (100K), i.e. ~$98 / ~$496 / ~$4,000 per month, with *reranking* the largest line item at scale (~10,400 tokens read per query).[^sd-9] The author explicitly labels this a cost model, not production bills — treat as an estimate. NLWeb's own quickstart prerequisites make the burden concrete: an Azure account, pgvector on Azure PostgreSQL Flexible Server, an Azure AI Foundry project, three deployed models (gpt-4.1, gpt-4.1-mini, text-embeddin [source]
**Supported backends.** Vector stores: Qdrant, Snowflake, Milvus, Azure AI Search, Elasticsearch, Postgres/pgvector, OpenSearch, Cloudflare AutoRAG.[^nw-4][^nw-6] LLM providers: OpenAI, Anthropic, Gemini, DeepSeek, Inception, HuggingFace, Azure OpenAI (the shipped default), Ollama.[^nw-4] [source]
**What the MCP wrapper actually buys, honestly accounted:** [source]
The whole difference is **who executes retrieval**. With a static file the agent pulls it, decides what else to fetch, and does selection and reasoning inside its own context window. With a live endpoint the agent sends a query and the *site* runs retrieval, ranking and synthesis, returning an answer. NLWeb's reference implementation makes the server-side stack concrete: the operator provisions a vector DB, an embedding model, an LLM provider, and a neural scorer.[^sd-1] [source]
**Standing verdict: dormant-to-decelerating, not abandoned.** The repo is not archived, Cloudflare shipped managed support,[^ad-7] and Yoast announced a WordPress integration (Nov 2025).[^ad-8] But every adoption claim traces to the project or an interested vendor; no neutral source confirms a production NLWeb endpoint, and there is **no measurement study, registry, or directory of live NLWeb deployments** — the honest answer to "how many sites run one" is *nobody knows*. Spot-probing the conventional `/ask` path on named launch partners returned 403/301/404 (inconclusive — the path convention [source]
**The pipeline — and its cost.** Per `docs/life-of-a-chat-query.md`: pre-retrieval runs *parallel* LLM calls for relevance checking, decontextualization against history, and memory extraction, with a speculative "fast track" that may discard results; an LLM then reads a `tools.xml` manifest to select a tool and extract parameters; retrieval queries the vector DB; then **LLM-based scoring and snippet generation runs per result**. The docs state a single query "might involve **over 50 LLM API calls**" — a design the project calls "Mixed Mode Programming."[^nw-7] This is the number that governs t [source]
The **W3C NLWeb Community Group** was proposed by Guha 2025-10-16 and launched that month with ~15 participants and **no chair elected**; W3C states explicitly that hosting "does not imply endorsement."[^ad-9] A Community Group is *not* a Working Group and carries no standards-track authority. NLWeb is a project convention, not a web standard. [source]
> **Trap.** A static llms-full.txt is not automatically the "cheap" option for the *agent*. A 3.7M-token file is unusable and > a 481k-token one is expensive to consume. Size discipline is what makes the static path cheap; an unbounded dump forfeits > the advantage without buying any of the endpoint's benefits. [source]
`github.com/microsoft/NLWeb` returns **HTTP 301** and resolves to `nlweb-ai/NLWeb` — a repository *transfer*, not a fork (`parent: null`).[^ad-1] Two independent methods date it: a repo wiki page titled "We've moved to our new home!", edited **2025-07-30**, stating the move went "from microsoft/NLWeb to our new independent GitHub organization at NLWeb-ai" for "a neutral space where contributors from any company or background can collaborate";[^ad-2] and a commit-branch bracket — the last merge from a `microsoft/`-prefixed branch is 2025-07-25, the first from an `nlweb-ai/`-prefixed branch is 2 [source]

Disagreements

**Two divergent interfaces.** The shipped Python implementation serves `/ask` and `/mcp` with flat parameters (`query`, `site`, `prev`, `mode` ∈ {list, summarize, generate}, `streaming`, `query_id`) and **no server-side state** — conversation context must be resent.[^nw-8] The **spec v0.55** on nlweb.ai defines a *different* surface: `POST /ask` with a nested body plus `POST /await` for long-running promise results and SSE streaming, with `/mcp` relegated to "Appendix A: MCP Binding", and a *separate* `POST /who` agent-discovery spec.[^nw-9][^nw-10] **Spec and implementation have diverged — check which one a claim refers to.** (There is no `/sites` endpoint in either.) [source] — spec v0.55 vs shipped Python implementation diverge on endpoint shape/statefulness
Evidence against a tidy convergence story: three mutually incompatible proposed well-known URIs for MCP discovery with zero IANA registrations;[^lm-3] a WebMCP API that renamed its root object twice in under a year;[^lm-9] NLWeb pinned to an MCP revision five releases stale while the spec underwent its largest-ever rewrite;[^lm-2] and NLWeb's own spec and reference implementation exposing *different* endpoint sets. Governance is consolidating (MCP, AGENTS.md and goose all under the Linux Foundation's Agentic AI Foundation since Dec 2025; A2A under the LF since Jun 2025)[^lm-7] — but consolidation of *stewardship* is not convergence of *mechanism*. [source] — corroborates the same spec-vs-implementation divergence as s01u000042

Related concepts

llms.txt — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
dynamic endpoint — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
schema.org — is a part of NLWeb and MCP as agentic-discovery alternatives to llms.txt
/ask — is a part of NLWeb and MCP as agentic-discovery alternatives to llms.txt
Streamable HTTP — is a part of NLWeb and MCP as agentic-discovery alternatives to llms.txt
authentication — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
RSS — is a part of NLWeb and MCP as agentic-discovery alternatives to llms.txt
/mcp — is a part of NLWeb and MCP as agentic-discovery alternatives to llms.txt
protocol version — is a part of NLWeb and MCP as agentic-discovery alternatives to llms.txt
vector DB — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
Mintlify — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
freshness — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
Guha — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
A2A — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
robots.txt — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
Pay Per Crawl — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
Cloudflare AutoRAG — is a part of NLWeb and MCP as agentic-discovery alternatives to llms.txt
.well-known — is a part of NLWeb and MCP as agentic-discovery alternatives to llms.txt
OAuth — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt
decision framework — is a related of NLWeb and MCP as agentic-discovery alternatives to llms.txt