robots.txt and the Content-Signal AI-preference extension — researched

robots.txt is the Robots Exclusion Protocol (RFC 9309, Sept 2022, Proposed Standard): a crawl-access file of User-agent groups with Allow/Disallow rules, MUST-level wildcards (*, $), octet-based longest-match precedence with an allow-wins SHOULD tie-break, a 500 KiB parsing floor, and explicit non-goals (it is not access authorization). Riding inside it is a newer, unstandardized AI-preference layer — Cloudflare's Content-Signal (search/ai-input/ai-train, plus a testing content-use field) and its IETF successor draft (Content-Usage, AIPREF) — which states what a crawler may do with content after fetching it, has no announced enforcing consumer, and is evidenced (Perplexity dispute, Consent in Crisis study, Google's 'no effects whatsoever') to be a stated preference rather than a technical control. Drawn from two /dr-authored references (RFC 9309 spec analysis and the Content-Signal/AIPREF layer), 34 sourced footnotes.

Definitions

RFC 9309 answers exactly one question — *may you fetch this URL*. It says nothing about what a crawler may then **do** with what it fetched. This reference covers the layer that has grown inside robots.txt to answer that second question: Cloudflare's `Content-Signal:` extension, the evidence on whether anyone obeys it, and the IETF work standardising a successor. For the protocol itself, see `references/robots-txt.md`. For whether any of this constitutes a legally effective opt-out under EU law, see the `eu-ai-act-tdm-opt-out` skill — this reference deliberately stops at the technical and evid [source] — defines RFC 9309's scope boundary
The framing sentence in §1 is the one every downstream decision falls out of, and it stands alone as its own paragraph: **"These rules are not a form of access authorization."**[^rfc-1] robots.txt is a cooperative signal to well-behaved automation — not a security boundary, not an ACL, not enforceable by the protocol. [source]

Structure and components

- **`Sitemap` is named in RFC 9309 — explicitly as a record that is *not part of the protocol*.** "Sitemap isn't in RFC 9309" is half right: the token appears, the semantics are disclaimed, and the only mandate is that it must not break group parsing. - **`Crawl-delay` does not appear in RFC 9309 at all.**[^rfc-1] Nor does `Host`. - **`Allow` *is* normative** — §2.2.2 is literally titled *"The 'Allow' and 'Disallow' Lines"*. Sources that group `Allow` with the non-standard extensions are describing the pre-RFC era. - **An empty `Disallow:` value has no defined meaning in RFC 9309.** The ABNF a [source]
This reference covers the **protocol**: what RFC 9309 actually says, where real crawlers diverge from it, and how to author, serve, parse and test the file. The AI-preference layer that now rides inside robots.txt — Cloudflare's `Content-Signal:`, the `content-use` extension, compliance evidence, and the IETF `Content-Usage` successor — lives in `references/robots-txt-content-signals.md`. [source]
> *"The Robots Exclusion Protocol **is not a substitute for valid content security measures**. > **Listing paths in the robots.txt file exposes them publicly and thus makes the paths > discoverable.** To control access to the URI paths in a robots.txt file, users of the protocol > should employ a valid security measure relevant to the application layer… for example, in the case > of HTTP, HTTP Authentication as defined in [RFC9110]."*[^rfc-1] [source]
> *"Crawlers MAY interpret other records that are not part of the robots.txt protocol -- for example, > "Sitemaps" [SITEMAPS]. Crawlers MAY be lenient when interpreting other records… Parsing of other > records MUST NOT interfere with the parsing of explicitly defined records in Section 2. For > example, a "Sitemaps" record MUST NOT terminate a group."*[^rfc-1] [source]
A crawler picks **exactly one** group and ignores the rest. Google's wording is explicit: *"Only one group is valid for a particular crawler… All non-matching text is ignored."* Multiple groups naming the same specific user agent are merged, but — the critical part — ***"User agent specific groups and global groups (`*`) are not combined."***[^dp-1] [source]
Both agree on the operationally critical part — **a specific group does not inherit the `*` group** (§2.2 below). [source]

How it works

**Prepending has a precedence consequence.** Because the managed block is prepended and the origin's own `User-agent: *` group follows it, a file can end up with two `User-agent: *` groups. Under RFC 9309/Google semantics those merge into one group, so the origin's `Disallow` lines still apply — but a naive parser that stops at the first matching group will silently drop the origin's rules. [source]
The HTTP-header half of `attach` is the more significant long-term change: it moves the preference from a **site-wide file** to a **per-response header**, which finally lets a preference travel with a single resource, survive CDN caching semantics, and apply to responses that have no URL path in a robots.txt sense (APIs, redirects, dynamically negotiated representations). [source] — how the Content-Usage header travels with a response
Three-valued, not boolean: `yes` grants, `no` refuses, and **absence is not permission** — *"the website operator neither grants nor restricts permission."*[^cs-1] Cloudflare applies that rule to itself: it sets `ai-train=no` for managed customers but deliberately omits `ai-input`, because *"We don't know their preference with respect to that signal, and we don't want to guess."*[^cs-1] [source]
Because `Crawl-delay` is parsed by some libraries and not others, crawler-side code should read it defensively — `Protego` exposes `crawl_delay()`; Python's stdlib `urllib.robotparser` exposes it too but lacks wildcard support (§3.3).[^dp-6] [source]
Google: *"When matching robots.txt rules to URLs, crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule."*[^dp-1] Rules are therefore **order-independent** — moving a line up or down changes nothing; only path length and, on a tie, permissiveness decide. [source]

How-to and procedures

- Exactly one file, at the **authority root**: `https://example.com/robots.txt`. Each scheme + host + port is a separate authority with its own file — `https://example.com` and `https://shop.example.com` and `http://example.com` do **not** share one. - Serve it as `text/plain`, UTF-8, HTTP `200`. A robots.txt that returns HTML (a soft-404 page, a SPA shell, a CDN error page) is a common and silent failure. - Remember §1.7: a `404` is interpreted as **no restrictions at all**. If a file is meant to restrict, monitor that it keeps returning `200`. - Keep it under **500 KiB** — Googlebot discards [source] — serving requirement for authors

Measurements and reference values

| ID | § | Reported | Substance | |---|---|---|---| | 7124 | 5.2 | 2022-09-10 | The precedence example's URI says `disallow.gif`, the rule says `disallowed.gif` — *"renders the example given in section 5.2 incorrect"* | | 7128 | 2.2.2 | 2022-09-13 | The encoding table's `U+E38384` should be `U+30C4` (ツ) — a codepoint/UTF-8-bytes confusion | | 7995 | 2.2 | 2024-06-18 | `path-pattern = "/" *UTF8-char-noctl` forbids the RFC's **own** §5.1 example `Disallow: *.gif$`; proposes `("/" / "*")` | | 8895 | 2.3.1.5 | 2026-04-28 | Fabrice Canel proposes OPTIONAL comma-separated user-agent tokens (`User-ag [source]
Matching is octet-based after normalisation: characters outside ASCII, and reserved characters, *"MUST be percent-encoded … prior to comparison"*; a percent-encoded **unreserved** ASCII octet *"MUST be unencoded prior to comparison"*. So `/foo/bar/%62%61%7A` is compared as `/foo/bar/baz`.[^rfc-1] [source]
The `4xx`-means-allow-all rule is the one with teeth: **a robots.txt that 404s is not a safe default, it is an open door**, while a robots.txt that 503s buys only 30 days of protection. [source]
Baseline adoption of the file itself remains high: a 2025 measurement study found **96.4%** of sampled mainstream sites served a valid robots.txt, against **73.8%** of sampled misinformation sites (the sample frame and size are not restated here — check the paper before quoting).[^ce-5] [source]

Problems, failure modes and limitations

- **A group may be headed by several consecutive `user-agent` lines**, all sharing one rule set. The first `allow`/`disallow` closes the header. A group is *"terminated by a user-agent line or end of file"*, and *"the last group may have no rules, which means it implicitly allows everything."*[^rfc-1] - **`identifier` forbids digits.** The production is hyphen, `A-Z`, `_`, `a-z` — `%x30-39` is absent, and §2.2.1 says so in prose too. Real tokens containing digits (`MJ12bot`) are **not valid product tokens** under RFC 9309's grammar.[^rfc-1] - **`product-token` is a bare token, not a pattern an [source]
| Condition | Googlebot behaviour | |---|---| | File size | *"Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored."* | | `4xx` (except `429`) | *"treat all `4xx` errors, except `429`, as if a valid robots.txt file didn't exist"* — i.e. crawl everything | | `5xx` / `429`, 0–12 h | stops crawling the site, keeps retrying robots.txt | | `5xx` / `429`, 1–30 days | uses the last good cached version | | `5xx` / `429`, > 30 days | behaves as if there is no robots.txt (if the site is otherwise reachable) | | Redirects | *"Google [source]
| Topic | RFC 9309 normative text[^rfc-1] | |---|---| | Parsing limit | *"Crawlers SHOULD impose a parsing limit… **The parsing limit MUST be at least 500 kibibytes [KiB].**"* | | Caching | *"Crawlers MAY cache… Crawlers **SHOULD NOT use the cached version for more than 24 hours**, unless the robots.txt file is unreachable."* | | 2xx | *"the crawler MUST follow the parseable rules"* | | 3xx | *"Crawlers **SHOULD follow at least five consecutive redirects, even across authorities**"*; if reached within five, rules apply *"in the context of the initial authority"*; beyond five, MAY treat as unav [source]
- **500 KiB is a floor on the crawler's obligation, not a ceiling on the publisher.** The RFC says nothing at all about what happens past it — "content after the limit is ignored" is Google's rule, not the standard's.[^dp-1] (Google's own open-source parser contains no size-limit logic; the cap lives in the crawler.[^rfc-6]) - **The RFC never mentions `429`, `451` or `503` by number** — zero occurrences of each.[^rfc-1] Under the plain 400–499 rule a **429 is "Unavailable", so a conformant crawler MAY crawl everything** — the opposite of what most operators expect, and the opposite of Googlebo [source]
The RFC also instructs *implementors* to treat robots.txt itself as **untrusted content**, to reject out-of-bound characters, and notes that §2.5's parsing floor doubles as OOM protection.[^rfc-1] Note the register: these are flat declaratives, not RFC 2119 keywords — there is no "MUST NOT be used for access control." See §7 for the anti-pattern this creates. [source]
Erratum 7995 is the consequential one: **the published ABNF and the published examples contradict each other**, and have for over two years. Erratum 8895 is not a correction but a feature proposal, which likely explains why it sits unadjudicated; a comma is not a legal `identifier` character today. [source] — unresolved RFC/example contradiction
TENTATIVE / vendor-single-source: `content-use` is documented only by Cloudflare, is labelled by Cloudflare as under test, and may change or disappear. Do not build a parser that requires it — but **do** build one that tolerates it, because it is already being served at scale. [source] — content-use reliability caveat
**Perplexity's account (2025-08-04/05).**[^ce-3] Perplexity called Cloudflare's *systems* *"fundamentally inadequate for distinguishing between legitimate AI assistants and actual threats"* (its words for the analysis itself were "technical errors" and a "basic traffic analysis failure") and attributed the 3–6 M daily requests to **BrowserBase**, an unrelated third-party cloud-browser service, saying its own use of that service is *"less than 45,000 daily requests"*. Its substantive defence is architectural, and it is the important part: *"User-driven agents, by contrast, only fetch content wh [source]
The distinction that matters and is constantly muddled: **robots.txt is subtractive and adversarial** (it removes permission from a party that may not cooperate), while **llms.txt is additive and cooperative** (it offers convenience to a party that already has access). They are not competitors and one cannot substitute for the other. A `Disallow`d path is not made crawlable by listing it in llms.txt; a path listed in llms.txt is not made *findable* by a crawler that never looks for the file. For llms.txt's own spec, grammar, and adoption evidence, see `references/llms-txt.md` and `references/l [source]

Comparisons and alternatives

| Belief | Verdict | Why, and what to do instead | |---|---|---| | "`Disallow` removes the page from Google's index." | **False** | Disallow blocks *crawling*, not *indexing*. A disallowed URL can still be indexed from external links, shown URL-only. Use `<meta name="robots" content="noindex">` or the `X-Robots-Tag` header — and the page must stay **crawlable** for either to be seen. | | "Belt and braces: `Disallow` it *and* add `noindex`." | **Actively harmful** | The crawler cannot fetch the page, so it never sees the `noindex`. Disallow defeats noindex. Pick one: crawlable + `noindex` to de [source]
The managed block also carries hard `Disallow: /` groups for eight named AI crawlers — `Amazonbot`, `Applebot-Extended`, `Bytespider`, `CCBot`, `ClaudeBot`, `Google-Extended`, `GPTBot`, `meta-externalagent`.[^cs-3] Note the interaction with `robots-txt.md` §2.2: those groups **replace**, not supplement, the `User-agent: *` group for those crawlers — which is precisely why the managed file must repeat `Disallow: /` in each one rather than relying on the wildcard group. [source]
This is not in the launch blog post, but it is **not an accident**: it mirrors the ABNF of the IETF AIPREF `Content-Usage` rule, which likewise allows an optional `path-pattern` before the preference (§3.2). Read it as Cloudflare pre-aligning its syntax with the standards track. [source] — Content-Signal path-scoping vs AIPREF ABNF
- **Managed robots.txt customers (3.8 million+ domains at launch)** had their served file updated to carry the policy plus `Content-Signal: search=yes, ai-train=no`.[^cs-1] This is Cloudflare's own self-reported customer count; third-party coverage **repeats** the figure rather than independently verifying it.[^cs-4] - **Free-plan zones with no robots.txt of their own** are served the **comment block only** — no `Allow`/`Disallow`, no actual signals. *"The users are the ones to choose and express their actual preferences if and when they are ready to do so."*[^cs-1] This is opt-out via **Secur [source]
The optional `path-pattern` in that ABNF is exactly the shape of Cloudflare's path-scoped `Content-Signal` (§1.4) — strong evidence the two are converging deliberately rather than by coincidence. contentsignals.org even describes itself as *"An up-to-date guide to the IETF's proposed new AI Preferences (aipref)."*[^cs-2] [source]
RFC 9309 standardises a deliberately small surface. Everything else in a real robots.txt is a **de-facto extension** that some crawlers read and others silently discard — and because RFC 9309 requires parsers to ignore unrecognised lines rather than reject the file, an unsupported directive fails silently rather than loudly. That is the single most important operational fact in this section. [source]
This section is also why unknown directives such as `Content-Signal` and the proposed `Content-Usage` are safe to serve: a conformant parser must ignore what it does not recognise rather than reject the file. See `references/robots-txt-content-signals.md`. [source]

Changes and history

| Milestone | Detail | |---|---| | Feb 1994 | Martijn Koster proposes the mechanism on the `www-talk` list while at Nexor[^rfc-4] | | 30 Jun 1994 | *"A Standard for Robot Exclusion"* records consensus on the `robots` list — and disclaims itself: *"It is not an official standard backed by a standards body… It is not enforced by anybody"*[^rfc-2] | | Nov/Dec 1996 | `draft-koster-robots-00`, *"A Method for Web Robots Control"*, Informational — **introduces `Allow`**; expires Jun 1997, never published as an RFC[^rfc-3] | | 1 Jul 2019 | Google (Zeller, Sassman, Illyes) submits the REP to the IETF w [source] — protocol origin timeline

Facts and statements

| Field | In RFC 9309? | Googlebot | Bingbot | Notes | |---|---|---|---|---| | `User-agent` | yes | yes | yes | group header | | `Allow` | yes | yes | yes | | | `Disallow` | yes | yes | yes | | | `Sitemap` | **named in §2.2.4 as a non-protocol "other record"** | yes | yes | sitemaps.org convention; group-independent; absolute URL; MUST NOT terminate a group[^dp-7] | | `Crawl-delay` | **no — absent from the RFC entirely**[^dp-7] | **no — explicitly unsupported**[^dp-1] | yes[^dp-2] | | | `Host` | no | no | no | Yandex-specific | | `Clean-param` | no | no | no | Yandex-specific | | `Noindex` | n [source]
[^rfc-1]: **RFC 9309 — *Robots Exclusion Protocol***, M. Koster, G. Illyes, H. Zeller, L. Sassman; IETF, September 2022. Status **Proposed Standard** (in-document `Category: Standards Track`), IETF stream, non-working-group, ART area, DOI 10.17487/RFC9309. https://www.rfc-editor.org/rfc/rfc9309.txt · https://www.rfc-editor.org/rfc/rfc9309.html · https://www.rfc-editor.org/info/rfc9309 · https://datatracker.ietf.org/doc/rfc9309/ — every normative quote in §1: ABNF, §2.2.1 group selection, §2.2.2 matching and precedence, §2.2.3 special characters, §2.2.4 other records, §2.3 access method, §2.3.1 [source] — bibliographic citation
[^dp-1]: Google Search Central — *How Google interprets the robots.txt specification*. https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt — supported fields and `crawl-delay` non-support, 500 KiB limit, 4xx-except-429 / 5xx phases / redirect handling, "only one group is valid" and non-combination with `*`, least-restrictive tie-break, wildcards, case rules, 24 h cache, host/protocol/port scoping, BOM handling (docs) [^dp-2]: Bing Webmaster Blog — *To crawl or not to crawl, that is BingBot's question*, **May 2012**. https://blogs.bing.com/webmaster/May-2012/To-crawl-o [source] — bibliographic citation
[^dp-3]: Google Search Central Blog — *A note on unsupported rules in robots.txt*, July 2019. https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt — retirement of `noindex`, `nofollow` and `crawl-delay` handling effective **2019-09-01**; recommended alternatives (robots meta tag, `X-Robots-Tag`, 404/410); Google's "contradicted by other rules in all but 0.001% of robots.txt files" rationale (docs) [^dp-4]: See [^ce-1] — Google's stated position on `Content-Signal`. [^dp-5]: See `references/robots-txt-content-signals.md` §3.2 — `draft-ietf-aipref-attach` an [source] — bibliographic citation
| File | Job | Enforcement | Standardisation | Read by AI crawlers? | |---|---|---|---|---| | `robots.txt` | **crawl access** — may you fetch this URL | advisory; RFC 9309 §1 says it is not access authorization | **RFC 9309**, IETF Proposed Standard, Sept 2022 | yes, heavily | | `robots.txt` + `Content-Signal:` | **post-access usage** — what may you do with it | advisory; a stated preference | Cloudflare policy, CC0, 2025-09-24 | no announced consumer[^ce-1] | | `robots.txt` + `Content-Usage:` | post-access usage, standardised | advisory | `draft-ietf-aipref-attach` — **`Updates: 9309 (if appr [source]
- Established, mainstream **search** crawlers comply with `Disallow` and have done for decades; this is not seriously contested. - **AI training crawlers** with published tokens (`GPTBot`, `ClaudeBot`, `Google-Extended`, `CCBot`, `Applebot-Extended`) generally honour token-scoped `Disallow`, which is why those tokens are worth setting. - **User-triggered agent fetches** are the live gap — several operators state on the record that robots.txt does not govern them. - **Post-access usage signals** (`Content-Signal`, `ai-train=no`) currently have **no announced consumer**. Google has said so expli [source]
[^ce-1]: Search Engine Roundtable — *Google: Content Signals & llms.txt have "no effects whatsoever"*, **2026-07-06**. https://www.seroundtable.com/google-cloudflare-content-signals-41631.html — John Mueller (Google), on Reddit r/TechSEO: *"it has no effects whatsoever for any crawler or llm. Using it just adds bloat & future maintenance to your robots.txt file."* The same statement is recorded in `references/llms-txt.md` §7 (its footnotes 36-37) (press) [^ce-2]: Cloudflare Blog — *Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives*, **2025-08-04**. https://b [source] — bibliographic citation
[^sf-1]: IETF AIPREF working group document list. https://datatracker.ietf.org/wg/aipref/documents/ — two active WG documents, both revised 2026-08-18/19, both targeting Proposed Standard (spec) [^sf-2]: `draft-ietf-aipref-vocab-07` — *A Vocabulary For Expressing AI Usage Preferences*, 19 August 2026, Standards Track, expires 2027-02-20. https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-07.html — `train-ai` and `search` category definitions; `y`/`n` value alphabet; *"in the absence of a statement of preference, all usage categories are assigned a preference value of 'unknown'"*; takes *" [source]
[^cs-1]: Cloudflare Blog — *Giving users choice with Cloudflare's new Content Signals Policy*, Will Allen, published **2025-09-24**, modified 2026-07-15. https://blog.cloudflare.com/content-signals-policy/ — the launch announcement: verbatim policy text, the three signal definitions, syntax, the 3.8 M managed-robots.txt figure, free-zone behaviour, CC0, "preferences not countermeasures" (vendor docs) [^cs-2]: ContentSignals.org — the policy site and generator. https://contentsignals.org/ — verbatim signal definitions, four default policies, the legal disclaimer, and the **path-scoped** "Advanc [source]
- **`draft-ietf-aipref-vocab-07`** — *"A Vocabulary For Expressing AI Usage Preferences."* Defines the preference categories, including `train-ai` (*"Using an asset to modify the learned parameters of an AI model that is used to generate synthetic content in one or more modalities"*) and `search`. Values are single characters: **`y`** and **`n`**. Crucially, *"in the absence of a statement of preference, all usage categories are assigned a preference value of 'unknown'"*, and the draft *"takes no position on what default might be assigned."*[^sf-2] - **`draft-ietf-aipref-attach-05`** — *"Assoc [source]
| Cloudflare `Content-Signal` | IETF `Content-Usage` | Difference | |---|---|---| | `ai-train=no` | `train-ai=n` | token order reversed; `no` → `n` | | `search=yes` | `search=y` | value alphabet | | `ai-input=no` | (see vocab categories) | not a 1:1 rename | | `use=reference` (testing) | — | Cloudflare-only extension | [source]
> *"The matching **SHOULD** be case sensitive. The matching **MUST** start with the first octet of the > path. **The most specific match found MUST be used. The most specific match is the match that has > the most octets.** … **If an "allow" rule and a "disallow" rule are equivalent, then the "allow" > rule SHOULD be used.** If no match is found … the URI is allowed. **The /robots.txt URI is > implicitly allowed.**"*[^rfc-1] [source]
| Tool | Language | Wildcards (`*`/`$`) | Notes | |---|---|---|---| | `google/robotstxt` | C++ (C++14), Apache-2.0 | yes | *"slightly modified production code used by Googlebot"*, released *"to help developers build tools that better reflect Google's robots.txt parsing and matching"*[^au-1] — the closest thing to a reference implementation | | `Protego` | pure Python | **yes** | Scrapy's default parser; *"compliant with Google's Robots.txt Specification"*; exposes `crawl_delay()`, `request_rate()`, `sitemaps`, `preferred_host`[^au-2][^au-3] | | `urllib.robotparser` (stdlib) | Python | **no** | [source]
Google's John Mueller, **2026-07-06**, on Reddit's r/TechSEO, on Content Signals and llms.txt: *"it has no effects whatsoever for any crawler or llm. Using it just adds bloat & future maintenance to your robots.txt file."*[^ce-1] This is the same statement recorded in `references/llms-txt.md` §7 — see there for the llms.txt side; the point for robots.txt is narrower and sharper: **Google honours `Disallow` and `Google-Extended`, and does not act on `Content-Signal`.** Cloudflare has never claimed otherwise; the policy's own text asks crawlers to opt in. [source]
``` User-Agent: * Allow: / Content-Usage: train-ai=n ``` ``` Content-Usage: /ai-ok/ train-ai=y ``` [source]
- **Googlebot ignores it entirely.** Google's field list is exhaustive and calls it out by name; Google's guidance is to use Search Console's crawl-rate controls instead.[^dp-1] - **Bingbot honours it**, but **not as a rate, and not as a simple inter-request delay**. Bing: *"One common mistake is that `Crawl-delay` does not represent a crawl rate. Instead, it defines the size of a time window (from 1 to 30 seconds) during which BingBot will crawl your web site only once."*[^dp-2] So `Crawl-delay: 5` caps Bing at *"a maximum of around 17,280 pages during the day"*.[^dp-2] The higher the number, [source]
[^au-1]: google/robotstxt — repository. https://github.com/google/robotstxt — Apache-2.0 C++ parser, *"slightly modified production code used by Googlebot"*, released *"to help developers build tools that better reflect Google's robots.txt parsing and matching"* (readme) [^au-2]: scrapy/protego — README comparison matrix (Protego / RobotFileParser / Reppy / Robotexclusionrulesparser). https://github.com/scrapy/protego (readme) [^au-3]: Scrapy documentation — downloader middleware / robots.txt parsers. https://docs.scrapy.org/en/2.11/topics/downloader-middleware.html — Protego is the default; * [source] — bibliographic citation
Google states the supported set exhaustively: *"Google supports the following fields (other fields such as `crawl-delay` aren't supported): `user-agent`, `allow`, `disallow`, `sitemap`."*[^dp-1] [source]
**Bing's group-selection order has its own trap.** BingBot honours exactly one section, in priority order: the `bingbot` section → the `msnbot` section (backwards compatibility) → the default wildcard. Consequence, in Bing's words: *"if you have old directives blocking MSNBot, you are also blocking BingBot altogether as a side effect."*[^dp-2] A decade-old `User-agent: msnbot` / `Disallow: /` block still de-indexes a site from Bing today. [source]
**Google's rule is different and stricter:** *"Only one group is valid for a particular crawler… finding … the group with the **most specific user agent** that matches the crawler's user agent. Other groups are ignored. All non-matching text is ignored (for example, both `googlebot/1.2` and `googlebot*` are equivalent to `googlebot`)."*[^dp-1] Google's prefix tolerance is doubly non-RFC: `/` and `*` are not legal characters inside an RFC `identifier`. [source]
> **Scope boundary.** This reference owns the **robots.txt directive** — its syntax, semantics and > evidence. Configuring the Cloudflare *products* around it (AI Crawl Control allow/block/charge, > pay-per-crawl / HTTP 402, Web Bot Auth / RFC 9421 signed agents, BotBase) belongs to the > `cloudflare-platform` skill. [source]
The blog and contentsignals.org write `Content-Signal:`; the managed robots.txt Cloudflare actually serves writes `Content-signal:`.[^cs-1][^cs-2][^cs-3] Since robots.txt field names are conventionally case-insensitive, this is harmless *if* consumers lowercase before comparing — but it is a live reminder to **match the field name case-insensitively** and never with a literal string equality test. [source]
Case rules are asymmetric and routinely got wrong: - The **field name** (`disallow`, `user-agent`) is case-insensitive.[^dp-1] - The **user-agent value** is case-insensitive — `GPTBot`, `gptbot` and `GPTBOT` all match.[^dp-1] - The **path value is case-SENSITIVE**.[^dp-1] `Disallow: /Private/` does not block `/private/`. [source]
**RFC 9309 §2.2.1:** *"Crawlers **MUST use case-insensitive matching** to find the group that matches the product token… **If there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group**… If no matching group exists, crawlers **MUST obey the group with a user-agent line with the `*` value**, if present. If no group matches … and there is no group with a user-agent line with the `*` value … **no rules apply**."*[^rfc-1] [source]
- **Google Search Console → Settings → robots.txt report** — shows the robots.txt files Google found for the **top 20 hosts** on the property, plus warnings/errors, and can request an emergency recrawl. It **replaced the standalone robots.txt Tester**, which Google sunset in **November 2023**.[^au-5] - **Bing Webmaster Tools still ships a robots.txt tester**, so it remains the quickest way to test a specific URL against a specific user agent interactively.[^au-5] - CI-side, the durable check is not "is the syntax valid" but **"does the verdict for these N URLs under these M user agents still m [source]
Announced **2025-09-24** by Will Allen on the Cloudflare blog, the Content Signals Policy is an addition to robots.txt that expresses *what a crawler may do with content after it has fetched it* — a question RFC 9309 deliberately never addresses. Cloudflare's own framing: robots.txt *"does not, however, let them know what they are able to do with your content after accessing it."*[^cs-1] The policy is released under **CC0**, explicitly to let anyone adopt it without Cloudflare.[^cs-1] [source]
1. Specificity is measured in **octets**, not characters — it differs for multi-byte and percent-encoded paths. 2. Longest-match is a **MUST**; the allow-wins tie-break is only a **SHOULD**. Two conformant crawlers may legitimately differ on a tie, so never author a file whose correctness depends on it. 3. The RFC's word is **"equivalent"**, which it never defines — not "the same length". [source]
1. **A human-readable policy preamble**, entirely inside `#` comments — the definitions plus a legal reservation of rights. Crawlers ignore it; lawyers do not. 2. **A `Content-Signal:` line**, a real directive line inside a `User-agent` group. [source]
| Signal | Cloudflare's verbatim definition[^cs-1] | Covers | |---|---|---| | `search` | *"building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website's contents). Search does not include providing AI-generated search summaries."* | classic search indexing **only** | | `ai-input` | *"inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers)."* | RAG, grounding, AI Overviews | | `ai-train` | *"training or fine-tuning AI models."* | [source]
The `search` definition carves AI summaries **out** of search. A publisher that sets `search=yes` and omits `ai-input` has therefore said nothing about AI Overviews — the exact case most publishers care about. That is a design consequence worth flagging to anyone deploying this. [source]
**The RFC defines no "most specific user-agent" rule.** The word *specific* appears in RFC 9309 only in the **path**-matching sentence. The RFC's algorithm is: case-insensitive exact match → merge all matching groups → else fall back to `*` → else no rules. [source]
**Cloudflare's account (2025-08-04).**[^ce-2] Method: Cloudflare registered *"multiple brand-new domains, similar to `testexample.com` and `secretexample.com`"* — newly purchased, unpublicised — served a robots.txt prohibiting all automated access, then asked Perplexity about those domains. Findings: Perplexity returned detailed content anyway; alongside its declared `Perplexity-User/1.0` agent (20–25 M requests/day) Cloudflare observed an undeclared generic Chrome user agent (3–6 M requests/day) *"utilized multiple IPs not listed in Perplexity's official IP range"* and *"requests coming from [source]
**What this does and does not establish.** It does not establish that Perplexity ran a stealth crawler — the attribution is contested and was never independently adjudicated. It *does* establish the load-bearing disagreement in the whole field: **whether robots.txt binds a fetch made on a live user's behalf.** Several operators run the same argument, and it is the reason a `Disallow` can be honoured by a vendor's training crawler while the same vendor's assistant fetches the page anyway. Any robots.txt strategy that ignores the user-triggered path is incomplete. [source]
- **Cloudflare, 2025-07-01 ("Content Independence Day")** changed the default for new domains to **block AI crawlers** unless they pay.[^ce-6] The same post frames the economics: it reports crawl-to-referral ratios making traffic acquisition *"750 times more difficult"* via OpenAI and *"30,000 times more difficult"* via Anthropic than the Google of old (the post does not state a measurement window — treat the figures as illustrative, not reproducible). - **Cryptographic bot identity** is the direction of travel: Web Bot Auth built on HTTP Message Signatures (RFC 9421), letting a crawler prove [source]
*"The rules MUST be accessible in a file named '/robots.txt' (all lowercase) in the top-level path of the service."* The identifier is `scheme:[//authority]/robots.txt`, and the RFC gives `ftp://ftp.example.com/robots.txt` alongside HTTPS — the protocol is **transport-agnostic**, not HTTP-specific.[^rfc-1] [source]
Note what the RFC does *not* say: it **never uses the words "subdomain" or "port".**[^rfc-1] Per-scheme, per-host, per-port scoping is implied *structurally* by binding the file to a `scheme` + `authority` (RFC 3986), never stated in prose. The explicit statement lives only in implementer docs — Google: *"The rules listed in the robots.txt file apply only to the host, protocol, and port number where the robots.txt file is hosted."*[^dp-1] [source]
[^cs-1]: Cloudflare Blog — *Giving users choice with Cloudflare's new Content Signals Policy*, Will Allen, **2025-09-24**. https://blog.cloudflare.com/content-signals-policy/ — cited here only for the `Content-Signal` example; full treatment in `references/robots-txt-content-signals.md` (vendor docs) [^ce-1]: Search Engine Roundtable — *Google: Content Signals & llms.txt have "no effects whatsoever"*, **2026-07-06**. https://www.seroundtable.com/google-cloudflare-content-signals-41631.html — John Mueller (Google) on Reddit r/TechSEO; full treatment in `references/robots-txt-content-signals.md` [source]
The file **MUST be UTF-8 encoded (RFC 3629)** and media type `text/plain`.[^rfc-1] Core ABNF:[^rfc-1] [source]
Google's reference C++ parser resolves the tie with a strict inequality in favour of allow (`return (disallow_.specific.priority() > allow_.specific.priority());`, both priorities being `pattern.length()`) — executable confirmation of the SHOULD.[^rfc-6] [source]
⚠ The RFC's own §5.2 precedence example is **broken as published**: the prose URI reads `disallow.gif` while the rule reads `disallowed.gif` (Erratum 7124, §1.9). [source]
Cloudflare's bots documentation (last updated **2026-08-03**) describes an extension **not present in the 2025-09-24 policy text and not offered by the contentsignals.org generator**: a `use=` field *"Cloudflare is testing … an optional extension to Content Signals."*[^cs-3] [source]
| Value | Cloudflare's meaning[^cs-3] | |---|---| | `use=immediate` | *"Interact, but store and reuse nothing."* | | `use=reference` | *"Index, excerpt, and link back."* | | `use=full` | *"Summarize and reproduce."* | [source]
Managed-robots.txt customers now receive it by default, which is why the live managed block reads:[^cs-3] [source]
contentsignals.org documents an "Advanced Usage" form that puts a **path prefix before the preference list**:[^cs-2] [source]
The IETF **AIPREF** working group has two active documents, both revised **2026-08-18/19**, both targeting Proposed Standard:[^sf-1] [source]
> Cite it as: *RFC 9309, "Robots Exclusion Protocol", Koster, Illyes, Zeller, Sassman, September 2022; > Proposed Standard (Standards Track), IETF stream, non-WG, ART area.* The RFC gives only > "September 2022" — do not assert a specific day. [source]
Cloudflare says so itself, twice over. From the announcement: *"content signals express preferences; they are not technical countermeasures against scraping. Some companies might simply ignore them."* The recommended pairing is WAF rules plus Bot Management.[^cs-1] The contentsignals.org generator is blunter still: *"Courts and regulators may conclude that robots.txt files do not impose enforceable legal obligations. For more information regarding your rights and the significance of robots.txt, consult a lawyer."*[^cs-2] [source]
GPTBot matches the second group, and therefore **ignores the first entirely** — it is free to crawl `/admin/` and `/internal/`. The rules must be repeated in every specific group that needs them: [source]
| URL | Longest matching rule | Verdict | |---|---|---| | `/downloads/report.pdf` | `Disallow: /downloads/` (11) | blocked | | `/downloads/free/guide.pdf` | `Allow: /downloads/free/` (16) | allowed | | `/downloads/free/beta1.zip` | `Disallow: /downloads/free/beta` (20) | blocked | [source]
`*` matches *"0 or more instances of any valid character"*; `$` designates *"the end of the URL."*[^dp-1] [source]
`Disallow: /tmp` being a **prefix** match, not a path-segment match, is a recurring source of over-blocking. [source]
Get the status string right, because it is routinely overstated. The RFC body says `Category: Standards Track`; the *maturity level* recorded by the RFC Editor and the IETF datatracker is **Proposed Standard** — the lowest rung of the standards track.[^rfc-1] It is an **individual submission in the ART area, not a working-group product**, and it **Updates/Obsoletes nothing** and is **Updated by nothing** to date.[^rfc-1] Authors: M. Koster; G. Illyes, H. Zeller, L. Sassman (Google LLC). DOI 10.17487/RFC9309. [source]
§2.2.3 is titled "Special Characters" and opens *"Crawlers **MUST** support the following special characters"*: `#` (line comment), `$` (end of match pattern), `*` (*"0 or more instances of any character"*; §5.1 adds that this includes *"the otherwise-required forward slash"*).[^rfc-1] To match one literally, percent-encode it — `%23`, `%2A`, `%24`. [source]
This corrects one of the most common claims in SEO literature: wildcards were a de-facto extension in the 1994/1996 era, but under RFC 9309 support is **MUST**-level. [source]
All four are status **Reported**, type Technical; none Verified, none Rejected.[^rfc-7] [source]
`/reports/q4-summary` → both rules are 11 characters → **allowed** (least restrictive). [source]
The Data Provenance Initiative's *"Consent in Crisis: The Rapid Decline of the AI Data Commons"* measured robots.txt restrictions across the head domains of C4, RefinedWeb and Dolma. In the single year **Apr 2023 → Apr 2024**, the paper's headline figures are **C4-specific**: *"rendering ~5%+ of all tokens in C4, or **28%+** of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full **45% of C4** is now restricted."*[^ce-4] Note both robots.txt figures are **floors** ("+"), not ranges, and they describe C4 — RefinedWeb [source] — names the Consent in Crisis study
[^ce-5]: *Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web*, arXiv 2510.10315. https://arxiv.org/html/2510.10315v1 — robots.txt adoption baseline: 96.4% of sampled mainstream sites served a valid file versus 73.8% of sampled misinformation sites (paper) [^ce-6]: Cloudflare Blog — *Content Independence Day: no AI crawl without compensation*, **2025-07-01**. https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ — default-block for new domains; crawl-to-referral ratio figures (750× OpenAI, 30,000× Anthropic, ~10× Google) with **no stated m [source] — bibliographic citation
robots.txt was an informal convention for 28 years before the IETF standardised it. [source]
Cloudflare's verbatim preamble:[^cs-1][^cs-2] [source]
`Crawl-delay: <n>` was never standardised and support is genuinely split — and the two vendors that matter disagree about what it even *means*. [source]

Related concepts

wildcard — is a part of robots.txt and the Content-Signal AI-preference extension
RFC 9309 — is a synonym of robots.txt and the Content-Signal AI-preference extension
Cloudflare — is a part of robots.txt and the Content-Signal AI-preference extension
precedence — is a part of robots.txt and the Content-Signal AI-preference extension
User-agent — is a part of robots.txt and the Content-Signal AI-preference extension
Disallow — is a part of robots.txt and the Content-Signal AI-preference extension
group — is a part of robots.txt and the Content-Signal AI-preference extension
Allow — is a part of robots.txt and the Content-Signal AI-preference extension
ai-train — is a part of robots.txt and the Content-Signal AI-preference extension
Protego — is a part of robots.txt and the Content-Signal AI-preference extension
ABNF — is a part of robots.txt and the Content-Signal AI-preference extension
Content-Usage — is a synonym of robots.txt and the Content-Signal AI-preference extension
AIPREF — is a part of robots.txt and the Content-Signal AI-preference extension
Crawl-delay — is a part of robots.txt and the Content-Signal AI-preference extension
Googlebot — is a hyponym of robots.txt and the Content-Signal AI-preference extension
Host — is a part of robots.txt and the Content-Signal AI-preference extension
search=yes — is a part of robots.txt and the Content-Signal AI-preference extension
GPTBot — is a hyponym of robots.txt and the Content-Signal AI-preference extension
Sitemap — is a part of robots.txt and the Content-Signal AI-preference extension
ai-input — is a part of robots.txt and the Content-Signal AI-preference extension