Stylometry and AI-text detector limits
Parent: Measuring LLM lexical tics: excess vocabulary, baselines, drift · Published reference · snapshot 2026-09-23
↓ Facts as markdownall context files
Source-anchored research for Stylometry and AI-text detector limits. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Measurements and reference values
- A 2025 audit (Jabarian & Imas, Chicago Booth) found Pangram, GPTZero and Originality.ai each under 1% false positives on human text, with false negatives of 2-4% (Pangram), 0-2% (GPTZero), 10-40% (Originality.ai); open-source RoBERTa was near random. All detectors lost accuracy on passages under 50 words. (medium confidence) (verified as of 2025-12-02) [source]
Comparisons and alternatives
- On 135,389 original vs professionally edited non-native academic manuscript pairs, 13 detectors (token-statistics, zero-shot, classifier) gave false-positive rates from 0.0% to 100.0% on identical human-written text; edit ratio correlated with score shifts (Spearman rho -0.418 to +0.343). Detectors partly measure linguistic polish, not origin. (medium confidence) (verified as of 2026-08-27) [source]
- Detector accuracy collapses roughly 25-50 percentage points on paraphrased, humanized, hybrid or lightly edited text versus unmodified LLM output. (medium confidence) (verified as of 2026-08-21) [source]
Facts and statements
- Detectors are better suited to population-level inference than to individual accusation; rescanning the same text can give inconsistent verdicts, and small per-document false-positive rates compound at scale. (medium confidence) [source]
- Detector performance is expected to drift as detectors, LLMs and humanizer tools compete; the Chicago Booth authors recommend regular re-audits. (medium confidence) [source]
Children
- No children recorded.