Excess-vocabulary frequency method
Parent: Measuring LLM lexical tics: excess vocabulary, baselines, drift · Published reference · snapshot 2026-09-23
↓ Facts as markdownall context files
Source-anchored research for Excess-vocabulary frequency method. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Measurements and reference values
- Headline estimate: lower bound of at least 13.5% of 2024 PubMed abstracts processed with LLMs, ranging roughly 5-40% across fields and countries. Earlier preprint versions reported 10%; the figure is a lower bound and moved between versions. (low confidence) (verified as of 2025-07-02) [source]
Problems, failure modes and limitations
- Frequency-based measurement is valid at population level only; it cannot label individual texts, and results from PubMed/arXiv abstracts may not generalise to other genres. (medium confidence) [source]
Comparisons and alternatives
- Excess words in 2024 are predominantly style words (about 66% verbs, 14% adjectives, e.g. 'meticulously', 'intricate', 'pivotal'), whereas COVID-era excess was mostly content nouns; this style-vs-content split is the diagnostic that separates LLM shift from topic shift. (low confidence) [source]
- Model-side comparison of Llama Base vs Instruct on continuations of 9,853 PubMed abstracts finds large post-alignment increases (e.g. 'nuanced' +8,342%), and 400 raters preferred variants with more such words (52.4% vs 47.6%), linking part of the overuse to RLHF; scope is Llama only, English abstracts, with some false-positive words. (low confidence) [source]
Facts and statements
- The method projects each word's expected 2024 frequency by linear extrapolation from 2021-2022 (2023 excluded as possibly already LLM-influenced), then flags words whose observed frequency exceeds the projection; it needs no labelled LLM text. (high confidence) [source]
- Two complementary metrics are used: frequency ratio r (observed/expected, surfaces rare words) and frequency gap delta (observed minus expected, surfaces common words); thresholds such as delta > 0.01 or a log-scaled ratio criterion were chosen so pre-pandemic years show minimal excess. (low confidence) [source]
- Once a word list is publicised, frequencies change: 'delve' and 'intricate' dropped sharply on arXiv from March-April 2024, while less suspicious words like 'significant' and 'additionally' kept rising. Excess counts of famous tic words therefore understate later LLM use. (low confidence) (verified as of 2025-02-13) [source]
Children
- No children recorded.