Verifying a vocabulary fix with before-and-after tic rates
Parent: Instruction-file hygiene for LLM lexical tics: linting, mirroring, hook filtering · Published reference · snapshot 2026-09-23
↓ Facts as markdownall context files
Source-anchored research for Verifying a vocabulary fix with before-and-after tic rates. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Comparisons and alternatives
- EQ-Bench's Slop Score is a weighted composite (60% slop words, 25% not-X-but-Y patterns, 15% slop trigrams) computed against human baselines (wordfreq for words, a human essay/creative corpus for trigrams); it is not an AI-vs-human classifier and may underreport outside creative writing and essays. (medium confidence) [source]
- Arena's baseline counts on Opus 5 (510 vs 158 words per response against Opus 4.5, em dashes 2.3x, 'load-bearing' about 2x, sentences 58% longer) show per-response rates that can serve as a before figure, but the guide's fix was written for Fable 5.1 and nothing shows it works on Opus 5. (low confidence) (verified as of 2026-08-12) [source]
Facts and statements
- The Verbal Tic Index (VTI) combines TicRate (share of responses containing a tic), MATTR lexical diversity, a sycophancy score and phrase repetition rate with weights 0.3/0.2/0.3/0.2; the authors say other weights may reorder models, so it is a comparative summary, not an absolute measure. Use response-level tic share as a simple before/after rate. (medium confidence) [source]
- The VTI study is observational (160,000 responses, 8 models); it reports no before/after intervention, so it gives baselines but no evidence of how much a prompt fix reduces tics. (medium confidence) [source]
- Anthropic's Fable 5.1 guide supplies a 'mannered prose' definition paragraph (or the short 'Please remove all mannered prose.') to add to a user message or system prompt, and tells users to test against their own evals, but publishes no measured before/after rate for it. (medium confidence) (verified as of 2026-09-23) [source]
- Automatic lexical metrics only partly capture slop: in a copy-editor annotation study, linear models on available automatic metrics reached AUPRC 0.52-0.55 on news and MS MARCO, and zero-shot GPT-4 judging scored kappa near 0, so a tic-rate drop does not prove the text is better. (medium confidence) [source]
Children
- No children recorded.