Mode collapse and typicality bias in preference tuning
Parent: Why LLMs converge on the same words: mode collapse and post-training lexical effects · Published reference · snapshot 2026-09-23
↓ Facts as markdownall context files
Source-anchored research for Mode collapse and typicality bias in preference tuning. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Measurements and reference values
- Post-training methods sharpen the output probability distribution and reduce diversity of generated responses; DivPO counters this by picking rarer, higher-quality responses as preferred and common lower-quality ones as rejected, reporting +45.6% persona attribute diversity and +74.6% story diversity with comparable quality. (medium confidence) [source]
- Verbalized Sampling, a training-free prompt asking the model to verbalize a distribution of responses with probabilities, reports 1.6-2.1x diversity gains in creative writing, with larger gains for more capable models. (low confidence) (verified as of 2026-07-15) [source]
Comparisons and alternatives
- RLHF significantly reduces output diversity compared with SFT across several measures, while generalising better out of distribution; the authors frame this as a generalisation-diversity tradeoff (tested on two base models, summarisation and instruction following). (high confidence) [source]
Facts and statements
- Verbalized Sampling paper identifies typicality bias in preference data (annotators systematically favour familiar text) as a data-level driver of mode collapse, formalised theoretically and checked on preference datasets; it claims collapse occurs even with a perfect reward model. (medium confidence) [source]
- Aligned (RLHF) Llama-2 models show lower token-prediction entropy, distinct clusters in embedding space, and 'attractor states', indicating constrained diversity; study covers only the Llama-2 family. (medium confidence) [source]
Children
- No children recorded.