# Mode collapse and typicality bias in preference tuning > Source-anchored research for Mode collapse and typicality bias in preference tuning. Run status: COMPLETED. Parent: [Why LLMs converge on the same words: mode collapse and post-training lexical effects](https://llms-explorer.com/tree/ai-isms-origins/) · 3 facets · 5 facts · page: https://llms-explorer.com/tree/mode-collapse-and-typicality-bias-in-preference-tuning/ ## Measurements and reference values - Post-training methods sharpen the output probability distribution and reduce diversity of generated responses; DivPO counters this by picking rarer, higher-quality responses as preferred and common lower-quality ones as rejected, reporting +45.6% persona attribute diversity and +74.6% story diversity with comparable quality. (medium confidence) — [source](https://arxiv.org/abs/2501.18101) - Verbalized Sampling, a training-free prompt asking the model to verbalize a distribution of responses with probabilities, reports 1.6-2.1x diversity gains in creative writing, with larger gains for more capable models. (low confidence) (verified as of 2026-07-15) — [source](https://arxiv.org/abs/2510.01171) ## Comparisons and alternatives - RLHF significantly reduces output diversity compared with SFT across several measures, while generalising better out of distribution; the authors frame this as a generalisation-diversity tradeoff (tested on two base models, summarisation and instruction following). (high confidence) — [source](https://arxiv.org/abs/2310.06452) ## Facts and statements - Verbalized Sampling paper identifies typicality bias in preference data (annotators systematically favour familiar text) as a data-level driver of mode collapse, formalised theoretically and checked on preference datasets; it claims collapse occurs even with a perfect reward model. (medium confidence) — [source](https://arxiv.org/abs/2510.01171) - Aligned (RLHF) Llama-2 models show lower token-prediction entropy, distinct clusters in embedding space, and 'attractor states', indicating constrained diversity; study covers only the Llama-2 family. (medium confidence) — [source](https://arxiv.org/abs/2406.05587)