Diversity-restoring decoding and prompting
Parent: Why LLMs converge on the same words: mode collapse and post-training lexical effects · Published reference · snapshot 2026-09-23
↓ Facts as markdownall context files
Source-anchored research for Diversity-restoring decoding and prompting. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- Verbalized Sampling is a training-free prompt that asks the model to verbalize a probability distribution over multiple responses (e.g. 'Generate 5 jokes about coffee and their corresponding probabilities') to circumvent mode collapse. (high confidence) [source]
Comparisons and alternatives
- VS authors attribute mode collapse to typicality bias in preference data and report 1.6-2.1x creative-writing diversity over direct prompting without hurting factual accuracy or safety; results are author-reported. (medium confidence) (verified as of 2026-07-01) [source]
- VS gains are larger for more capable models than for smaller mini/flash variants. (medium confidence) [source]
- A reanalysis (Schaeffer et al., June 2025) finds min-p does not beat baseline samplers on quality, diversity or their trade-off once hyperparameter count is controlled, cites omitted human-eval data and inconsistent LLM-judge reporting, and notes adoption figures were unsubstantiated and removed. (medium confidence) [source]
Facts and statements
- Min-p sampling scales the truncation threshold by the top token's probability; the original paper claims better quality and diversity than top-p at high temperature and reports adoption in Hugging Face Transformers and vLLM. (medium confidence) [source]
Children
- No children recorded.