Generation configurations: temperature, top-k, top-p, and test time compute
Named practitioner synthesis
#196
- Canonical URL
- https://huyenchip.com//2024/01/16/sampling.html
- Topic
- Practitioner Notes
- First seen
- 2026-07-16 19:08:00
- Last seen
- 2026-07-16 19:08:00
Source raw items (1)
- Blog / Newsletter2026-07-16 19:07:19Generation configurations: temperature, top-k, top-p, and test time compute
<p>ML models are probabilistic. Imagine that you want to know what’s the best cuisine in the world. If you ask someone this question twice, a minute apart, their answers both times should be the same. If you ask a model the same question twice, its answer can change. If the model thinks that Vietnamese cuisine has a 70% chance of being the best cuisine and Italian cuisine has a 30% chance, it’ll answer “Vietnamese” 70% of the time, and “Italian” 30%.</p> <p>This probabilistic nature makes AI great for creative tasks. What is creativity but the ability to explore beyond the common possibilities, to think outside the box?</p> <p>However, this probabilistic nature also causes inconsistency and hallucinations. It’s fatal for tasks that depend on factuality. Recently, I went over 3 months’ worth of customer support requests of an AI startup I advise and found that ⅕ of the questions are because users don’t understand or don’t know how to work with this probabilistic nature.</p> <p>To und