The model gives chances. Something picks.
A language model doesn’t write a sentence; it scores every token it knows as the next one. A decoding rule then picks one, and repeats. Temperature, top-k and top-p are the dials on that rule.
- Turn a model’s scores into chances
- Explain greedy decoding, temperature, top-k and top-p
- Say why the same prompt can give different answers
You need: Lesson 1, Tokens.
-
A chance for every token
Predict first
After “On a rainy evening, there is nothing better than a hot cup of”, which word does Qwen 3 (0.6B) rate likeliest?
Just show me
“tea”, at 54.4%; “coffee” 33%; “soup” 2.9%.
For every step, the model gives one score (a logit) to each of the 1,51,936 tokens in its vocabulary. Softmax turns the scores into chances that add up to 1. These are Qwen 3 (0.6B)’s own chances, recorded at full precision for our temperature playground.
·tea54.4%·coffee33%·soup2.9%·steam1.7%·water0.8%·hot0.7%The next token’s chances, at temperature 1
Quick check
What does softmax do here?
It only converts; picking is a separate step.
-
Greedy: the top token, every time
Predict first
Always taking the likeliest token: what kind of text do you get?
Just show me
The same every time. Here it’s:
On a rainy evening, there is nothing better than a hot cup of tea. The problem is that the tea is not coming out of the kettle.
Greedy decoding is predictable, which suits facts, code and extraction. It can also be flat and repetitive, because the safest token is rarely the most interesting one.
Quick check
You’re extracting dates from invoices. Greedy or sampling?
For one right answer, take the model’s best guess each time.
-
Temperature reshapes the chances
Predict first
Raise the temperature from 1 to 1.5. What happens to “tea”’s chance?
Just show me
It falls from 54.4% to 22.6%; lower it to 0.5 and tea rises to 72.8%.
Temperature divides every score before softmax. Below 1 the chances sharpen toward the favourite; above 1 they flatten and unlikely tokens get a real chance. It changes how adventurous the picks are, not what the model knows.
·tea72.8%·coffee26.9%·soup0.2%·steam0.1%·water0.0%·hot0.0%·tea54.4%·coffee33.0%·soup2.9%·steam1.7%·water0.8%·hot0.7%·tea22.6%·coffee16.2%·soup3.2%·steam2.3%·water1.4%·hot1.3%The same six tokens’ chances at each temperature, worked out from the model’s own scores
Move temperature, top-k and top-p in the playground
Quick check
A brainstorming assistant gives dull ideas. Which dial?
A bit more temperature lets less obvious tokens through.
-
Top-k and top-p cut the tail
Predict first
With top-p set to 0.9, how many tokens stay in the running here?
Just show me
3: “tea”, “coffee” and “soup” already add up to more than 90%. At top-p 0.5, only “tea” stays.
Top-k keeps the k likeliest tokens. Top-p (nucleus sampling) keeps the smallest group whose chances add up to p, so it keeps few tokens when the model is sure and many when it isn’t. Both stop the long tail of odd tokens from ever being drawn.
Quick check
Why does top-p adapt better than a fixed top-k?
The cut follows the shape of the chances, step by step.
-
Same prompt, different answers
Predict first
Sample four times at temperature 1. Same four answers?
Just show me
No. Here are Qwen 3 (0.6B)’s four, recorded with a seeded draw:
- …cup of tea. Today, the humid temperature range is from 80°F to 92°F at a location.
- …cup of tea, a comfortable pace, and the quiet and peacefulness of the home environment.
- …cup of tea, and weighing 50 grams for a normal-sized cup is optional, but a liter – you need to cook
- …cup of K-cup coffee, or a stop on the Baffin Bridge.
Each step draws a token by its chances, so one different early pick changes everything after it. For repeatable output, fix the seed or set temperature to 0, and even then batching and hardware can shift tiny numbers.
Quick check
Temperature is 0, yet answers still differ slightly across runs. A likely reason?
Floating-point sums in different orders can flip a near tie.
Lesson 4 done.
Next: Context and the KV cache. Everything a model works with has to fit in its context window, and every token in that window takes memory while it writes.
In the interview: picking the next token.
EasyWhat do temperature, top-k and top-p do?
MediumImplement temperature and top-p (nucleus) sampling for one decoding step.
MediumTemperature is set to 0, yet the same prompt gives different answers. Why?
EasyGreedy decoding, beam search or sampling: how do they differ, and when would you use each?
Ask Prism about this lesson
Sources
- Holtzman and others, “The Curious Case of Neural Text Degeneration”, ICLR 2020: nucleus (top-p) sampling.
- Qwen, Qwen 3 (0.6B), Apache 2.0: the chances, recorded at full precision on 1 Nov 2026.