How a model picks its next word.

A language model scores every token it knows, then a few settings decide which one it says. Move them and watch real chances from Qwen 3, an open model, change.

Guess first

After “On a rainy evening, there is nothing better than a hot cup of”, Qwen 3 gives “tea” a 54% chance. Turn the temperature up from 1 to 2. Does that chance go up or down?

Pick a prompt

On a rainy evening, there is nothing better than a hot cup of

Divides every score before it becomes a chance. Lower: surer. Higher: more surprising.
Keeps only the k likeliest tokens. At the right end, it’s off.
Keeps the likeliest tokens until their chances add up to p.

The next token’s chances

Every token can still be drawn: the likeliest sixty and 1,51,876 others.

tea54%
coffee33%
soup2.9%
steam1.7%
water0.8%
hot0.7%
instant0.5%
java0.4%
ste0.3%
0.3%
cocoa0.2%
drink0.1%
The rest4.8%

· marks a space: ·tea is “tea” with a space before it, as the model writes it.

Chances: Qwen 3 (0.6B), from Alibaba (Apache 2.0), run at full precision on 1 Nov 2026: each prompt’s 60 likeliest next tokens, the rest’s exact share at each temperature, and continuations it wrote at temperature 0, 0.7, 1 and 1.5 (each drawn from its whole vocabulary). The playground applies temperature, then top-k, then top-p, as Hugging Face’s and vLLM’s samplers do.

Three settings, one draw.

  • Temperature divides the scores

    The model gives every token a score. Each score is divided by the temperature, then softmax turns them into chances that add up to 100%. Below 1 the likeliest token gains; above 1 the chances even out; at 0 the likeliest token is picked every time.

  • Top-k keeps the k likeliest

    Every token outside the k likeliest gets no chance, and the chances left are scaled back up to 100%. Fan, Lewis and Dauphin sampled from the top 10 to write stories (2018).

  • Top-p keeps the likeliest share

    Called nucleus sampling: keep the likeliest tokens until their chances add up to p, say 0.9, and drop the long tail of unlikely ones. Holtzman and colleagues showed it writes more natural text than always taking the likeliest token (2020).

In the APIs: the newest models choose for you

These settings still shape every open model and every model you run yourself. The newest closed models are taking them out of your hands (read 1 Nov 2026):

  • OpenAI. Temperature runs from 0 to 2 and top_p from 0 to 1, both 1 by default, and OpenAI advises changing one, not both. On its newest reasoning models they work only with reasoning effort set to none, and older GPT-5 models refuse them.
  • Anthropic. Claude’s API version of 21 Oct 2026 removed temperature, top_p and top_k: a request that sends any of them gets an error, even at the default value.
  • Google. Gemini’s API takes a temperature from 0 to 2, but Gemini 3.6 Flash and newer ignore it, and top-p and top-k, without a word. For Gemini 3, Google advised keeping the default, 1.0.

Sources

  1. Qwen/Qwen3-0.6B and its ONNX copy, onnx-community/Qwen3-0.6B-ONNX at revision 58ca03a (Apache 2.0): the chances and continuations here.
  2. Holtzman and others, “The Curious Case of Neural Text Degeneration”, ICLR 2020: top-p, or nucleus, sampling.
  3. Fan, Lewis and Dauphin, “Hierarchical Neural Story Generation”, ACL 2018: top-k sampling.
  4. OpenAI, Chat Completions reference: temperature and top_p; and its latest-model guide: “When reasoning effort is not none, remove temperature, top_p, and top_logprobs” (read 1 Nov 2026).
  5. Anthropic, migrating to API version 2026-10-21: the sampling parameters are removed (read 1 Nov 2026).
  6. Google, Gemini API: GenerationConfig, and its Gemini 3 guide: keep the temperature at 1.0 (read 1 Nov 2026).