Decoding strategies: temperature, top-k, top-p

Suppose the next-token probabilities are 0.6, 0.3, 0.1. Greedy always chooses the first. Top-k with k=2 keeps the first two; top-p with p=0.8 also needs the first two to reach its probability-mass threshold. Sampling then uses the renormalized distribution. The model has not changed, but its selection rule has. The model only supplies a distribution; truncate the same distribution differently and the candidate pool changes.

The model outputs a probability distribution; the decoding strategy does the choosing

After each forward step, the model outputs a probability distribution over the vocabulary, for example:

" the"   0.42
" a"     0.18
" my"    0.09
" one"   0.05
...      (the other 50k tokens share the remaining 0.26)

With the weights fixed, the decoder can still use different rules to select from this distribution. Reproducing a generation therefore requires the model version and sampling settings; the answer text alone rarely explains why behavior changed.

Temperature: adjusting how steep the distribution is

$$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$

Usually, divide logits by $T$ before softmax. Multiplying probabilities by a common factor and renormalizing does nothing; raising them to the power $1/T$ and renormalizing is equivalent to this formula.

Positive temperature preserves the ranking but changes concentration. Lowering it does not add knowledge or guarantee a more correct answer.

The formula is undefined at $T=0$. Some interfaces use it as shorthand for greedy decoding; others have a separate switch. In Transformers 5.20.0 (checked October 2026), for example, do_sample=False, num_beams=1 explicitly selects greedy decoding.

Top-k: keep only the k most probable

Sort the distribution, keep the top $k$, set the rest to zero, renormalize, and then sample.

Keeping two tokens can retain very different amounts of probability mass. Consider two made-up distributions:

Top-k fixes the count, not the retained probability mass. It can work well, but the same k truncates different positions differently. Top-p changes that selection criterion. Ties in this table are broken by token ID.

Top-p (nucleus sampling): truncate by cumulative probability

Change the rule: accumulate probabilities from highest to lowest, take the smallest set whose cumulative probability just reaches ≥ p, discard the rest, and renormalize.

$$\text{keep the smallest set } V^{(p)} \text{ such that} \sum_{i \in V^{(p)}} p_i \ge p$$

With the same $p = 0.9$:

• when the distribution is steep, the first token alone may already have 0.93, and the candidate pool holds just 1 token;
• when the distribution is flat, it may take accumulating down to the 80th token to reach 0.9, and the candidate pool holds 80 tokens.

The candidate count adapts to the distribution, rather than being chosen in advance. But 0.9 means retained model probability mass, not a 90% chance of a correct answer. Whether top-p is preferable still depends on the task and observed outputs.

The two can be combined. For example, apply top-k, renormalize the survivors, then apply top-p. That top-p threshold now refers to the remaining distribution, not the original one. Check the configuration for which filters are enabled and in what order.

The usual processing order

Check the framework instead of treating one order as universal. This comparison explicitly chooses:

Temperature → TopK → TopP

Temperature comes first. This is not an inconsequential detail, because it changes how large the nucleus cut out by top-p is.

Positive temperature does not change the top-k candidate set: ranking is preserved, provided ties use the same rule. But top-p looks at cumulative probability, and temperature changes how quickly that mass accumulates.

Use probabilities 0.40, 0.25, 0.15, 0.08, 0.05, 0.03, 0.025, 0.015, convert them to logits, and set top_p=0.9:

With the order reversed, the same parameters give a different candidate pool. At $T=1$ temperature leaves the distribution unchanged, so the two are necessarily equal, so testing only at the default temperature will never reveal this difference.

Record temperature and top-p together. Fix the processing order before tuning so you can distinguish flattening the distribution from truncating its support.

Frequency penalty and presence penalty

Another family of methods uses generation history to lower logits for tokens that have appeared, also changing the final distribution:

$$z_i \leftarrow z_i - \alpha_{\text{presence}}\cdot\mathbb{1}[c_i > 0] - \alpha_{\text{frequency}}\cdot c_i$$

Here $c_i$ counts occurrences of token $i$ in the generated text. Assume nonnegative coefficients for this discussion; negative coefficients instead encourage previously used tokens. The difference is in the formula:

• Presence penalty: subtract a fixed amount once a token has appeared; 1 occurrence and 10 occurrences cost the same. Unused tokens gain a relative advantage.
• Frequency penalty: subtract in proportion to the accumulated count; the more often a token has appeared, the harder it is pushed down. Its effect is to suppress repetition.

These penalties track tokens, not topics. Suppressing previously used tokens does not guarantee new ideas; paraphrasing the same idea may evade the penalty.

⚠️ Both are blunt instruments, because some repetition is supposed to happen. Punctuation, pronouns, indentation and brackets in code, and a person's name that the text keeps mentioning are all meant to occur frequently. Turn the penalties up too far and the model starts avoiding these necessary tokens; the output becomes awkward or even ungrammatical. Code generation is especially intolerant of this.

Keep an unpenalized baseline and compare outputs on the same prompts. Check the framework's parameter ranges and whether counts include the prompt; there is no universal step size or useful maximum across models.

Beam search: higher probability need not mean a better answer

Beam search expands candidates and retains $k$ by score at each step. The simplest score is cumulative log-probability; implementations may add length penalties. The final choice is the best surviving candidate, not necessarily the global optimum.

Consider two steps: A has probability 0.6 and B has 0.4. The best continuation after A has probability 0.5; after B it has 0.9. Greedy commits to A and reaches 0.30. A width-2 beam can retain B and find 0.36. Keeping several prefixes helps, but a pruned prefix is usually gone for good.

Translation and speech recognition often use beam search, not because answers are unique, but because searching by model score can be useful. Score, length handling, and task quality still need to agree. In open-ended generation, likelihood maximization can produce repetition; see Neural Text Degeneration.

Math does not require greedy decoding either. Self-consistency samples multiple reasoning paths and aggregates answers; parallel sampling and verification covers how to split the budget and what to verify with. Choose a strategy together with its candidate budget, verification method, and final metric.

How to set the parameters in practice

A few easy traps:

Low temperature plus low top-p can leave just one candidate. Inspect the actual support if you want diversity. This need not be a problem when deterministic output is the goal, but enabling sampling does not by itself ensure varied outputs.

Distinguish the target policy from the actual rollout distribution. Temperature and truncation can change that distribution, so specify which probabilities are recorded and whether correction is applied. Truncation also removes support: adding an importance ratio cannot recover actions that were never sampleable.

To reproduce a result, fix the random seed and record every decoding parameter. Recording only the model version is not enough — the same weights with T=0.7 and with T=1.0 are two different behaviors.

A top-p implementation with testable boundaries

Let vocabulary IDs 0, 1, 2 have probabilities 0.1, 0.6, 0.3, with threshold 0.8. Sorting selects ID 1, then ID 2: cumulative mass rises from 0.6 to 0.9. Keep the token that crosses the threshold. Renormalizing gives probabilities 2/3 and 1/3.

This function computes the final distribution without drawing a random sample, making the result easy to check. It accepts finite logits and is a teaching implementation, not a GPU kernel.

import math

def nucleus_distribution(logits, threshold=0.9, temperature=1.0):
    if not logits or not 0 < threshold <= 1 or temperature <= 0:
        raise ValueError("Expected logits, 0 < threshold <= 1, and temperature > 0")
    if not math.isfinite(temperature) or not all(math.isfinite(value) for value in logits):
        raise ValueError("Expected finite inputs")
    if temperature >= 1:
        scaled_logits = [value / temperature for value in logits]
    else:
        maximum = max(logits)
        scaled_logits = [(value - maximum) / temperature for value in logits]
    maximum = max(scaled_logits)
    weights = [math.exp(value - maximum) for value in scaled_logits]
    total = sum(weights)
    probabilities = [weight / total for weight in weights]
    ordered_ids = sorted(range(len(logits)), key=lambda token_id: probabilities[token_id], reverse=True)
    selected_ids = []
    cumulative = 0.0
    for token_id in ordered_ids:
        selected_ids.append(token_id)
        cumulative += probabilities[token_id]
        if threshold < 1 and cumulative >= threshold:
            break
    return {token_id: probabilities[token_id] / cumulative for token_id in selected_ids}

example = nucleus_distribution([math.log(.1), math.log(.6), math.log(.3)], .8)
assert list(example) == [1, 2]
assert math.isclose(example[1], 2 / 3)

Return original vocabulary IDs, not sorted positions. Also test threshold=1 (no truncation), a single candidate, tied maxima, and large positive or negative logits. One convenient example is not enough.

Scaling order has numerical boundaries too. Logits [1e308, -1e308] at T=1e308 should give roughly [0.881, 0.119]. Subtracting the logits first overflows before temperature can reduce the difference. The code scales first at high temperatures; at low temperatures, it centers first to avoid overflowing a large common offset divided by a small temperature.

Common interview questions

Self-check

Where to read next

• Decoder-only: autoregressive generation: how these distributions are produced step by step
• On-policy and off-policy: how temperature and top-p change the rollout behavior distribution, and which log-probs to record

Quick learning: model probabilities and the decoding policy are two layers