Skip to content
AI360XpertBeta

Temperature Sampling

Temperature is the knob that sets how wildly the model draws its next token. Turn it down for steady answers you can rerun, turn it up when you want fresh options and can live with misses.

Dividing logits [2, 1, 0] by T = 2 before softmax softens the odds from 0.665, 0.245 and 0.090 to 0.506, 0.307 and 0.186
Dividing logits [2, 1, 0] by T = 2 before softmax softens the odds from 0.665, 0.245 and 0.090 to 0.506, 0.307 and 0.186

Why Does This Exist?

A trained model never hands you a word. It hands you odds: one score per token across tens of thousands of candidates. Next-token prediction is where those scores come from. The question on this page is narrower: once the scores exist, how wildly should you draw from them?

The model's native odds carry one fixed appetite for risk. Draw from them as they are and you get one personality forever. That single personality is wrong at both extremes. A strict extraction job wants the same answer on every run, while a brainstorming session wants fresh options. Neither wish changes the weights, and retraining for each mood would be absurd. You need a knob that reshapes the odds at draw time.

That knob is temperature, a single positive number that scales the scores before they become probabilities. This page covers that reshape rule and its arithmetic. The full menu of picking rules lives in decoding strategies. The cutoffs that delete the tail before the draw live in top-p sampling and top-k sampling.

One idea first: raw model outputs are logits, plain scores that can be negative and sum to nothing in particular. Softmax turns them into probabilities that sum to 1: exponentiate each score, add everything up, divide each by the total. Temperature slips one division in before that step, and everything below follows from it.

Think of It Like This

A raffle drum with a contrast knob

Your club raffle has three prizes left. The drum holds 665 tickets for the headphones, 245 for the mug, and 90 for the mystery box. You draw one ticket and award that prize.

Now someone bolts a contrast knob onto the drum. Turn it toward cold and the crew quietly multiplies every gap between the piles: the headphones pile towers, the other two shrink to a whisper, and you will draw headphones all night. Turn it toward hot and the crew evens the piles out: the mystery box suddenly holds a real share of the drum, and strange results start coming up.

Point for point: tickets are the scores, the knob is temperature, the draw is sampling. And the knob never throws tickets away, it only resizes the piles. Removing hopeless tickets entirely is a different tool, the cutoff rules, not this one.

It stops here: drum tickets sit still between draws, but every token you pick rewrites the odds for the next one. And a drum is fair by construction while model scores can be confidently wrong, so a sharp setting on bad scores just repeats the same mistake.

How It Actually Works

Write the candidate scores as z1,z2,…,zVz_1, z_2, \ldots, z_V. Here ziz_i is the logit for token ii, VV is the vocabulary size, and TT is the temperature, a positive number that defaults to 1. The probability of token ii is

pi=exp⁡(zi/T)∑jexp⁡(zj/T)p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

In plain words: divide every score by TT first then run the usual softmax. The symbol exp⁡\exp is the exponential function, and the sum runs over every candidate token jj, so the pip_i values sum to 1. That is the whole mechanism. One division moves mass around; nothing else changes.

Cold, neutral, and hot

Cold means TT below 1, and it sharpens. Dividing by 0.5 doubles every score, which stretches the gaps between them, and the exponential turns stretched gaps into a near monopoly for the leader. You will mostly draw the favourite, with reruns that look nearly identical.

Neutral means T=1T = 1, which leaves the scores untouched. You draw exactly what the model learned, favourite leading, tail trailing, no thumb on the scale.

Hot means TT above 1, and it flattens. Dividing by 2 halves every score, which squeezes the gaps toward zero, and the exponential turns squeezed gaps into closer odds. The tail gains real share. Push TT very high and the odds slide toward uniform, where every token draws about equally and the text falls apart.

Two limits fix the picture. As TT approaches 0, the rule approaches greedy picking: the top token takes everything. As TT grows large, it approaches a uniform draw. In practice most calls stay near 1, with factual jobs dialled down and open ended writing dialled up.

Worked example: logits [2, 1, 0]

Call the three candidates cat, dog, and bird, with logits z=[2,1,0]z = [2, 1, 0]. At T=1T = 1 there is no scaling. Exponentiate: [exp⁡(2),exp⁡(1),exp⁡(0)]=[7.389,2.718,1.000][\exp(2), \exp(1), \exp(0)] = [7.389, 2.718, 1.000]. Sum: 11.10711.107. Divide each by the sum:

p=[0.665, 0.245, 0.090],sum 1.000p = [0.665,\ 0.245,\ 0.090], \qquad \text{sum } 1.000

Cat holds two thirds of the mass, dog about a quarter, bird about one draw in eleven.

Now set T=2T = 2. Divide first: [1.0,0.5,0.0][1.0, 0.5, 0.0]. Exponentiate: [2.718,1.649,1.000][2.718, 1.649, 1.000]. Sum: 5.3675.367. Divide:

q=[0.506, 0.307, 0.186],sum 1.000q = [0.506,\ 0.307,\ 0.186], \qquad \text{sum } 1.000

Same ranking, softer gaps. Cat falls from 0.665 to 0.506 while bird more than doubles from 0.090 to 0.186, so roughly one draw in five now goes to the outsider. For contrast, T=0.5T = 0.5 doubles the scores to [4,2,0][4, 2, 0], giving [0.867,0.117,0.016][0.867, 0.117, 0.016]: cat nearly locked in, bird nearly gone. Those T=1T = 1 and T=2T = 2 bars are the two panels in the diagram above, and the snippet below reprints all three rows.

The trade: variety for repeatability

Temperature buys variety with repeatability, and the price is paid in errors. Low values give stable output you can test and grade, but the text turns flat and loops on open ended prompts. High values break loops and surprise you, but surprising draws include bad ones, and long generations give the tail many chances to strike.

So do not use it where the answer must be stable. Graded extraction, math with a checked answer, and tool calls with strict schemas want the top token every time, or better still constrained decoding, which forbids illegal tokens outright. Save high temperatures for drafting and brainstorming, pair them with a cutoff, and keep them away from anything you grade.

The rule rests on one assumption: higher probability means more desirable. That holds when the model is well calibrated and the task rewards the likely answer. It fails when safe phrasing outscores good phrasing, or when the scores themselves are wrong. Temperature cannot fix bad scores. It only decides how faithfully you obey them.

Code

The snippet reprints the worked example, so you can check the arithmetic instead of trusting it.

import numpy as np

def temperature_softmax(logits: np.ndarray, temperature: float) -> np.ndarray:    scaled = logits / temperature  # the whole trick: divide first    e = np.exp(scaled - scaled.max())  # shift so the largest input to exp is 0    return e / e.sum()

z = np.array([2.0, 1.0, 0.0])  # logits for cat, dog, birdp = temperature_softmax(z, 1.0)print([f"{v:.3f}" for v in p])  # -> ['0.665', '0.245', '0.090']q = temperature_softmax(z, 2.0)print([f"{v:.3f}" for v in q])  # -> ['0.506', '0.307', '0.186']r = temperature_softmax(z, 0.5)print([f"{v:.3f}" for v in r])  # -> ['0.867', '0.117', '0.016']

The max subtraction changes nothing mathematically and saves you from overflow on large scores. The three printed rows match the hand computation: neutral odds, flattened odds, sharpened odds.

Watch Out For

Raising temperature without a cutoff invites tail junk

High temperature feels creative until a one in eleven word starts landing every few sentences. The symptom is rare fluent nonsense: nine good generations, then bird in a paragraph about cats and dogs. In the example above T=2T = 2 lifts bird from 0.090 to 0.186, so the weird draw stops being rare.

Cap the tail before you draw. Set top-p around 0.9 or top-k to a small count so hopeless candidates hold zero mass, as top-p sampling shows, and keep temperature moderate on anything factual. You will keep the variety and lose most of the gibberish.

Passing temperature 0 straight into the formula

Zero sounds like the deterministic setting, but the rule divides by TT, so z/0z / 0 is undefined, not clever. The symptom is an API error, a silent clamp to some tiny value, or code that runs on one provider and throws on the next.

Branch instead. When you want the top token every time, call the argmax path directly, the greedy rule in decoding strategies, and validate that TT stays above 0 wherever sampling runs. Determinism deserves its own code path, not an edge case of a division.

The Quick Version

  • Temperature divides every logit by TT before softmax, so values below 1 sharpen the favourite and values above 1 spread mass toward the tail.
  • With logits [2, 1, 0], T=1T = 1 gives odds of 0.665, 0.245 and 0.090, while T=2T = 2 softens them to 0.506, 0.307 and 0.186.
  • As TT approaches 0 the rule approaches greedy picking, and as TT grows large it approaches a uniform draw over the vocabulary.
  • Temperature never deletes a candidate, it only resizes shares, so pair high values with top-p or top-k to keep the hopeless tail at zero.
  • Skip sampling where answers must be stable, such as graded extraction or strict tool calls, and use greedy or constrained decoding there instead.
  • The knob assumes higher probability means more desirable, so on miscalibrated scores a cold setting repeats the same mistake with confidence.