Temperature: The One Number That Decides How Confident a Model Sounds

Copy a text-generation tutorial and you'll almost always find this line, unexplained:

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=1.0,
)

That's the standard Hugging Face transformers library convention for generating text — max_new_tokens just caps output length and isn't covered here; temperature is. Left at its default, tweaked without explanation, or blamed vaguely for a model being "too random" or "too repetitive" — most people setting this number couldn't say exactly what it does to the math underneath. Temperature doesn't pick a word. It reshapes the entire probability distribution the model is choosing from, before anything downstream ever runs. This post builds a tiny, fully verifiable version of that reshaping from scratch, so every number below is something you can rerun yourself, not a claim to take on faith.

What the Model Actually Outputs

At every step, a language model (LLM, short for large language model) doesn't output a word directly. It outputs a logit — a raw, unbounded score — for every entry in its vocabulary (the fixed list of words/tokens it's allowed to output), which can run into tens of thousands of candidates. A function called softmax turns those raw scores into a proper probability distribution: every value between 0 and 1, all of them summing to exactly 1.

To keep the numbers small enough to check by hand, this post uses a 12-token toy vocabulary (the full list of candidates the model is choosing between here) standing in for a model answering "How many days do I have to recover a deleted file?" — the correct answer is 30. A few distractors (wrong answers designed to look plausible) are mixed in: a wrong-but-plausible number belonging to a different rule entirely (14 — a refund window, not a recovery window), a wrong number tied to a different, enterprise-specific extension window (90), and three ungrounded words a model could drift toward if it stopped trusting the numbers at all (immediately, never, unsure). A real model's vocabulary works the same way, just with tens of thousands of tokens instead of 12. The logits themselves are made up by hand for this demo — but every probability and percentage computed from them below is real, run code, not an estimate.

How Temperature Reshapes the Distribution

Temperature is applied before softmax runs, dividing every logit by T:

softmax(logits / T)

Run the same logits through three temperatures and the correct answer's own probability moves from 76.69% at T=0.5, to 50.61% at T=1.0 (the untouched distribution), to 28.83% at T=2.0. The three ungrounded words combined tell the sharper story: their share of the total probability goes from 0.0052% at T=0.5, to 0.6733% at T=1.0, to 5.6448% at T=2.0 — over a thousand-fold increase (roughly 1,079 times), on identical logits.

Doesn't raising temperature just mean "more random," in some vague sense? Concretely, it means every candidate word that wasn't already the clear favorite gets a larger, calculable share of the total probability — including the ones you'd least want a factual system trusting. Divide by a number below 1 (T < 1) and the distribution sharpens: the leading token gets more likely, everything else shrinks. Divide by a number above 1 (T > 1) and it flattens: the tail gets a real, measurable chance it didn't have before. Temperature doesn't pick a word. It decides how much of the distribution's original confidence survives before the model picks one.

Get the Code

The full softmax and temperature code behind every number above is under 40 lines. Save it as temperature.py and run it with python temperature.py. It needs only numpy.

import numpy as np

VOCAB = ["30", "14", "90", "60", "45", "7", "10", "365", "0",
         "immediately", "never", "unsure"]

# Raw logits for: "How many days do I have to recover a deleted file?"
LOGITS = np.array([4.0, 3.2, 2.8, 1.5, 1.0, 0.6, 0.4, -0.5, -0.8, -1.0, -1.5, -2.0])
WRONG_TOKENS = ["immediately", "never", "unsure"]

def softmax(logits):
    shifted = logits - np.max(logits)  # subtract max for numerical stability
    exp = np.exp(shifted)
    return exp / exp.sum()

def apply_temperature(logits, T):
    return softmax(logits / T)

wrong_idx = [VOCAB.index(t) for t in WRONG_TOKENS]

probs = {T: apply_temperature(LOGITS, T) for T in (0.5, 1.0, 2.0)}
for T, p in probs.items():
    print(f"T={T}: P('30') = {p[0]:.4%}, ungrounded mass = {p[wrong_idx].sum():.4%}")

ratio = probs[2.0][wrong_idx].sum() / probs[0.5][wrong_idx].sum()
print(f"Ungrounded mass, T=2.0 vs T=0.5: {ratio:,.0f}x")

Running it prints:

T=0.5: P('30') = 76.6900%, ungrounded mass = 0.0052%
T=1.0: P('30') = 50.6141%, ungrounded mass = 0.6733%
T=2.0: P('30') = 28.8295%, ungrounded mass = 5.6448%
Ungrounded mass, T=2.0 vs T=0.5: 1,079x

Summary

Temperature answers one question — how much of a probability distribution's original confidence survives before a model turns it into one chosen word. On identical logits, T=0.5 dropped the combined probability of three ungrounded words to 0.0052% while the correct answer held 76.69%; T=2.0 raised that same ungrounded share to 5.6448% while the correct answer's own probability fell to 28.83%, a roughly 1,079x increase in how likely those ungrounded words become. It isn't a vague "randomness" dial; it's a specific, calculable reshaping of the distribution itself, applied before the model picks anything.