Lesson 03 — the difference between “the model said it” and “the model could have said it”. Sampling is where probability becomes language.
From logits to a chosen token
After the last block of the transformer, the model produces a vector of logits — one real number per token in the vocabulary. They’re unbounded. They aren’t probabilities yet. The path from this vector to a single chosen token has three knobs:
- Temperature rescales the logits before softmax.
softmax(logits / T).T → 0sharpens (the top token dominates).T → ∞flattens (everything becomes uniform).T = 1leaves the distribution untouched. - top-k keeps only the k highest-probability tokens and zeros the rest. Then renormalises and samples within that set. Small k means safer, more predictable output.
- top-p (nucleus) sorts by probability descending and keeps the smallest set whose cumulative probability reaches p. Adapts the size of the sampling pool to how confident the model is.
top-k and top-p are usually combined: top-k as a hard ceiling, top-p as the adaptive floor. Both apply after temperature.
Try it
The prompt is "The cat sat on the". Logits are pre-baked from a plausible LLM output. Move each knob and watch the distribution change. Click Draw a sample to commit to one of the eligible tokens.
The cat sat on the ▍
What you’ll feel
A few things become obvious as you move the sliders:
- Low temperature looks deterministic but isn’t. At T ≈ 0.3 the top token usually wins, but there’s still a small chance of an alternative. Production systems sometimes use greedy sampling (literally argmax) for true determinism.
- High temperature is where hallucinations live. When the model is uncertain and you flatten the distribution, low-probability tokens get drawn. The same model that “knew” the answer becomes the one making things up.
- top-p is the safer default for general text. It adapts: if the model is confident, the pool is small; if not, it widens. top-k alone can be too restrictive in confident moments and too permissive in uncertain ones.
- Filtering changes which token gets sampled, not the model’s beliefs. Excluded tokens are still in the model’s “head” — you’ve just decided not to consider them.
Architect-grade implications
- “Temperature 0” doesn’t fix hallucinations. Greedy sampling makes the model deterministic, not correct. If the highest-probability token is wrong, lowering temperature just guarantees you get the wrong answer every time.
- Reproducibility requires temperature 0 and a fixed seed if the platform offers one. Most hosted APIs only guarantee determinism on rare paths; treat reproducibility as a tested behaviour, not an assumed one.
- Sampling parameters belong in evaluation runs. If you tune temperature/top-p without re-running your evals, you’ve changed the system without measuring. The same logits, different sampling → different model behaviour.
The 12 candidate tokens above are illustrative, not from a real model run. In production the softmax runs over the full vocabulary (≈128k tokens for Llama-3); the long tail is what filters actually clip away. Tokenisation runs entirely in your browser.