Lesson 03 — the difference between “the model said it” and “the model could have said it”. Sampling is where probability becomes language.

## From logits to a chosen token

After the last block of the transformer, the model produces a vector of **logits** — one real number per token in the vocabulary. They’re unbounded. They aren’t probabilities yet. The path from this vector to a single chosen token has three knobs:

- **Temperature** rescales the logits before softmax. `softmax(logits / T)`. `T → 0` sharpens (the top token dominates). `T → ∞` flattens (everything becomes uniform). `T = 1` leaves the distribution untouched.
- **top-k** keeps only the k highest-probability tokens and zeros the rest. Then renormalises and samples within that set. Small k means safer, more predictable output.
- **top-p** (nucleus) sorts by probability descending and keeps the smallest set whose cumulative probability reaches p. Adapts the size of the sampling pool to how confident the model is.

top-k and top-p are usually combined: top-k as a hard ceiling, top-p as the adaptive floor. Both apply *after* temperature.

## Try it

The prompt is `"The cat sat on the"`. Logits are pre-baked from a plausible LLM output. Move each knob and watch the distribution change. Click *Draw a sample* to commit to one of the eligible tokens.

 Prompt `The cat sat on the ▍`   Temperature   0.1 sharpens. 2.0 flattens.   top-k 12  Keep only the k highest-probability tokens.   top-p   Keep the smallest set whose probability sums to p.     Draw a sample Reset to defaults Clear draws  Last 10 draws   

## What you’ll feel

A few things become obvious as you move the sliders:

- **Low temperature looks deterministic but isn’t.** At T ≈ 0.3 the top token usually wins, but there’s still a small chance of an alternative. Production systems sometimes use greedy sampling (literally argmax) for true determinism.
- **High temperature is where hallucinations live.** When the model is uncertain and you flatten the distribution, low-probability tokens get drawn. The same model that “knew” the answer becomes the one making things up.
- **top-p is the safer default for general text.** It adapts: if the model is confident, the pool is small; if not, it widens. top-k alone can be too restrictive in confident moments and too permissive in uncertain ones.
- **Filtering changes which token gets sampled, not the model’s beliefs.** Excluded tokens are still in the model’s “head” — you’ve just decided not to consider them.

## Architect-grade implications

1. **“Temperature 0” doesn’t fix hallucinations.** Greedy sampling makes the model deterministic, not correct. If the highest-probability token is wrong, lowering temperature just guarantees you get the wrong answer every time.
2. **Reproducibility requires temperature 0 *and* a fixed seed if the platform offers one.** Most hosted APIs only guarantee determinism on rare paths; treat reproducibility as a tested behaviour, not an assumed one.
3. **Sampling parameters belong in evaluation runs.** If you tune temperature/top-p without re-running your evals, you’ve changed the system without measuring. The same logits, different sampling → different model behaviour.

---

*The 12 candidate tokens above are illustrative, not from a real model run. In production the softmax runs over the full vocabulary (≈128k tokens for Llama-3); the long tail is what filters actually clip away. Tokenisation runs entirely in your browser.*