Lesson 04 — the single most operationally important fact about LLM serving. Most teams discover this the expensive way.

## Why caching K and V matters

To generate token N, self-attention has to read the keys and values of every previous token. Token N+1 needs the same plus one more. Without caching, position N would re-compute K and V for positions 1..N-1. That’s O(N²) work just for attention.

The fix is obvious: cache K and V once per position, then keep them. The cost is memory. The size of that memory is the question this lesson exists to answer.

## The formula

For one model under one config, the KV cache consumes:

```
KV bytes = 2 × batch × seq_len × layers × kv_heads × head_dim × bytes_per_element
```

- The **2** is for K and V (separately).
- **kv_heads** is critical: in GQA models like Llama-3, this is much smaller than query heads. Llama-3 uses 8 KV heads against 32 query heads — a 4× saving over MHA.
- **bytes_per_element** reflects the dtype: 2 for fp16/bf16, 1 for fp8, 0.5 for 4-bit quantised KV.
- **seq_len** is the longest sequence in the batch. The cache scales linearly with it.

## Try it on real models

Pick a model, dial up batch size and context length, and watch what fits where. Everything is computed live in the browser from the formula above. The colored bar turns dark orange when the KV cache alone exceeds the GPU’s total memory — and you still need room for the weights and activations.

 KV cache memory `2 × batch × seq_len × layers × kv_heads × head_dim × bytes_per_element`   Model  Llama 3 8B Llama 3 70B Mistral 7B Gemma 2 9B     Batch size 1  Concurrent sequences in one forward pass.   Context length   Tokens stored in the cache per sequence.   Dtype  fp16 — 2 B / element fp8 — 1 B / element int4 — 0.5 B / element  Lower precision = smaller cache.      vs. GPU memory  RTX 4090 (24 GB)      / 24 GB   A100 40 GB      / 40 GB   A100 80 GB      / 80 GB   H100 80 GB      / 80 GB  

## Things you’ll notice

- **Context length is the dominant multiplier.** Going from 8k to 128k context multiplies the cache by 16. Long context isn’t free, even when the model “supports” it.
- **Batch size is the second-most powerful lever.** Concurrent users multiply KV linearly. A model that fits at batch 1 may not at batch 16.
- **GQA is doing a lot of work.** Compare Llama-3-70B (8 KV heads) to a hypothetical MHA-70B (it would need 4× the cache for the same context). GQA is why long-context serving on commodity hardware is feasible at all.
- **Quantised KV is the cheap win.** Switching from fp16 to fp8 halves the cache for usually-negligible quality loss. Int4 quarters it. vLLM, TensorRT-LLM, and most modern serving stacks support this.

## Architect-grade implications

1. **“Can we run this on one A100?” depends on context × batch, not just model size.** An 8B model can fit happily on a 24GB consumer GPU at small batch and short context, and need an A100 once you turn the dials up. The model card alone won’t tell you.
2. **128k context windows are marketing-true and serving-false.** Yes, the model accepts 128k tokens. No, you cannot serve 50 concurrent users at 128k each on commodity hardware. Long-context features cost more than they look.
3. **If you’re hitting OOM, the answer is rarely “bigger GPU”.** First lever: shorter context. Second: lower KV precision. Third: smaller batch. Fourth: paged attention (vLLM-style) which lets the cache live in non-contiguous blocks and trades a small overhead for much better utilisation.

---

*Model specs come from published configs. The calculator gives the theoretical lower bound on KV memory; real serving stacks add overhead for prefetch, padding, and paged-attention bookkeeping — usually 10–25% on top.*