Lesson 16 — cost has to be measured per task, not per token. A model that’s cheap-per-token but retries ten percent of the time is not actually cheap. This is the calculator that makes the unit economics visible before you ship. ## Why the rate-card lies Every model provider quotes price-per-million-tokens. `gpt-4o-mini` is $0.15 in / $0.60 out. `gpt-4o` is $2.50 in / $10 out. Self-hosted Llama-3-8B has no per-token rate at all — just the hourly cost of the GPU you parked it on. These are the rate-cards. The rate-card is not the per-task cost. Real production traffic has four factors that bend the curve, and three of them get worse as you scale: retry rate (calls that fail their eval gate and re-run), fallback rate (escalations to a premium model when the primary refuses), judge sampling (the eval pipeline’s recurring cost), and the reranker’s latency tax (no API cost but slower → fewer requests per second per worker). The calculator below makes all four visible. > A cheap-per-token model that retries 10% of the time is 10% more expensive than its rate-card claims. A premium fallback model used on 5% of requests adds the entire premium model’s cost on top of the primary. A 25% judge-sampling rate adds a quarter of an eval call to every task. These are not edge cases — they are the default shape of production RAG. ## The calculator Set the daily volume, drag the three sliders, toggle the reranker. The three model columns recompute on every tick. The cheapest column gets a Marker Teal border; self-hosted gets a dashed top border to flag the hardware-floor shape. cost_inputs.json missing ## The four levers - **Reranker latency tax.** The reranker toggle is not “no cost, just slow”. The ~650 ms p50 latency from lesson 07 means each worker serves fewer requests per second; the same daily volume now needs more worker-time. We model this as a flat $0.0007 per task at a $0.001/sec worker rate, applied across every column. The toggle actually moves the curve, not just the latency budget. - **Volume.** Daily request count drives whether the self-hosted model’s hardware floor amortises. At 100 requests per day, $28.80 of daily GPU cost is $0.288 per request — uncompetitive. At 100,000 requests per day, the same hardware is $0.000288 per request — half a milli-cent. The hardware floor pays back at scale; below that scale, pay-per-token wins. - **Retry rate.** Drag the retry slider up to 10%. Every column’s “effective per task” number rises by 10%. Retries are the most under-modelled cost in early-stage AI builds — teams budget for the happy path and discover the retry tax in production. The eval pipeline from lessons 10–13 is what tells you the real retry rate *before* the bill arrives. - **Fallback rate.** Drag the fallback slider up to 5%. The `gpt-4o` column’s effective cost barely changes — it’s already the premium model. The `gpt-4o-mini` column jumps significantly because each fallback adds a full `gpt-4o` call ($0.0035) on top of the `$0.00018` base. Fallback rate is where rate-card economics most diverge from production economics. - **Judge sampling.** Drag judge sampling up to 100%. Every response now also pays for a `gpt-4o-mini` judge call (~$0.00007). At 25% sampling it’s a quarter of that. The eval pipeline from lessons 10–13 has a recurring cost that scales with traffic, and the architect’s call is what fraction of production traffic to score — too low and you don’t catch regressions, too high and you’re double-paying for every request. ## The crossover point Switch the volume from 100/day up through 100k/day with the other levers at their defaults. Watch which column wins at each volume. - At **100/day**, `gpt-4o-mini` wins easily. Self-hosted’s hardware floor is $28.80/day spread over 100 requests — about 1,500× more expensive than `gpt-4o-mini` per task. - At **1k/day**, `gpt-4o-mini` still wins. Self-hosted is $0.0288/task vs `gpt-4o-mini`‘s ~$0.0002/task — still about 150× more expensive. - At **10k/day**, the gap narrows. Self-hosted is $0.00288/task — now about 15× more expensive than `gpt-4o-mini`. Sovereignty starts paying for itself in this volume band. - At **100k/day**, the curve bends. Self-hosted falls to $0.000288/task; `gpt-4o-mini` at the rate-card is $0.00018/task — they cross over depending on retry and judge sampling. For L3+ workloads that can’t leave the country, the self-hosted column is the only column anyway. ## Margin at the bottom The bottom row of each card shows margin at an indicative €0.50 per request — a placeholder Tier-2 price. Green margin means the column is profitable at that price; red means the column would lose money. Drag the sliders enough and you can put any column into red. For Tier-2 engagements the architect’s job is to set this price defensibly: cost-plus-margin works if the eval pipeline is honest about retry and judge rates; flat-rate-per-request only works when the variance is well-understood. The calculator is the conversation that picks the model. ## What this lesson doesn’t say - **It doesn’t say self-hosted’s full cost is in the model.** The hardware floor ($28.80/day) and the operational overhead ($30/day for sysadmin / on-call / patching / procurement) are both modelled. What is NOT modelled: the cost of the architect designing the deployment, hardware lead-time risk, GPU availability constraints, and the security audit overhead specific to running EU-AI-Act-graded sovereign infrastructure. The calculator measures the unit economics; the total cost of ownership is a wider conversation. - **It doesn’t say `gpt-4o-mini` is “the cheap option”.** It’s the cheapest API-rate-card option in this comparison. A team running 1k requests/day with low retry/fallback/judge rates absolutely should default to it. A team running 100k requests/day across L3 workloads has no choice but self-hosted. The calculator doesn’t recommend; it makes the trade-off visible. - **It doesn’t say €0.50 per request is the right price.** Real engagements set their price per engagement based on procurement, competitive context, and risk appetite. The figure is a teaching scale — it lets the margin row mean something. ## Next: security One last lesson closes the Production Architecture track. Lessons 14, 15, and 16 cover what the system *is*, how to *see* it, and what it *costs*. Lesson 17 covers the failure mode the other three don’t surface: a poisoned document that gets retrieved into the prompt and tells the model to ignore everything it was instructed to do. Indirect prompt injection — same Maltese corpus, one bad chunk, and a 30-line validator that catches it.