·

Cost: per task, not per token

5 min read

Lesson 16 — cost has to be measured per task, not per token. A model that’s cheap-per-token but retries ten percent of the time is not actually cheap. This is the calculator that makes the unit economics visible before you ship.

Why the rate-card lies

Every model provider quotes price-per-million-tokens. gpt-4o-mini is $0.15 in / $0.60 out. gpt-4o is $2.50 in / $10 out. Self-hosted Llama-3-8B has no per-token rate at all — just the hourly cost of the GPU you parked it on. These are the rate-cards.

The rate-card is not the per-task cost. Real production traffic has four factors that bend the curve, and three of them get worse as you scale: retry rate (calls that fail their eval gate and re-run), fallback rate (escalations to a premium model when the primary refuses), judge sampling (the eval pipeline’s recurring cost), and the reranker’s latency tax (no API cost but slower → fewer requests per second per worker). The calculator below makes all four visible.

A cheap-per-token model that retries 10% of the time is 10% more expensive than its rate-card claims. A premium fallback model used on 5% of requests adds the entire premium model’s cost on top of the primary. A 25% judge-sampling rate adds a quarter of an eval call to every task. These are not edge cases — they are the default shape of production RAG.

The calculator

Set the daily volume, drag the three sliders, toggle the reranker. The three model columns recompute on every tick. The cheapest column gets a Marker Teal border; self-hosted gets a dashed top border to flag the hardware-floor shape.

Cost model

Maltese regulated RAG workload — 38-chunk corpus, ~800 tokens prompt, ~100 tokens response. Real baselines from lessons 09–13: ~$0.02 for 15 traces, ~$0.03 for the 10-question eval, ~$0.10 for 24 judge calls. gpt-4o-mini + gpt-4o at 2025 list pricing; self-hosted Llama-3-8B amortised over hardware floor.

Fraction of requests that need a second call (eval gate failure, malformed JSON, timeout).

Fraction of requests that escalate to gpt-4o (used as fallback model for the primary). Adds the fallback model’s cost on top.

Fraction of responses scored by an LLM judge (gpt-4o-mini) for the eval pipeline (lesson 12). 0% = no judging; 100% = score every response.

Cross-encoder rerank step (lesson 07). No API cost, but ~650 ms p50 latency tax — converted here into a per-task worker cost (~$0.0007/task at $0.001/sec worker rate). The toggle moves every column’s effective cost.

The four levers

  • Reranker latency tax. The reranker toggle is not “no cost, just slow”. The ~650 ms p50 latency from lesson 07 means each worker serves fewer requests per second; the same daily volume now needs more worker-time. We model this as a flat $0.0007 per task at a $0.001/sec worker rate, applied across every column. The toggle actually moves the curve, not just the latency budget.
  • Volume. Daily request count drives whether the self-hosted model’s hardware floor amortises. At 100 requests per day, $28.80 of daily GPU cost is $0.288 per request — uncompetitive. At 100,000 requests per day, the same hardware is $0.000288 per request — half a milli-cent. The hardware floor pays back at scale; below that scale, pay-per-token wins.
  • Retry rate. Drag the retry slider up to 10%. Every column’s “effective per task” number rises by 10%. Retries are the most under-modelled cost in early-stage AI builds — teams budget for the happy path and discover the retry tax in production. The eval pipeline from lessons 10–13 is what tells you the real retry rate before the bill arrives.
  • Fallback rate. Drag the fallback slider up to 5%. The gpt-4o column’s effective cost barely changes — it’s already the premium model. The gpt-4o-mini column jumps significantly because each fallback adds a full gpt-4o call ($0.0035) on top of the $0.00018 base. Fallback rate is where rate-card economics most diverge from production economics.
  • Judge sampling. Drag judge sampling up to 100%. Every response now also pays for a gpt-4o-mini judge call (~$0.00007). At 25% sampling it’s a quarter of that. The eval pipeline from lessons 10–13 has a recurring cost that scales with traffic, and the architect’s call is what fraction of production traffic to score — too low and you don’t catch regressions, too high and you’re double-paying for every request.

The crossover point

Switch the volume from 100/day up through 100k/day with the other levers at their defaults. Watch which column wins at each volume.

  • At 100/day, gpt-4o-mini wins easily. Self-hosted’s hardware floor is $28.80/day spread over 100 requests — about 1,500× more expensive than gpt-4o-mini per task.
  • At 1k/day, gpt-4o-mini still wins. Self-hosted is $0.0288/task vs gpt-4o-mini‘s ~$0.0002/task — still about 150× more expensive.
  • At 10k/day, the gap narrows. Self-hosted is $0.00288/task — now about 15× more expensive than gpt-4o-mini. Sovereignty starts paying for itself in this volume band.
  • At 100k/day, the curve bends. Self-hosted falls to $0.000288/task; gpt-4o-mini at the rate-card is $0.00018/task — they cross over depending on retry and judge sampling. For L3+ workloads that can’t leave the country, the self-hosted column is the only column anyway.

Margin at the bottom

The bottom row of each card shows margin at an indicative €0.50 per request — a placeholder Tier-2 price. Green margin means the column is profitable at that price; red means the column would lose money. Drag the sliders enough and you can put any column into red.

For Tier-2 engagements the architect’s job is to set this price defensibly: cost-plus-margin works if the eval pipeline is honest about retry and judge rates; flat-rate-per-request only works when the variance is well-understood. The calculator is the conversation that picks the model.

What this lesson doesn’t say

  • It doesn’t say self-hosted’s full cost is in the model. The hardware floor ($28.80/day) and the operational overhead ($30/day for sysadmin / on-call / patching / procurement) are both modelled. What is NOT modelled: the cost of the architect designing the deployment, hardware lead-time risk, GPU availability constraints, and the security audit overhead specific to running EU-AI-Act-graded sovereign infrastructure. The calculator measures the unit economics; the total cost of ownership is a wider conversation.
  • It doesn’t say gpt-4o-mini is “the cheap option”. It’s the cheapest API-rate-card option in this comparison. A team running 1k requests/day with low retry/fallback/judge rates absolutely should default to it. A team running 100k requests/day across L3 workloads has no choice but self-hosted. The calculator doesn’t recommend; it makes the trade-off visible.
  • It doesn’t say €0.50 per request is the right price. Real engagements set their price per engagement based on procurement, competitive context, and risk appetite. The figure is a teaching scale — it lets the margin row mean something.

Next: security

One last lesson closes the Production Architecture track. Lessons 14, 15, and 16 cover what the system is, how to see it, and what it costs. Lesson 17 covers the failure mode the other three don’t surface: a poisoned document that gets retrieved into the prompt and tells the model to ignore everything it was instructed to do. Indirect prompt injection — same Maltese corpus, one bad chunk, and a 30-line validator that catches it.