Observability: the trace that explains a decision

Lesson 15 — in lesson 09 the trace was the query and the model’s answer. In production it is twelve more fields, and most of them are also the audit log. Same data; new lens.

From “what did it say” to “why did it say it”

Lesson 09 showed fifteen RAG traces — each one a query, a top-5 retrieved set, a model response, and an architect-applied failure-mode label. That’s enough to teach diagnostic vocabulary. It is not enough to ship to a regulated client.

This lesson takes the same fifteen traces and layers production-grade observability on top: request_id and tenant_id, timestamp and model_version, time-to-first-token and time-per-output-token, input and output token counts, and per-request cost. The compliance overlay then re-views the same fields under Article 12 (automatic logging, ≥6 months retention) and Article 86 (right to explanation) of the EU AI Act.

Production observability is not a separate layer from compliance. It is the same telemetry seen through a different lens. For Maltese regulated work, observability IS compliance — Article 12 and Article 86 obligations are satisfied by the fields you already need for debugging, latency analysis, and cost control.

Inspect a trace

Click any of the fifteen cells in the strip at the top to switch the panel to a different trace. Then toggle the compliance overlay — “Article 12 logging” lights up the fields that must survive in the audit log; “Article 86 right-to-explanation” lights up the fields that reconstruct why the model gave the answer it did. Most traces light up under both, because the same field often satisfies both obligations.

Reading the latency waterfall

The waterfall splits end-to-end latency into two parts. TTFT (time to first token) is the model’s warm-up cost — prompt encoding, attention setup, the first forward pass. It is largely fixed per request; in lesson 04’s vocabulary, this is when the KV cache gets primed. TPOT (time per output token) is the decoding cost — every subsequent token’s autoregressive step. End-to-end latency is TTFT plus TPOT times the number of output tokens.

Two architect-relevant consequences fall out of this split. Long responses are mostly decoding, so if a user complains about latency, the first question is “how many tokens are we asking the model to generate?” — output token count drops latency more than any prompt optimisation. Short responses are mostly TTFT, so prompt caching and warm pools are what move the needle for chat-style interfaces.

Article 12 — automatic logging

Toggle the “Article 12 logging” overlay. The fields that light up are the seven the EU AI Act requires to be kept for at least six months under Article 12 — timestamp, tenant_id, request_id, model_version, the prompt, the retrieved chunks, and the model response. They are not optional. They are not “if we have time.” They are the audit log a regulator will ask for under inspection rights.

For most production systems these fields are already being collected for non-compliance reasons — debugging, latency analysis, cost attribution, post-incident review. The compliance overlay is a relabelling exercise: the audit log is the operational log; ship one pipeline that satisfies both.

Article 15 — accuracy, robustness, cybersecurity

Toggle the “Article 15 robustness” overlay. The highlighted fields are different again — model_version (so regressions are trackable across rollouts), the retrieved chunks (was the right source surfaced), the model response (substantive answer or appropriate refusal), and the architect-applied eval label + corrective action. These are the fields a robustness audit asks for: not “what did the system do this time” but “is the system stable across versions and inputs”.

Notice that the eval label appears under both Article 15 (robustness — quantifying behaviour over time) and Article 86 (explanation — telling the user why this answer). Same field, two roles. Lesson 10’s golden-set scorecard is the artifact a regulator would read for the Article 15 evidence; it lives on the same trace that satisfies Article 86 to the user.

Article 86 — right to explanation

Toggle the “Article 86 right-to-explanation” overlay. The fields that light up are a smaller set — the model response, the retrieved chunks that grounded it, the eval label the architect applied, the corrective action. Together they answer the question a user has the right to ask under Article 86: why did the system give me this answer.

Notice what is NOT on the Article 86 list. The user does not have the right to see your model_version, your tenant_id, or your per-request cost. Those are audit-log fields. The Article 86 surface is narrower and more presentation-ready — it is the answer + the citations + a short explanation of how the system arrived at it.

The architect’s observability discipline

• Log everything, redact at query time. Capture the full prompt and full response into the audit log. Redact PII and sensitive context only when the log is queried by anyone other than the auditor. Trying to redact at log-write-time loses the original record and breaks the audit trail.
• One trace_id across the stack. The request_id you see in this lesson should flow through the API gateway, the RAG pipeline, the model gateway, the model itself (where supported), and any tool calls. When a user complains, the trace_id is the join key. Without it, debugging is guessing.
• TTFT and TPOT as separate SLOs. Don’t conflate them into “p95 latency”. TTFT regressions usually mean infrastructure (cold pools, key vault latency, prompt cache miss). TPOT regressions usually mean model (a version bump, a context-length change, a quantisation rollback). They have different on-call playbooks.
• Cost per trace, not per token. Lesson 16 will turn this into a live calculator. For now: every trace in this lesson has its actual list-price cost attached. Sum across the fifteen traces and you have a measured baseline for what a thousand-trace day costs. Multiply for the volumes you expect and you have a procurement-ready estimate.

What this lesson doesn’t say

• It doesn’t say tenant_id is measured. The TTFT / TPOT / tokens / elapsed / cost values in this block are real — captured by streaming gpt-4o-mini responses through the chat-completions API. The tenant_id, request_id, and timestamp are still synthesised because the original lesson 09 traces don’t carry them (and we want the multi-tenant audit story to be concrete). In a real production system those would also be measured from the surrounding application context.
• It doesn’t say Articles 12, 15, and 86 are the only obligations. The EU AI Act has further requirements (Article 26 user transparency, Article 50 disclosure for AI interactions, Article 60 testing requirements). Those map to other fields on the trace and other artifacts elsewhere in the stack. This lesson focuses on the three obligations that are clearly observability-shaped.
• It doesn’t say a single tracing tool covers all twelve components. The “single pane of glass” pitch is convenient but real systems use App Insights for the platform, OpenTelemetry for the RAG pipeline, and a custom audit-log writer for the regulator-facing record. The discipline is consistent trace_id propagation, not single-tool consolidation.

Next: cost

Each trace in this lesson has a per-request cost on the panel. Sum the fifteen and you have a baseline. Lesson 16 turns that baseline into a live calculator — sliders for volume, retry rate, fallback rate, judge-call frequency, with side-by-side comparison of gpt-4o-mini vs gpt-4o vs self-hosted Llama on the same Maltese workload. The trace tells you what one request costs; the cost model tells you what a thousand-trace day costs, what margin you have at Tier 2 pricing, and where retries quietly destroy the unit economics.

Markdown version of this page: https://mokh.tech/lessons/observability-the-trace-that-explains-a-decision.md