Lesson 37 — the Layer 10 evaluation gate. Pass this layer when you can hand a safety review board the agent’s bounded contract, its failure-mode coverage, and its trace-replay evidence. The audit is the deployment permit.
The conversation with the safety reviewer
“Can the agent take an action you would not approve?” The honest answer is: only if a failure mode you have not instrumented for occurs. The safety reviewer’s next question is: which modes have you instrumented for? The audit document answers that, line by line.
The agent safety audit is the artifact that turns the heatmap, the trace tooling, and the bound contract into a defensible whole.
The rule: No agent deployment without an audit document covering: bounds, tools, failure-mode coverage, escalation policy, trace retention, and human-oversight surface. The audit is the deployment permit.
The audit sections
- Bounds — explicit numbers for step count, tool calls, tokens, wall-clock, cost.
- Tool surface — every tool the agent can call, with the parameter schemas and the security boundaries.
- Failure-mode coverage — for each of the seven recurring modes, what detection and what mitigation exists.
- Escalation policy — when the agent stops and asks for human input, and how.
- Trace retention — how long traces are kept, who can access them, how they can be deleted.
- Human oversight surface — the dashboard, the alerts, the on-call rotation.
For EU AI Act high-risk deployments, the audit also maps to Article 14 (human oversight), Article 15 (accuracy and cybersecurity), and Article 86 (right to explanation). The same audit is the evidence for all three.
Try it yourself
The safety audit tool below walks you through the six sections for a sample agent deployment. Score each section. The composite score tells you whether the deployment is reviewable, deployable with caveats, or not yet ready.
agent_audit.json missing
What you just saw
The section that is most often weak is failure-mode coverage. Teams instrument for the modes they remember, miss the ones that have not bitten them yet. The heatmap from the previous lesson is the checklist; the audit is the evidence the checklist was used.
The second-most-often weak section is trace retention. The trace is the explanation surface; if it is not retained for at least the lifetime of the user’s recourse window, the explanation cannot be produced when requested. For Article 86 compliance, this is a real risk.
Three architect-grade arguments
- The audit is the permit, not the formality. A deployment without the audit is unbounded scope on production traffic.
- Failure-mode coverage is your weak section. Use the heatmap from the previous lesson; mark each cell green when instrumented.
- Re-audit every architectural change. A new tool added, a bound relaxed, a model upgrade: each requires the audit to be re-run. Document the diff.
This closes the Agentic Systems track. With it, all 12 engineering layers have at least one lesson. The remaining work is editorial: refining the voice, replacing the placeholders that remain, and shipping the synthesis lessons that tie the layers together.