I spent nine years building deterministic software. Same input, same output, forever — and every tool in the craft depends on that promise. The unit test asserts equality. The bug report says “steps to reproduce.” CI is green or red. Then I started shipping LLM systems, and the promise quietly disappeared: the same input now produces a distribution of outputs. Most of the pain I watch teams suffer with AI comes from not updating their habits for that one sentence.
What actually changes
Assertions become thresholds. You cannot assert that the answer equals a string. You measure that correctness and faithfulness stay above a bar, across a gold set. The unit test’s child is the eval harness, and “passing” becomes a rate, not a yes or no.
Reproduction becomes tracing. “Steps to reproduce” stops working when step three is a sampled token. What replaces it is the trace — the recorded inputs, retrieved context, and outputs of the run that actually failed. In deterministic software, the log is nice to have, because you can replay reality. In non-deterministic software, the trace is reality. There is no replay.
Regression becomes statistical. A prompt change that helps eight cases and silently breaks three is invisible to the eye. My CI gate for agents compares every run against a frozen gold set with an explicit tolerance — a few percent of noise is expected, and beyond that the build fails. It feels strange the first time: a test suite that accepts imperfection by design. It is also the only honest contract available.
Correctness gets a third value. Deterministic functions are right or wrong. A probabilistic system has an important middle state: knowing that it does not know. Refusing and asking for clarification are not failures — on some inputs they are the correct output, and they belong in the spec as first-class labels. A system that answers everything is not confident. It is unsafe.
The skill is drawing the boundary
The real skill is not choosing between the two worlds. It is deciding where each one belongs. A production AI system is mostly deterministic machinery wrapped around a small probabilistic core: deterministic retrieval feeding it, schema checks and citation checks catching it, routers and approval gates bounding it. Thirty lines of deterministic Python that verify a citation exists will beat any amount of prompt-engineering hope. Put determinism everywhere you can afford it. Spend non-determinism only where it buys you something language-shaped.
The failures live at a badly drawn boundary. Some teams trust the model with jobs a validator should own — parsing, routing, arithmetic. Other teams lock the model down so tightly it cannot do the one creative job it was brought in for. Both mistakes come from discomfort with the distribution, instead of engineering around it.
How I spot who gets it
When I work with engineers on AI systems, this is the divide I look for. Not “do you know the frameworks” — frameworks change every year. I listen for someone who says rate instead of works, who asks for the trace instead of the steps to reproduce, and who treats a refusal as a feature they designed rather than something to hide. Engineers cross that line in weeks once they see it. The ones who never see it ship demos that have not broken yet — and call them products.