The most expensive mistake in AI engineering right now is treating the model as the system. Teams spend weeks comparing models, switching providers, tuning prompts — while the real failure lives somewhere else: in retrieval, in chunking, in a missing refusal rule, in a feedback loop nobody built.
I once mapped a production LLM system end to end and counted the components: ingestion, chunking, embedding, retrieval, prompt assembly, generation, validation, observability, evaluation, cost control, governance, and the humans in the loop. Twelve components. The model was one of them. When I weighted them by cost share, the LLM came out around 12% of the system — smaller than retrieval and governance combined. Systems thinking is the discipline of acting on that fact.
Find the real constraint
Every system has one part that actually limits it, and it is rarely the famous part. My favorite example: a RAG system gives a wrong answer. The instinct says “the model hallucinated — get a better model.” Then you measure recall@k and discover the right document was never retrieved at all. No model can fix a context window that never contained the answer. The limit was retrieval. The model was just the part everyone could see.
This is why I build measurement before I build features. The eval harness is how you find the real constraint instead of guessing. One variable per experiment, a gold set that does not move, and the honesty to accept that the bottleneck is usually in a layer you did not want to look at.
Measurement also protects you from your own assumptions. In one experiment I compared my default pipeline (hybrid retrieval plus a reranker) against a variant with the reranker turned off — and the variant won, by 10 percentage points on my combined metric. The component I was proud of was costing me quality. I would never have found that by feel. The regression test catches what confidence ships.
Feedback loops beat components
An average pipeline with a working feedback loop will beat an excellent pipeline without one, given a few months. The loop I trust most is old and boring: support tickets. Every resolved ticket is a labeled example — a real question, a verified answer, a known source document. I have mined eval sets from exactly that. It grounds the system in what users actually ask, not in what the team imagines they ask.
The same thinking applies to decay. Systems do not fail at launch. They fail eleven months later, when the docs changed, the content drifted, and nobody re-ran the evals. A regression gate in CI is a feedback loop against your own entropy. It is the difference between a system and a demo that has not broken yet.
Ask what happens two steps later
Systems thinking also means asking what a change does after the first step. Automate the easy support answers, and the humans now handle only the hardest ones — did you plan for that? Add an AI assistant, and users stop reading the docs the assistant depends on — who notices when those docs rot? Cut turnaround time 40% with automation, as I have done, and the bottleneck simply moves downstream to whoever consumes the output. None of these are reasons not to build. They are reasons to draw the whole loop before shipping one node of it.
My degree is in pharmacy, and I think that is where this instinct comes from. Nobody in pharmacology asks only “does the drug work?” They ask what it interacts with, what the side effects are, and what happens at ten times the dose. AI engineering needs the same posture. The component is easy to praise. The system is what the patient actually takes.