·

Diagnosis: retrieval, chunking, or generation failure

5 min read

Lesson 09 — when a RAG system gives a wrong answer, the architect’s first job is to name what failed. Three modes, three fixes, one diagnostic vocabulary.

Why diagnosis matters more than blame

Every wrong RAG answer prompts the same conversation. The product manager says the model is bad. The engineer says the prompt needs tuning. Someone suggests switching to a newer model. Sometimes the actual problem is none of those — it’s that the right chunk never made it into the prompt, or the chunker split the answer across two pieces, or the model genuinely had everything it needed and still hallucinated.

An architect’s job in that conversation is to refuse to guess. Three failure modes account for almost every wrong RAG output. Each has a different fix. Mixing them up is how teams spend three months tuning prompts when the actual bug is in the chunker.

I can debug RAG without blaming the model first.

The three failure modes

  • Retrieval failure. The chunk that would have answered the question is not in the top-5 the system handed to the model. The system can’t answer with what it didn’t surface. Fixes live in the retriever (BM25 + dense + RRF tuning), the corpus (what’s ingested), the chunker (coverage), or query rewriting. The model is innocent.
  • Chunking failure. The relevant chunks are in the top-5 but the chunking split the answer across them, dropped the definition article through a min-length filter, or broke a cross-reference. The fix is in how the corpus is broken into units. Sometimes it’s parent-child retrieval (lesson 08). Sometimes it’s just looser chunker thresholds.
  • Generation failure. The right chunk is in the top-5 and contains the answer. The model still gets it wrong — fabricates a citation, mis-attributes a source, gives the wrong article number despite the chunk metadata being correct. The fix is prompt engineering and, more importantly, citation validation. In regulated work this is the most dangerous mode because the answers read confident and correct.

Try diagnosing 15 real traces

Fifteen recorded RAG runs against the same Maltese AI corpus used by lessons 05 to 08. Same hybrid + cross-encoder retrieval pipeline. Generation by gpt-4o-mini with a citation-instruction prompt. For each trace you see the query, the actual top-5 chunks the system retrieved, and the model’s response.

Read the trace. Classify the failure mode. The block reveals the actual label, why it’s the right one, and the architect’s corrective action. The set spans all three failure modes plus four genuinely-correct baselines — recognising when the system is working is part of the diagnosis.

traces_curated.json missing — run tools/build-lesson-09-traces.py then tools/curate-lesson-09-traces.py

What you’ll feel

  • Retrieval failure dominates the budget. Six of the fifteen traces are retrieval failures — either the corpus doesn’t contain the answer, or the right chunk didn’t make top-5. This is the boring infrastructure problem most teams don’t want to spend time on, and it’s where most of the wrong answers actually come from.
  • Generation failures are the scariest in regulated domains. Trace 14 (“Who appoints the Information and Data Protection Commissioner?”) gives a content-correct answer with a fabricated chunk citation. In a regulated audit trail that’s a critical bug — the citation doesn’t link to a real source. The answer looks right. It is only wrong if you check the citation.
  • Some failures are the model being honest. When the corpus doesn’t have the answer, the model refusing is correct behaviour. The system-level failure (we don’t ingest the EU AI Act) is real, but the model’s individual response is not the problem. Naming this is part of the discipline.
  • Two-mode failures exist. Trace 12 (“Quote Article 9(1) of LN 226”) is both a retrieval failure (the LN 226 Article 9 chunk wasn’t in top-5) and a generation failure (the model quoted LN 227’s Article 9 and called it LN 226’s). Architects need a primary label to direct the fix, but should call out the secondary mode in the post-mortem.

The architect’s debugging discipline

  • Always read the retrieved chunks first. Before reading the model’s response, look at what the retriever chose. If the right chunk isn’t there, no prompt tuning will fix the answer — work upstream.
  • Check citations against the prompt’s chunk set. A model that cites a chunk_id that wasn’t in the retrieved set has fabricated. This is cheap to detect with a regex over the response and impossible to detect by reading prose.
  • Distinguish “corpus gap” from “retrieval gap”. If the chunk that would have answered the question literally doesn’t exist in the corpus, fixing the retriever can’t help. Ingest more. If the chunk exists but wasn’t ranked well, that’s where retrieval tuning lives.
  • Refusal is information. A model that says “the chunks don’t contain enough” is telling you about your corpus and your retriever. Reward that behaviour in production — don’t pressure it out with prompt engineering.

What the eval gate looks like

This lesson is the Layer 9 eval gate from the LLM Systems Engineering Roadmap made interactive. A team that can sit down with a recorded trace, classify the failure mode, and name the corrective action without blaming the model first is a team ready to ship RAG into production. A team that immediately reaches for “let’s switch to GPT-4o” or “we need a bigger embedding model” is a team that hasn’t earned that decision yet.

The vocabulary is the asset. The traces are the curriculum.

The five-lesson arc, end to end