Lesson 05 — the decision a retriever can never undo. Most teams discover this only after a wrong citation goes out to a regulator.
Before retrieval, a chunker has already decided what’s findable
A RAG system is usually drawn as a pipeline: documents come in on the left, an answer comes out on the right. Between them sit the model and the retriever, and almost all of the conversation focuses on those two. The chunker is drawn as a small box; sometimes it isn’t drawn at all.
This understates it. The chunker decides what the retriever is even capable of finding. If the answer to a question lives in a piece of text that no single chunk contains, no retriever, no matter how clever, can return it whole. The model will then improvise. The citation will be wrong. The user will see a confident answer drawn from an incomplete reading of the source.
This is the failure mode this lesson is about, and it is invisible until you draw it.
The rule
Chunking is a retrieval question, not a document question. Ask: would a citation fit in a single chunk?
That rule sounds simple. It is. The reason it gets violated is that the default chunking strategy in most RAG tutorials is the one that violates it most often: fixed-size chunks with overlap, sized to fit the embedding model’s input. The strategy is convenient because it requires no understanding of the document. That is also its problem.
Try it on real Maltese law
Below is Article 3 of Legal Notice 226 of 2025, Malta’s national implementation framework for the EU AI Act. It is a single piece of legal prose with several substantive references inside it. Article 68 of the EU AI Act. Annex I. Annex III. Section A of Annex I. regulation 9 of this same Legal Notice. These are the citations any retriever working over this corpus must be able to return whole.
Pick a strategy. Watch the chunk list on the right reform. Watch the references in the source either stay underlined or get cut in half. The headline metric is at the bottom: references split. The goal is zero.
The chunker
The source
Each chunk is what the retriever can find as a unit.
No chunks yet. Pick a strategy.
What you’ll feel
- Section-based chunking gets zero splits on this document, every time. The structural markers Malta’s parliamentary drafters use — the numbered sub-regulations, the lettered sub-paragraphs — are exactly the boundaries a retriever wants. The drafters did the chunker’s work decades before RAG existed.
- Fixed-size chunking, even at 600 characters, splits Section A of Annex I of Regulation (EU) 2024/1689 across two chunks. That reference appears twice in Article 3, in sub-paragraph (c). It is the answer to several plausible questions about scope. Fixed-size cannot return it.
- Semantic chunking happens to work here because Maltese legal sentences are paragraph-length. Sentence boundaries and paragraph boundaries overlap in this prose, so semantic and section produce similar shapes. Drop the chunk size below 240 characters and the picture changes: now a single legal sentence overflows, semantic falls back to fixed-size’s failure mode, and references split.
- Sliding window hides the problem, it does not fix it. With large enough overlap you can usually return a chunk that contains the full reference, but you also return three other chunks for the same query, you pay for the redundancy at retrieval time, and you have not actually made the chunker structure-aware.
Architect-grade implications
Three claims worth carrying out of this lesson.
- Chunking is part of the architecture, not part of the parsing. It deserves a design decision per corpus, not a default copied from a notebook. The retriever, the reranker, and the eval set all assume the chunker’s contract; changing the chunker invalidates the index and the retrieval baselines.
- For structured corpora — legal, regulatory, scientific, technical — section-based wins by default. The structural markers are the document’s own admission of where the meaningful boundaries are. Use them. Fall back to other strategies only when the corpus genuinely has no structure.
- The right metric is references split, not chunk size. Average chunk size and chunk count are operational metrics; they tell you what the index costs. References split is the only metric that tells you what the index can answer. If it is greater than zero on your corpus, you are shipping a retriever that cannot return its primary unit of truth.
The Maltese angle
Maltese legal drafting follows the same conventions as English-language UK and EU drafting. Articles are numbered. Sub-articles are parenthesised. Sub-paragraphs are lettered. This is good news: a section-based chunker tuned for one Maltese Legal Notice transfers to most of them and to the EU regulations they implement. The chunker’s contract is a single regex over Article and sub-regulation markers.
The harder problem is Maltese-language prose without legal structure. Newspaper articles, parliamentary debates in Maltese, agency press releases. These need semantic chunking that respects Maltese morphology, and that brings the lesson back to Layer 1: tokeniser quality determines what a semantic chunker can even see. The two layers compound.
What comes next
This lesson assumed the retriever already exists and asked: what do we hand it? Lesson 06 reverses the question. Given chunks, what kind of retriever should look at them? BM25 against dense embeddings against the combined hybrid, on the same corpus, on the same queries, with the failure modes of each made visible.