Reranking: precision purchased with latency

Lesson 07 — hybrid retrieval gets the right chunk into the top twenty. Reranking gets it into the top three. The cost is real, measurable, and almost always worth paying for regulatory work.

Two encoders, two jobs

Hybrid retrieval (lesson 06) hands the prompt builder a ranked list. The chunks at the top of that list were chosen by a bi-encoder — query and document encoded separately, then their vectors compared by cosine similarity. The encoding never sees query and chunk in the same pass. That is exactly why it is cheap: the document vectors are computed once at indexing time, and every new query only needs one encode plus a vector lookup.

The cheapness comes at a cost. The bi-encoder cannot model the interaction between a specific query term and a specific document phrase. It cannot tell that “who supervises” and “the competent supervisory authority is” are pointing at the same fact in the same direction. It can only project both into a 384-dimensional space and hope the projections land close. Often they do. Sometimes the right chunk lands at rank 17.

A cross-encoder does the opposite trade. It takes the query and one chunk together, passes them through a transformer that attends across both, and emits a single relevance score. It does this once per (query, chunk) pair. So scoring twenty candidates means twenty forward passes. There is no caching to do at indexing time — the model never sees the chunk without a query.

The bi-encoder is a lookup. The cross-encoder is a reading. Reranking is what you do when you can afford twenty readings but not twenty thousand.

Watch the reshuffle

The block below runs cross-encoder/ms-marco-MiniLM-L-6-v2 over the hybrid top-twenty from lesson 06’s corpus. Same six queries, same thirty-eight chunks. Press Run reranker and watch the rows reshuffle. Each post-rerank row carries an arrow badge showing how many positions it moved. The top five — what the prompt would actually carry to the LLM — are marked with a teal band on the left.

The total time is precomputed but real: it is the wall-clock time the cross-encoder actually took to score those twenty pairs on a local CPU. The counter ticks up in real time so the latency feels like what it costs, not what a marketing slide claims it costs.

What you’ll feel

• The dramatic climbers are the right answers. For “LN 226 Article 3”, Article 2 of LN 226 climbs from position 17 to position 2 — because the cross-encoder reads “Article” and the chunk together and sees that this is the exact section the query is pointing at. The bi-encoder couldn’t tell.
• The paraphrased query benefits most. “What penalties apply for breaking AI rules” lifts a Cap. 586 article on data-protection penalties from rank 20 to rank 4. The bi-encoder put it almost out of the top-twenty; the cross-encoder reads “penalties” and “fines” together and recognises the alignment.
• Already-good ranks barely move. “Who supervises high-risk biometric AI systems in Malta” was already a good hybrid query — the top-five was almost right. Reranking shuffles by ±1–3 positions and confirms what hybrid suspected. This is the honest case: reranking does not always pay off.
• The cost is not invisible. Six hundred to nine hundred milliseconds for twenty pairs on a CPU. Around thirty milliseconds per pair. Scale that to a hundred pairs and you have three seconds of latency before the first generated token. This is the bill.

The architect’s decision rule

Reranking is not free. It is also not optional everywhere. The rule comes from the cost function of being wrong.

• Regulatory and compliance work: always rerank. The cost of citing the wrong article of LN 226 to a regulator is a corrected filing or, worse, a fine. Half a second of added latency is rounding error against that. Rerank.
• Internal knowledge bases and search: usually rerank. The user is waiting. They will accept five hundred milliseconds for a noticeably better answer. The “noticeably better” matters: if hybrid is already at ~90% top-3 accuracy on your eval set, reranking buys you the last five percent and you should measure whether it is worth it.
• Conversational assistants and chatbots: maybe. The user expects a fast first token. Reranking adds half a second before generation starts. This may break the conversation’s rhythm worse than the occasional wrong-rank-2 hurts the answer. Measure both, then choose.
• Real-time agents and tool-use loops: rarely. If you are calling the RAG in a loop ten times per agent turn, the reranker becomes the dominant cost. Use a much smaller cross-encoder or skip reranking entirely.

Why ms-marco-MiniLM-L-6-v2

The reranker shipped with this block is cross-encoder/ms-marco-MiniLM-L-6-v2 — a six-layer MiniLM distilled from a larger cross-encoder, fine-tuned on the MS MARCO passage-ranking dataset. Twenty-two million parameters. Small enough to run on a CPU. Good enough to win on most public retrieval benchmarks against bi-encoders three times its size.

The architect’s question is not “is this the best cross-encoder?” — it is “what cross-encoder is calibrated for my domain?”. MS MARCO is English web-passage data. For Maltese regulation, an architect should plan to fine-tune the cross-encoder on a small (~200 pair) labelled set of (query, relevant chunk) pairs from the actual corpus. The ranking lift from domain fine-tuning typically exceeds the lift from a bigger off-the-shelf model. The bigger model is the easier purchase; the fine-tune is the unfair advantage.

What comes next

Reranking gets the right chunk into the prompt. Sometimes the right chunk is also too small to interpret on its own — a paragraph that points at a definition two paragraphs above, or a sub-clause that loses its meaning without its parent article. Lesson 08 separates “what we retrieve” from “what we serve” with parent-child retrieval: search small chunks for precision, but ship the surrounding parent chunk into the prompt for interpretive scope.

Markdown version of this page: https://mokh.tech/lessons/reranking-precision-purchased-with-latency.md