Retrieval: BM25 vs dense vs hybrid

Lesson 06 — why most RAG production systems run two retrievers, not one. Pick the wrong one and the model never sees the right document.

Two retrievers, two failure modes

BM25 is the lexical retriever. It scores documents by how often the query’s tokens appear, adjusted for document length and rare-term weight. It is forty years old, deterministic, fast, language-agnostic, and ruthlessly literal. If the query says “Article 18” and the document says “art. 18”, BM25 misses.

Dense retrieval is the semantic retriever. It encodes the query into a vector via a small transformer (here, all-MiniLM-L6-v2, 384 dimensions), encodes every chunk the same way, and ranks by cosine similarity. It is younger, probabilistic, slower, language-sensitive, and forgivingly approximate. If the query says “penalties for late reporting” and the document says “administrative fines for delayed submission”, dense usually finds it.

The architect-grade insight is not that one wins. It is that they fail on opposite queries. Use the wrong one and the corpus that has the right answer goes invisible.

The rule

BM25 and dense are not competing options. They are complementary axes. The question is never which one; it is in what proportion.

The most common production answer is to run both and combine their rankings with Reciprocal Rank Fusion — RRF — which is exactly what the third column below does.

Try it on the Maltese AI corpus

The corpus below is real: three Maltese legal notices on AI (LN 226 of 2025, LN 227 of 2025, Cap. 586 the Data Protection Act) plus an engineering reference. Thirty-eight chunks. Six queries deliberately chosen so each one exposes a different retrieval mode.

Pick a query. Read the three columns. Notice the rank dots — B, D, H — that mark when the same chunk appears in another retriever’s top five. Notice the italic verdict line at the bottom.

What you’ll feel

• ID-style queries can fail BM25 too. The first query asks for “LN 226 Article 3” — and BM25’s top hit is from a completely different document (Cap. 586) because the string “LN 226” does not appear inside LN 226’s own text. The corpus does not refer to itself by its notice number. This is a real failure mode and a real architect concern: even lexical retrieval needs the query to use the corpus’s vocabulary, not the user’s.
• Dense gets paraphrased questions. The query “what penalties apply for breaking AI rules” never says “penalties” in the corpus the way it does in the question; the legal text says “administrative penalties” and discusses fines indirectly. Dense lifts the conceptually relevant chunks; BM25 latches onto whatever sentence contains the most “penalty” hits, which may not be the one that answers the question.
• Acronyms are BM25’s home. The query “MDIA conformity assessment notifying authority” has three near-unique strings, and BM25’s top hit is the exact article that designates the Notifying Authority. Dense returns a plausible but wider scatter — it doesn’t know “MDIA” is rare and therefore informative.
• Hybrid is the conservative architect’s default. Over these six queries, hybrid is never the worst column. It is sometimes a tie with BM25, sometimes a tie with dense, often the only column that finds chunks from both documents on cross-document queries. The cost is constant: one extra retrieval pass, no extra prompt tokens, score-fusion in microseconds.

How Reciprocal Rank Fusion actually works

RRF is the simplest hybrid combiner that is still production-grade. For each chunk that appears in either retriever’s ranked list, its hybrid score is the sum of one over its rank plus a small constant k, taken once per retriever:

rrf(d) = sum over retrievers r of   1 / (k + rank_r(d) + 1)

The constant k = 60 is the default and works almost everywhere. The reason RRF beats score-normalisation hybrids is that it ignores the raw scores entirely — it only consumes ranks. BM25’s score scale (0 to ~50) and cosine similarity’s scale (-1 to 1) cannot be added meaningfully; their ranks can.

Architect-grade implications

• Pick retrievers by query mix, not by personal taste. If clients ask paraphrased questions, dense leads. If they ask using legal citations or product SKUs, BM25 leads. Most real systems see both, which is why most real systems run hybrid.
• Index two indexes, not one. The dense index is a vector store (here, an in-process matrix; in production, LanceDB or Qdrant). The BM25 index is a sparse inverted index (here, rank_bm25; in production, Tantivy or OpenSearch). Both indexes consume the same chunks but store different summaries of them.
• The cost of hybrid is the cost of “almost free”. An extra retrieval pass against a sparse index over 38 chunks is sub-millisecond; over 38 million chunks, single-digit milliseconds. Rank fusion is free. The cost that matters is later, in the reranker — which is lesson 07.
• Embedding model choice is a Maltese-specific architectural decision. all-MiniLM-L6-v2 was trained on mostly English; it underperforms on Maltese text because the tokeniser fragments Maltese morphology (see lesson 01). For Maltese RAG, the dense retriever needs a multilingual model — multilingual-e5-small or larger — or it falls back to behaving like a worse BM25.

What comes next

Hybrid retrieval reliably puts the relevant chunk in the top ten. It does not reliably put it in the top three. The prompt typically only takes three. That gap is where reranking lives — running a more expensive cross-encoder on the top fifty candidates to re-sort by joint query-document interaction. Lesson 07.

Markdown version of this page: https://mokh.tech/lessons/retrieval-bm25-vs-dense-vs-hybrid.md