Retrieval, Explained: BM25, Dense Embeddings, SPLADE, and ColBERT

type

status

date

slug

summary

1. A map before the equations

A retrieval system maps a query and a corpus to a small ranked candidate set. A later stage may reorder those candidates, assemble context, or generate an answer. A good score is not enough: the search procedure must actually find documents with high scores.

Method	What is stored for a passage?	How does query meet document?	Main engineering question
BM25	Term counts and length statistics	Shared analyzed terms receive corpus-based weights	Does the analyzer preserve the terms that matter?
Single-vector dense retrieval	One learned vector	A dot product or another trained similarity	Does the representation preserve the required distinction?
SPLADE	A sparse vector over vocabulary tokens	Dot product over shared, possibly expanded coordinates	Which useful terms are activated, and at what index cost?
ColBERT	Multiple contextual token vectors	Each query vector selects its best document match	Is finer matching worth the storage and search work?
Cross-encoder	Usually text; no reusable final document representation for this score	Query and document are jointly encoded	Which candidates deserve expensive pairwise scoring?

The mechanisms come from the BM25 authors' technical report, DPR, SPLADE, ColBERT, and BERT passage reranking.

Two corrections matter. Sparse does not mean non-neural: SPLADE uses a neural encoder. Dense does not mean GPU-only: vector search can run on CPUs. Also, an inverted index is not necessarily a word index; FAISS uses inverted lists of vector clusters too. See the FAISS index documentation.

BM25 and a neural retriever define scores. HNSW and IVF are search/indexing choices. Reranking is a pipeline role. Calling all of them competing “retrieval algorithms” hides the decisions we need to make.

2. BM25: reward distinctive terms, but discount repetition

BM25 scores matching query terms using three signals: frequency within a document, rarity across documents, and document length. A conventional form is:

Here, f(t,d) counts occurrences of term t in document d. |d| is document length. avgdl is the corpus average. We use unique query terms in the examples. Query-frequency variants also exist. The original authors' report, especially its weighting section, is a useful starting point.

TF and document frequency answer different questions

Term frequency, or TF, asks how often a term appears in this document. Document frequency, df(t), asks how many documents contain it. Repeating a term 1,000 times in one document adds 1,000 occurrences, but only one document to df.

For the examples, use the positive IDF convention described in Elastic's BM25 explanation:

N is the number of documents. Different implementations use different IDF variants and score scaling. The examples are not a bit-for-bit recreation of a search engine.

What k1 and b actually change

Set k1 = 1.2 and hold document length at the average. The TF multiplier becomes 2.2f / (f + 1.2). One occurrence contributes 1.0; five contribute about 1.77. As frequency grows, the multiplier approaches 2.2, not infinity.

That is the role of k1: it changes the shape of the saturation curve, not just a final score multiplier.

Now hold TF at one and set b = 0.75. At average length, the multiplier is 1.0. At twice average length, it is about 0.71. At half average length, it is about 1.26. The same occurrence counts as stronger evidence in a shorter passage. Setting b = 0 removes this length adjustment.

These calculations isolate one variable at a time. In an actual corpus, editing a document can also change average length and term statistics.

Where BM25 stops

Consider the query car repair. A passage containing only automobile maintenance has no matching terms under a basic analyzer. Changing k1 or b cannot create missing overlap. Synonym expansion or a different representation must supply it.

Conversely, an exact identifier such as ERR_CONNECTION_RESET can be highly useful evidence—but only if the analyzer and field configuration preserve it. Exact-match requirements may deserve a dedicated keyword field rather than ordinary tokenized text.

BM25 also does not turn a list of terms into a logical specification. “Supports encryption” and “does not support encryption” share most words. Phrase queries, Boolean conditions, and structured filters are additional machinery, not consequences of the BM25 score.

My working model is: BM25 measures how much distinctive lexical evidence a document contains, after adjusting for repetition and length. TF-IDF is a related family, not one unique formula that BM25 merely patches. For the broader derivation, read Robertson and Zaragoza's The Probabilistic Relevance Framework: BM25 and Beyond.

3. Dense retrieval: learn which texts should be neighbors

A single-vector bi-encoder separately maps the query and passage into vectors:

The passage vectors can be computed before the query arrives. DPR uses separate BERT encoders and trains their dot products to distinguish relevant passages from negatives. Its core loss makes the positive passage more probable within a candidate set:

See DPR, Sections 3.1–3.2. This is why “take an arbitrary BERT embedding and run nearest-neighbor search” is not equivalent to using a trained retriever.

For car repair, a positive might describe automobile maintenance. A useful hard negative might discuss car racing. If every negative is unrelated cooking text, the model can separate topics without learning the distinction between repairing and racing a car. Negative selection helps define what relevance means.

Similarity is learned, not guaranteed by the word “embedding”

A question and its answer need not be paraphrases. “When does this credential expire?” should retrieve a passage stating the expiration rule, not necessarily a passage asking a similar question.

This also locates SimCSE correctly. It is a sentence-representation learning method, with unsupervised dropout-based and supervised variants. It is not an index or a complete retrieval pipeline. Sentence similarity training and query–passage relevance training can produce useful but different geometries.

The single-vector design commits to compressing each passage into one reusable representation. My concern is not that such a vector contains “only the topic.” It can encode much more. The practical question is whether this checkpoint preserves the specific identifiers, relations, and constraints our queries need.

Separate representation errors from ANN errors

Approximate nearest-neighbor search, or ANN, tries to find high-scoring vectors without exhaustively scoring every vector. It can miss a document that the model itself would rank highly. FAISS provides both exhaustive indexes, such as flat inner-product search, and approximate choices. Official index reference.

This gives a diagnostic experiment: on a manageable corpus, compare exact search with the deployed ANN index using the same embeddings. If exact search retrieves the right passages but ANN does not, first inspect search settings and compression. If both miss them, increasing ANN search effort cannot repair the learned score.

“ANN recall” often means overlap with exact vector-search results. That is different from recall against human relevance judgments. An index can reproduce the model's top results perfectly while those results remain irrelevant.

4. SPLADE: encode semantics into named retrieval keys

SPLADE produces a sparse vector whose coordinates are vocabulary tokens. A coordinate can become active even when its token is absent from the original text. The semantic bridge is created during encoding, while the final score still matches the same coordinates. The authors' introductory article explains the connection between expansion and inverted indexing.

Here is a constructed example. These weights are illustrative, not outputs from a checkpoint:

Their sparse dot product is:

There is no direct car × automobile comparison. Encoding activated shared keys, so ordinary same-coordinate multiplication can recover a semantic match.

Follow the tensor shapes

For a text with n tokens, the encoder produces n × h contextual hidden states. The masked-language-model head maps each state to logits over a vocabulary of size |V|, producing n × |V| values.

In the SPLADE v2 max-pooling formulation:

The max removes the input-position axis, leaving one |V|-dimensional vector. The original SPLADE used sum aggregation; v2 changed this to max. These versions should not be described as identical. Original paper · SPLADE v2, Section 3.2.

There is no vocabulary softmax in this formula. Several coordinates can be active at once, and their weights do not sum to one. The ranking objective may use a softmax over candidate documents; that is a different axis and a different purpose.

The inverted index executes the same score

A physical index stores nonzero entries as postings:

For each active query key, visit its postings and accumulate query_weight × document_weight. This computes the same sparse dot product without scanning zero coordinates. Notice that passage_7 can appear in the repair posting even though the original passage never used that word.

This is the distinction I find most useful: a vocabulary coordinate names a learned retrieval feature, not necessarily an observed word. Its weight is neither a probability nor an IDF value.

What makes expansion useful rather than noisy?

A masked-language-model initialization supplies linguistic structure. Retrieval supervision teaches which activations separate relevant passages from competing ones. In the example above, activating maintenance can help distinguish the positive from car racing results. Training changes shared encoder parameters, not a separately optimized vector for each example.

Sparsity regularization discourages indiscriminate activation. Otherwise, semantic expansion can create huge postings and expensive queries. The FLOPS-style regularizer penalizes broadly active coordinates; it is a proxy for matching work, not a guarantee of a particular latency. Distillation and negative mining also matter. SPLADE v2 · official implementation and model variants.

A readable expansion can still be wrong. battery replacement might activate battery disposal, which is related but may not answer the question. Inspecting keys explains part of the score; it does not certify relevance. I would compare expansion errors by query type before concluding that a sparse neural model is more trustworthy than a dense one.

5. ColBERT: keep token evidence until the query arrives

A transformer normally produces a contextual vector at each token position. Self-attention exchanges information between positions; it does not automatically merge them into one vector.

ColBERT uses separate query and document encodings, then projects and normalizes token vectors. An illustrative BERT-based shape path is n × 768 → n × 128. The result is multiple vectors per passage, not repeated copies of one passage embedding. Documents can still be encoded offline. ColBERT architecture · official implementation.

Let query vectors form Q, with m rows, and document vectors form D, with n rows. Compute:

For normalized vectors, each matrix entry is a cosine similarity. This MaxSim score is the defining late-interaction operation.

Work through one matrix

Take this constructed matrix. The labels are mnemonic; these are not measured similarities for the words:

Query vector / document vector	automobile	maintenance	manual
car	0.90	0.10	0.00
repair	0.10	0.80	0.20

For car, choose 0.90. For repair, choose 0.80. Add them: 1.70.

Each query vector chooses its own document witness. The model does not choose one token for the entire query, and it does not require both witnesses to be the same token.

The mental model from my notes is query requirements versus document evidence. But it has a boundary: actual vectors include subwords and special or query-augmentation positions. They cannot all be interpreted as independent English words or explicit logical requirements.

Why max over document tokens?

Max asks whether a strong match exists somewhere in the passage. It avoids averaging a single useful match with every unrelated document position.

Hold the matrix fixed and duplicate a column. The maximum does not change. Sum pooling would count the duplicated evidence again. This is a property of the aggregation, not a claim that duplicating text leaves contextual embeddings unchanged.

Max has the opposite weakness too: a second independent piece of evidence contributes nothing to a row once it falls below the best match. One accidental spike can dominate that row.

Why sum over query tokens—and why not reverse it?

Summing row maxima rewards coverage of different query vectors. Reversing the axes would ask how well the query explains every document token. A relevant passage can contain background that the query never asked about, so those are different objectives.

Even the intended direction is not Boolean AND. Consider three row maxima:

A wins despite its weak third match. If that third requirement is mandatory, the sum does not enforce it. The learned vectors may help, but the formula supplies no hard guarantee.

For “a cache that is thread-safe and expires entries,” separate token matches also need not establish that the same cache has both properties. Negation, entity relations, numeric limits, and permissions need explicit evaluation. Enforce hard metadata and access constraints outside a soft relevance score.

Why not sum all token-pair similarities?

Changing both reductions to sums gives:

The score collapses to a dot product between pooled vectors. This follows directly by distributing the dot product. It is not literally identical to a separately trained CLS bi-encoder, but it loses the selective “best witness per query vector” operation. More background similarities and repeated columns can now accumulate score.

Would Top-K be better than Top-1?

For one query vector, compare document similarities:

Max prefers A. Averaging the best three prefers B. This changes the preference from one strong match toward several supporting matches. It is not automatically an improvement. A single exact answer may be diluted, while neighboring subwords may be mistaken for independent support.

For a fixed K, Top-K sum and Top-K average differ only by a positive constant, so they induce the same ranking for that query. Training can still change because the scale of the logits changes. Variable K breaks the constant-scale argument.

These are deductions from the scoring rule and constructed examples, not claims that a Top-K variant outperforms ColBERT.

What can we actually explain?

We can display each query vector's winning document position and score contribution. That is useful local attribution. It does not explain why the encoder assigned that similarity or prove the passage satisfies the request.

The White Box Analysis of ColBERT and its authors' blog offer a more precise finding: in their studied setup, term importance correlated moderately with IDF, and exact matching remained important, especially for rare terms. ColBERT has no explicit BM25-style scalar IDF in its MaxSim formula, yet training can shape matching behavior in a related direction.

6. ColBERTv2 and PLAID: make multi-vector search affordable

Three labels answer different questions. ColBERTv2 identifies a model/version and its associated improvements. Full retrieval means generating candidates from the corpus. Reranking means scoring candidates selected elsewhere. A v2 checkpoint can participate in either role. Official ColBERT project.

Compression is centroid plus correction

ColBERTv2 combines improved supervision with a compact representation. Each token vector is approximated by a shared centroid plus a quantized residual:

For 128 dimensions, the paper stores a four-byte centroid ID and one or two bits per residual dimension:

Token-vector payload	Bytes
FP32 illustration: 128 × 4	512
FP16 baseline: 128 × 2	256
Centroid ID + 1-bit residual	20
Centroid ID + 2-bit residual	36

The paper's explicit comparison is against the 256-byte FP16 encoding. Its reported 6–10× space reduction is not the arithmetic 512 / 20. Whole indexes also contain auxiliary structures. ColBERTv2, Section 3.3.

For a constructed budget with 100 million stored token vectors, these payloads alone require 25.6 GB at FP16, 2.0 GB at one-bit residuals, or 3.6 GB at two bits. Those decimal-GB calculations exclude centroids, identifiers outside the payload, postings, text, and runtime buffers.

Deleting residuals entirely makes vectors assigned to the same centroid indistinguishable at scoring time. It can erase distinctions needed for final ranking. It is reasonable to test centroid-only candidate screening; it is not reasonable to claim an unmeasured universal quality penalty for zero-bit residuals.

Full retrieval does not mean exhaustive token-pair scoring

PLAID uses centroid-based candidate generation, approximate centroid interaction and pruning, then residual decompression and MaxSim on a smaller final set. Query–centroid scores can be reused across passages. PLAID, Figure 5 and Section 4.

There is a subtle search problem here. Suppose two query vectors give these best-match scores:

Selecting only the top token neighbor for each query vector finds A and B, not C. Yet C wins the document-level sum. Therefore, a naive union of token Top-1 results does not guarantee the best MaxSim document. Candidate breadth and pruning must be evaluated against the aggregate objective.

My interpretation is that compression and candidate selection are part of retrieval quality, not merely deployment optimizations. A strong scoring function cannot rescue a passage discarded before final scoring.

7. Cross-encoders and hybrid retrieval: combine capabilities without confusing them

A cross-encoder spends computation on each pair

A BERT cross-encoder jointly processes the query and passage. Its document-side hidden states can depend on the query, so the final passage representation is not reusable in the same way as a bi-encoder's. Passage Re-ranking with BERT.

This supports richer interaction, but requires a forward pass per pair. A common design retrieves broadly with a cheaper method, then applies a cross-encoder to candidates. The Sentence Transformers guide illustrates this separation. A ColBERT scorer can also rerank; “reranker” does not imply “cross-encoder.”

Suppose ten passages are relevant, but candidate generation returns only six. A component that only reorders those candidates cannot retrieve the missing four. Total relevant-document recall is capped at 60%. It may still greatly improve the order of the six it received. Candidate coverage and ranking quality are separate measurements.

Hybrid search needs a fusion rule, not just two scores

BM25 and dense retrieval may find complementary passages. A simple experiment is to retrieve from both, merge by stable passage ID, fuse rankings, and rerank a bounded set. The sizes should be tuned rather than treated as architectural constants.

Raw BM25, cosine, SPLADE, and MaxSim scores have different scales. A weighted sum can work, but its weights and any normalization need validation.

Reciprocal Rank Fusion, or RRF, avoids direct score comparison:

Ranks start at one. In truncated lists, a missing document contributes zero. The original RRF paper used a smoothing constant of 60. That constant is not the number of documents retrieved.

For lists [A, B] and [C, B], with c = 60, B receives 2/62 ≈ 0.0323. A and C each receive 1/61 ≈ 0.0164. Agreement puts B first even though neither retriever ranked it first.

RRF discards score magnitude and can amplify agreement between similarly biased systems. I would treat it as a transparent baseline, not proof that fusion must beat every individual retriever. Inspect which query groups gain and lose.

8. A comparison I would trust

I would run two experiments, because they answer different questions.

End-to-end retrieval: let each method search the same corpus and measure the passages it actually returns. This tests representation, index, and search settings together.

Controlled reranking: give each scorer the same candidate set and measure ordering. This isolates scoring more closely, but it cannot establish full-corpus retrieval quality.

Keep corpus versions, passage boundaries, eligibility rules, and relevance judgments fixed. Record truncation, checkpoint, training data, quantization, candidate depth, hardware, and batch size. A modern distilled checkpoint beating an older baseline does not isolate architecture as the cause. The BEIR benchmark is useful precisely because it tests retrieval across different domains rather than only one familiar distribution.

Split queries into meaningful groups: exact names and error codes; paraphrases; multiple constraints; negation and numeric limits; long passages; and unanswerable requests. Measure where each method fails, not only its average score.

For relevant set R_q and returned set C_K, document recall is |R_q ∩ C_K| / |R_q|. Hit@K is simply whether the intersection is nonempty. Returning one of four relevant documents gives 25% recall but a hit of 1. DPR's answer-containing-passage accuracy is a hit-style measure, not “fraction of all relevant documents recovered.” DPR's evaluation definition.

Also report an early-ranking metric such as NDCG or MRR, p50/p95 latency, query-encoding time, index size, and indexing/update work. With incomplete relevance labels, qualify recall against the judged set. Do not silently label every unjudged result irrelevant.

For access-controlled retrieval, eligibility is a hard condition. Do not feed unauthorized passages to a reranker or generator and hope a low score will hide them. Evaluate coverage within the allowed corpus and account for filtering in candidate search.

Which method would I try first?

My default experiment would establish an analyzer-aware BM25 baseline and one trained dense baseline, inspect their errors, then test fusion. I would add SPLADE when learned lexical expansion is a promising fit for the infrastructure. I would test ColBERT when finer matching seems valuable enough to justify a multi-vector index. I would add pairwise reranking when candidate coverage is adequate but ordering remains weak.

This is a starting strategy, not a universal ranking of methods. A small collection may permit direct cross-encoding. Tight storage, rapid updates, or a different language can change the choice.

9. A small lab: verify the mechanics before downloading models

Download the full practice project

The companion LLM Engineering Lab is now public under the MIT License. It turns the mechanisms in this article into six exercises: BM25 TF weighting, sparse scoring, MaxSim, reciprocal rank fusion, recall, and candidate screening. Each exercise includes a contract, hints, tests, and a reference solution.

Download ZIP · Start the exercises · Open the notebook in Colab

Use Python 3.10 or later. The project uses only the standard library and needs no API keys, GPU, or model downloads. Download and unzip the project, then run these commands from its root:

Next, implement one function in labs/retrieval/exercises.py and run python3 -m labs.retrieval.check --only tf_weight to check the first exercise. The learner functions intentionally start with NotImplementedError; the repository test suite checks the reference implementation. The self-contained notebook offers the same six exercises in a browser.

The published project includes 41 unit tests and a local validator that executes all 16 notebook code cells and checks the six embedded reference exercise groups. See GitHub Actions for hosted CI results. These are constructed mechanics examples; they do not benchmark trained retrievers.

Original standalone lab

The companion retrieval_scoring_lab.py uses only Python's standard library. It implements conventional BM25 scoring, sparse dot products and equivalent postings traversal, MaxSim over supplied similarity matrices, and RRF. It contains 11 tests, including the examples above.

retrieval_scoring_lab.txt

7.77 KiB

The cloud attachment is plain text. Save it as retrieval_scoring_lab.py and run it with Python 3.10 or later.

Run:

The tests check saturation, length adjustment, pooling axes, a padding bug, sparse-score equivalence, Top-K preference reversal, candidate-generation misses, the sum–sum identity, fusion, recall versus hit, and payload arithmetic. They have been executed successfully. They do not measure the quality or speed of trained retrieval systems.

One useful bug to reproduce: pad a similarity row with zero when every valid similarity is negative. The padding wins the max. Exclude padded document positions before reduction; do not assume a zero vector is harmless. This is a scoring implementation issue, separate from whether the learned embeddings are good.

After the mechanics are clear, the official ColBERT implementation, official SPLADE implementation, and Sentence Transformers retrieval examples are better starting points for model-backed experiments than treating this toy lab as a search engine.

10. Where the other papers in my notes fit

Some related papers change a different layer. They should be connected to this map, not forced into the same scoring comparison.

Work	What it changes	What not to confuse it with
SimCSE	Sentence representation training	It does not define corpus indexing or a complete query–passage pipeline.
HyDE	Generates a hypothetical document, then embeds it to retrieve real documents	The generated text is a search aid, not evidence to cite as fact.
REALM	Retrieval-augmented language-model pretraining	It concerns learning with retrieval, not only a standalone similarity function.
Self-RAG	Learning when to retrieve and how to reflect on retrieved evidence and generation	It changes the retrieval–generation control process.
AgentIR	Embeds an agent's current reasoning with its query, using turn-level retrieval supervision	It changes the information need presented to the retriever and its training distribution.

AgentIR is the clearest bridge to a later agent-focused article. A short query may omit what the agent has already established and what it is trying to resolve. Its 2026 paper is evidence for the tested reasoning-aware setup, not proof that adding arbitrary conversation history always helps.

Query routing, tree navigation, retrieving a child passage then loading its parent, and permission filtering also belong in the broader system map. Mixing them into one “best embedding method” comparison would hide their actual roles.

11. Reading path: originals first, explanations second

For the foundational sequence, read the relevant sections of these papers rather than treating the list as a prerequisite to building:

Read	Focus
Robertson and Spärck Jones, Simple, proven approaches to text retrieval	Early BM25 weighting and inverted files. The linked report updates the 1994 original.
Robertson and Zaragoza, BM25 and Beyond, 2009	The probabilistic framework, assumptions, and variants; a later synthesis, not the first BM25 paper.
DPR, 2020	Independent encoders, contrastive training, and negatives.
SPLADE, 2021 and SPLADE v2, 2021	Vocabulary-space representations; then max pooling and stronger supervision.
ColBERT, 2020	Section 3 for scoring; Section 4 for retrieval and evaluation.
ColBERTv2	Sections 3.2–3.5 for supervision, compression, and search. Preprint in 2021; NAACL publication in 2022.
PLAID, 2022	Figure 5 and Section 4 for candidate refinement before final MaxSim.
BERT passage reranking, 2019 and RRF, 2009	Pairwise scoring versus rank fusion.

For explanations, these are the sources I would use alongside the papers:

Elastic: Practical BM25, Part 2. Useful for the effect of each variable and Lucene-oriented intuition. Treat implementation details as implementation-specific, not as the only possible BM25 formulation.

NAVER LABS: the authors' SPLADE introduction. Useful for understanding why semantic expansion can still use vocabulary coordinates and postings. Pair it with v2 for the changed pooling rule.

NAVER LABS: A White Box Analysis of ColBERT. Written by the researchers behind the analysis. Useful for separating inspectable token contributions from stronger claims about interpretability. Its empirical findings belong to the studied models and data.

Sentence Transformers: Retrieve & Re-Rank. Maintainer documentation with executable examples. Useful for implementing the pipeline distinction; not evidence that any particular checkpoint wins on a new corpus.

These sources are selected for proximity to the algorithms or implementation, not popularity. Where an intuitive explanation and a precise equation differ, identify the model version and return to the paper or code.

The distinction I most want to retain

SPLADE and ColBERT both use contextual language models, and both can use max pooling. They preserve different information until query time.

SPLADE pools token positions into a vocabulary vector before query–document matching. ColBERT preserves document token vectors and chooses a witness for each query vector after the query arrives. A cross-encoder goes further: the document's hidden states themselves can depend on that query.

My general lesson is to ask which decisions are made offline, which information is discarded, and which decisions remain possible online. Then test whether the information retained is enough for the task. That question is more useful than treating sparse, dense, and late interaction as successive levels on a single ladder.