Every retrieval system answers one question, namely where in the computation graph query information first meets document information. This post traces that single design axis from BM25 through DSSM, kernel pooling, monoBERT, DPR, ColBERT, SPLADE, PreTTR, and generative retrieval, then closes it into a collectively exhaustive taxonomy with a rank-bound argument, a FLOP model, open problems, and portfolio projects.
Bi-encoder, cross-encoder, and late interaction are not three competing architectures; they are three points on a single continuum defined by how much of the scoring computation you are willing to do before you have seen the query.
Every retrieval and ranking system in existence computes one function: a relevance score $s(q, d)$ between a query $q$ and a document $d$. Everything else in the system, including index size, query latency, whether approximate nearest neighbor search is even applicable, and the accuracy ceiling you can reach with unlimited training data, is downstream of a single architectural decision.
That decision is: where in the computation graph does query information first meet document information?
Meet early, and every layer of the network can condition on both sides at once. You get maximum expressiveness and you pay for it by recomputing everything at query time for every candidate. Meet late, and the document side can be computed once, months before the query exists, and stored. You get sublinear search over billions of documents and you pay for it with a hard mathematical ceiling on what scores you can express. The field spent fifty years discovering that this is a genuine trade and that the interesting engineering lives strictly between the two extremes.
The vocabulary is unfortunately fragmented. Practitioners say “bi-encoder” for late meeting, “cross-encoder” for early meeting, and “late interaction” for a specific middle point invented by ColBERT, which is confusing because a bi-encoder interacts even later than “late interaction” does. This post replaces the vocabulary with a two-parameter coordinate system, places every published architecture in it, and shows which cells are still empty. The field has traveled through eight waves:
Before any history, the formal frame. Every scoring function that can be deployed at scale admits a factorization:
\[s(q, d) \;=\; g\big(\phi(q),\; \psi(d)\big)\]where $\psi: \mathcal{D} \to \mathcal{Z}_d$ is the document-side encoder that runs without access to the query, $\phi: \mathcal{Q} \to \mathcal{Z}_q$ is the query-side encoder that runs without access to the document, and $g$ is the interaction operator that is the first place the two sides meet.
This factorization is not an assumption; it is a definition with a trivial degenerate case. A cross-encoder is the choice $\psi = \text{id}$, meaning $\mathcal{Z}_d$ is the raw document text and $g$ is the entire transformer. So the frame is complete: every architecture is a choice of what $\psi$ is allowed to emit and how expressive $g$ is allowed to be. Those two choices are the coordinate system for the rest of this post.
Three quantities fall directly out of the factorization, and they are the only three that matter for systems work:
\[\underbrace{B_d = \lvert \mathcal{Z}_d \rvert}_{\text{index bytes per doc}}, \qquad \underbrace{F_\psi}_{\substack{\text{offline FLOPs,}\\ \text{paid } N \text{ times, once}}}, \qquad \underbrace{F_g}_{\substack{\text{online FLOPs,}\\ \text{paid } \lvert \mathcal{C} \rvert \text{ times per query}}}\]where $N$ is the corpus size and $\mathcal{C}$ is the candidate set actually scored. The whole field is a fight over the third quantity, because $F_\psi$ is amortized to nothing and $B_d$ is merely expensive, while $F_g$ multiplies by $\lvert \mathcal{C} \rvert$ on every single query.
Here is the one theorem that explains the entire field, and it is three lines long.
Suppose $g$ is an inner product on $\mathbb{R}^k$, which is the bi-encoder choice. Collect all queries and all documents into the score matrix $S \in \mathbb{R}^{\lvert Q \rvert \times \lvert D \rvert}$ with $S_{ij} = s(q_i, d_j)$. Then by construction
\[S \;=\; \Phi \Psi^\top, \qquad \Phi \in \mathbb{R}^{\lvert Q \rvert \times k}, \quad \Psi \in \mathbb{R}^{\lvert D \rvert \times k} \quad \implies \quad \operatorname{rank}(S) \;\le\; k\]No amount of training data, model scale, or clever loss function changes this. If the true relevance matrix $S^\star$ has rank higher than $k$, the bi-encoder cannot represent it, and the Eckart-Young theorem gives the exact size of the irreducible error:
\[\min_{\operatorname{rank}(S) \le k} \lVert S^\star - S \rVert_F \;=\; \sqrt{\sum_{i > k} \sigma_i^2(S^\star)}\]where $\sigma_i(S^\star)$ are the singular values of the true relevance matrix in decreasing order. This is why bi-encoders plateau, why distillation from a cross-encoder never fully closes the gap, and why the two other major families exist. Late interaction escapes the bound by making $g$ non-linear. Learned sparse retrieval escapes it by making $k$ equal to the vocabulary size. Cross-encoders escape it by refusing to factorize at all.
Keep this inequality in mind for the rest of the post. Every architecture after 2019 is an answer to it.
The standard story is that classical IR is not vector interaction at all. That story is wrong, and correcting it is the first payoff of the frame above.
Term weighting begins with Spärck Jones’s inverse document frequency (Journal of Documentation, 1972) and reaches its mature form in BM25 (Robertson and Walker, SIGIR 1994):
\[s_{\text{BM25}}(q,d) \;=\; \sum_{t \in q \cap d} \text{IDF}(t) \cdot \frac{f(t,d)\,(k_1 + 1)}{f(t,d) + k_1\left(1 - b + b\,\frac{\lvert d \rvert}{\text{avgdl}}\right)}\]where $f(t,d)$ is the term frequency, $\lvert d \rvert$ the document length, $\text{avgdl}$ the mean document length over the corpus, and $k_1, b$ are saturation and length-normalization constants, conventionally $k_1 \approx 1.2$ and $b \approx 0.75$.
Now read it through the factorization. Define $\psi(d) \in \mathbb{R}^{\lvert V \rvert}$ with $\psi(d)_t$ equal to the saturated, length-normalized term frequency, and $\phi(q)_t = \text{IDF}(t) \cdot \mathbb{1}[t \in q]$. Then
\[s_{\text{BM25}}(q,d) \;=\; \langle \phi(q),\, \psi(d) \rangle\]BM25 is exactly a bi-encoder. It has a hand-designed, non-learned encoder into a sparse space of dimension $\lvert V \rvert \approx 10^6$. Its rank bound is $\operatorname{rank}(S) \le \lvert V \rvert$, which is effectively no bound at all. This single observation explains the most stubborn empirical fact in modern IR: BM25 remains competitive out of domain, and often beats fine-tuned dense retrievers on unfamiliar corpora, because it has enormous representational rank and zero training-distribution dependence.
Its failure is equally structural. If $q \cap d = \emptyset$ the score is exactly zero, no matter how semantically related the two are. A query for “how to fix a leaky faucet” scores zero against a document titled “repairing a dripping tap”. This is the vocabulary mismatch problem, and every neural retrieval architecture that follows is an attempt to solve it.
Latent Semantic Indexing (Deerwester et al., JASIS, 1990) attacked mismatch by truncating the SVD of the term-document matrix $X \approx U_k \Sigma_k V_k^\top$, giving
\[\psi_{\text{LSI}}(d) = \Sigma_k^{1/2} V_k^\top e_d \in \mathbb{R}^k, \qquad \phi_{\text{LSI}}(q) = \Sigma_k^{1/2} U_k^\top q\]with $k$ typically 100 to 300. This is the first dense bi-encoder, and it is the first system to hit the rank bound, by construction and on purpose: the score matrix has rank exactly $k$. LSI fixed vocabulary mismatch and lost precision, because collapsing to 300 dimensions destroys the exact-match signal that BM25 gets for free. The tension between recall from smoothing and precision from exact matching is visible in 1990 and is not resolved in 2026; it is why hybrid retrieval is still the default production answer.
DSSM (Huang et al., CIKM 2013) is the first learned neural bi-encoder. It hashes words into letter-trigrams to keep the input dimension bounded, pushes each side through an independent multilayer perceptron to 128 dimensions, and trains on clickthrough data with a softmax over one clicked document and sampled unclicked ones:
\[P(d^+ \mid q) \;=\; \frac{\exp\big(\gamma \cos(\phi(q), \psi(d^+))\big)}{\sum_{d \in \{d^+\} \cup \mathcal{D}^-} \exp\big(\gamma \cos(\phi(q), \psi(d))\big)}\]where $\gamma$ is a learned temperature. Every ingredient of DPR seven years later is already here: two towers, cosine similarity, sampled negatives, contrastive softmax. What is missing is a good encoder, and that would not arrive until BERT.
Between 2016 and 2018 the neural IR community concluded that squashing a document to one vector destroys too much, and built a family of models around the opposite intuition: compute a full token-by-token similarity matrix first, learn on top of it second.
DRMM (Guo et al., CIKM 2016) histograms the cosine similarities. MatchPyramid (Pang et al., AAAI 2016) treats the similarity matrix as an image and runs a CNN over it. The cleanest formulation is K-NRM (Xiong et al., SIGIR 2017), which builds the matrix
\[M_{ij} \;=\; \cos\big(e_{q_i},\, e_{d_j}\big), \qquad i = 1 \ldots n_q,\; j = 1 \ldots n_d\]and then applies $K$ radial basis kernels to turn each query token’s row into a soft term-frequency profile:
\[K_k(M_i) \;=\; \sum_{j=1}^{n_d} \exp\!\left(-\frac{(M_{ij} - \mu_k)^2}{2\sigma_k^2}\right), \qquad s(q,d) \;=\; w^\top \tanh\!\Big(W \big[\log K_1(M_i), \ldots, \log K_K(M_i)\big]_{i=1}^{n_q}\Big)\]where $\mu_k$ is the kernel center, sliding from $-1$ to $1$, and $\sigma_k$ its bandwidth. A kernel centered at $\mu = 1$ counts near-exact matches; a kernel at $\mu = 0.5$ counts soft matches. Conv-KNRM (Dai et al., WSDM 2018) extends this to cross-matching n-grams of different lengths.
Guo et al. named the split in their 2016 paper: representation-focused models compress each side independently, interaction-focused models build the matching signal first. That taxonomy is the direct ancestor of this post. What nobody noticed at the time is that interaction-focused models were still fully cacheable on the document side, because static word vectors do not depend on the query. K-NRM is structurally ColBERT with word2vec instead of BERT. The idea was right and the encoder was not good enough, so the whole family was swept away in 2019 and had to be reinvented.
Nogueira and Cho’s Passage Re-ranking with BERT (arXiv 1901.04085, 2019; arXiv preprint, not peer-reviewed, though it became the field’s standard baseline) does the least imaginative thing possible and it works enormously well. Concatenate the query and the passage into one sequence, run BERT over it, put a linear layer on the [CLS] output:
On the MS MARCO passage ranking development set, BM25 scores roughly 0.187 MRR@10. Reranking BM25’s top 1000 with BERT-large pushes that to roughly 0.365. Nearly doubling the primary metric of a mature benchmark in one paper is rare, and it reset the field’s expectations overnight.
The reason it works is exactly the rank bound running in reverse. There is no $\psi$ and no factorization, so there is no constraint on $\operatorname{rank}(S)$ whatsoever. Every one of BERT’s twenty-four layers of self-attention can put query tokens and document tokens in the same attention computation. The model can represent “this document is relevant to this query for a reason that generalizes to no other query,” which is precisely the high-rank structure a bi-encoder is forbidden from expressing.
Variants followed quickly. CEDR (MacAvaney et al., SIGIR 2019) feeds BERT’s contextual embeddings into K-NRM style kernel pooling, explicitly grafting Act II onto Act III. Birch (Yilmaz et al., EMNLP 2019) scores sentences and aggregates to document level. monoT5 (Nogueira et al., Findings of EMNLP 2020) reframes ranking as conditional generation, scoring with the normalized logit of a single token:
\[s(q,d) \;=\; \frac{\exp(z_{\texttt{true}})}{\exp(z_{\texttt{true}}) + \exp(z_{\texttt{false}})}\]where $z_{\texttt{true}}$ and $z_{\texttt{false}}$ are the decoder logits for the words true and false at the first output position, given the prompt “Query: q Document: d Relevant:”. RankT5 (Zhuang et al., SIGIR 2023) later showed that training this with a real ranking loss rather than the language modeling loss is worth several points.
The cost model is brutal. A transformer forward pass over $n$ tokens with hidden width $k$ and $L$ layers costs approximately
\[F_{\text{CE}} \;\approx\; \lvert \mathcal{C} \rvert \cdot L\,\big(12\,n\,k^2 \;+\; 2\,n^2 k\big), \qquad n = n_q + n_d\]where the first term is the feedforward and projection matrices and the second is the attention score and value products. Put in realistic numbers: $n_q = 32$, $n_d = 256$, $k = 768$, $L = 12$, and a rerank depth of $\lvert \mathcal{C} \rvert = 1000$. That is about $2.2 \times 10^9$ FLOPs per layer per pair, $2.6 \times 10^{10}$ per pair, and $2.6 \times 10^{13}$ FLOPs per query. A single A100 at 312 TFLOP/s of dense bfloat16 needs roughly 83 milliseconds of pure arithmetic at 100 percent utilization, for one query, at rerank depth 1000.
Compare a bi-encoder, which encodes only the 32-token query and then does an approximate nearest neighbor lookup: about $2.7 \times 10^9$ FLOPs total, independent of $\lvert \mathcal{C} \rvert$. That is a factor of roughly 9,500.
This chart is the reason the next five acts exist.
Reimers and Gurevych (Sentence-BERT, EMNLP 2019) opened with the arithmetic that made the problem concrete: finding the most similar pair among 10,000 sentences requires about 50 million cross-encoder inferences, roughly 65 hours on a V100. Encoding all 10,000 sentences once and then doing cosine similarity takes about 5 seconds. The paper’s contribution is a siamese fine-tuning recipe that makes BERT’s pooled output actually usable for cosine similarity, which vanilla BERT’s [CLS] emphatically is not.
DPR (Karpukhin et al., EMNLP 2020) applied this to open-domain question answering with two independently parameterized BERT encoders and in-batch negatives, which is the trick that makes the whole thing trainable at reasonable cost: every other document in the batch serves as a negative for every query, so a batch of size $B$ yields $B^2$ training pairs for $B$ forward passes.
\[\mathcal{L} \;=\; -\log \frac{\exp\big(\phi(q)^\top \psi(d^+) / \tau\big)}{\exp\big(\phi(q)^\top \psi(d^+)/\tau\big) + \sum_{j=1}^{B-1} \exp\big(\phi(q)^\top \psi(d_j^-)/\tau\big)}\]where $\tau$ is a temperature. DPR beat BM25 on Natural Questions top-20 accuracy by roughly nine points, which was the moment dense retrieval became the default first stage rather than a research curiosity.
Once the architecture is fixed at “two towers plus a dot product,” the only remaining levers are the data and the loss, and the field pulled them hard for two years.
ANCE (Xiong et al., ICLR 2021) diagnosed the core problem precisely: in-batch negatives are drawn from the corpus at random, so they are trivially easy, and the gradient norm they contribute vanishes as training proceeds. ANCE maintains an asynchronously refreshed index of the current model and mines negatives from the model’s own top-ranked non-relevant documents.
RocketQA (Qu et al., NAACL 2021) added cross-batch negatives across GPUs and, crucially, used a cross-encoder to denoise hard negatives. Top-ranked non-relevant documents in MS MARCO are frequently relevant but unlabeled, so training on them directly teaches the model to demote correct answers.
TAS-B (Hofstätter et al., SIGIR 2021) composes batches from topically clustered queries with balanced margin sampling, reaching competitive quality on a single consumer GPU.
Margin-MSE distillation (Hofstätter et al., arXiv 2020, not peer-reviewed) is the most conceptually important of the group. Rather than matching a teacher’s absolute scores, which live on an incompatible scale, it matches the teacher’s margin between a positive and a negative:
\[\mathcal{L}_{\text{Margin-MSE}} \;=\; \text{MSE}\Big(\; s_S(q,d^+) - s_S(q,d^-),\;\; s_T(q,d^+) - s_T(q,d^-) \;\Big)\]where $s_S$ is the bi-encoder student and $s_T$ is a cross-encoder teacher ensemble. This is how essentially every strong dense retriever after 2021 is trained, including the E5, GTE, and BGE families.
Condenser and coCondenser (Gao and Callan, EMNLP 2021 and ACL 2022) attacked a different layer: standard masked language modeling never trains [CLS] to aggregate anything, so the pretrained initialization is actively bad for bi-encoding. Adding a pretraining objective that forces information through the [CLS] bottleneck improves every downstream retrieval number.
By 2021 the recipe was mature, and the residual gap to a cross-encoder had stopped shrinking. That gap is the rank bound, and no training trick removes it.
ColBERT (Khattab and Zaharia, SIGIR 2020) makes one change to the bi-encoder: do not pool. Keep one vector per token on both sides, and replace the dot product with a sum of per-query-token maxima:
\[s_{\text{ColBERT}}(q, d) \;=\; \sum_{i=1}^{n_q} \; \max_{j = 1, \ldots, n_d} \; \phi(q)_i^\top \, \psi(d)_j\]Two properties make this the most interesting point in the design space.
It escapes the rank bound. The maximum is not a bilinear form, so the factorization $S = \Phi\Psi^\top$ does not apply and $\operatorname{rank}(S)$ is not capped at $k$. ColBERT uses $k = 128$, one sixth of a typical bi-encoder’s width, and still outperforms it, because the expressiveness comes from the operator rather than the dimension.
It stays cacheable. $\psi(d)$ does not depend on the query, so the entire corpus can still be encoded offline. Retrieval then works in two steps: put every document token vector into one large ANN index, retrieve candidate documents by their individual tokens, then rescore those candidates with the full MaxSim.
The cost sits neatly between the extremes. Scoring $\lvert \mathcal{C} \rvert$ candidates costs $2 \lvert \mathcal{C} \rvert\, n_q\, n_d\, k$ FLOPs, which for the running example of 1000 candidates, 32 query tokens, 256 document tokens, and $k = 128$ is about $2.1 \times 10^9$: comparable to encoding the query once, and about four orders of magnitude below a cross-encoder at the same depth.
The bill arrives as index size. A single-vector index over the 8.8 million MS MARCO passages at 768 dimensions in float32 is about 27 GB. ColBERT stores roughly one 128-dimensional vector per token, on the order of 150 GB for the same corpus at full precision. A factor of five to six here is the difference between fitting in RAM and not.
Five years of work went into paying that bill down, and the sequence is a good case study in how a research community optimizes a fixed architecture:
ColBERTv2 (Santhanam et al., NAACL 2022) observed that token vectors cluster tightly around a modest number of centroids, and replaced each vector with a centroid ID plus a one or two bit residual per dimension, cutting the index by six to ten times with negligible quality loss.
PLAID (Santhanam et al., CIKM 2022) attacked latency rather than size, pruning candidate documents using centroid-level scores before touching any residuals, reporting speedups of roughly 45 times on CPU.
ColBERTer (Hofstätter et al., CIKM 2022) reduced the number of vectors by aggregating subword pieces into whole words and learning to drop uninformative ones.
CITADEL (Li et al., ACL 2023) routes each token vector to a predicted lexical key and only lets query and document tokens interact if they share a key, which converts the all-pairs max into a sparse lookup and recovers ColBERTv2 quality at much lower cost.
XTR (Lee et al., NeurIPS 2023) made the sharpest observation of the group: the standard ColBERT pipeline retrieves candidate documents by token, then gathers all of their token vectors to rescore. XTR changes the training objective so the retrieved tokens alone are sufficient, imputes the contribution of unretrieved tokens, and eliminates the gather stage entirely, making scoring two to three orders of magnitude cheaper while improving BEIR nDCG@10 by 2.8 points without distillation.
MUVERA (Dhulipala et al., NeurIPS 2024) is the conceptual endpoint. It constructs asymmetric Fixed Dimensional Encodings of the query and document vector sets such that a plain inner product between the two fixed-dimension vectors approximates MaxSim with an $\varepsilon$ guarantee:
\[\big\lvert \langle \text{FDE}(q), \text{FDE}(d) \rangle - s_{\text{ColBERT}}(q,d) \big\rvert \;\le\; \varepsilon \quad \text{with high probability}\]This is the first single-vector proxy for multi-vector similarity with a theoretical guarantee, and it matters structurally: it says late interaction can be approximately projected back onto the bi-encoder axis, so the two families are not as separate as the architecture diagrams suggest. Reported results reach the same recall as prior heuristics while retrieving two to five times fewer candidates, with about 10 percent better recall at 90 percent lower latency on BEIR.
ColPali (Faysse et al., ICLR 2025) took late interaction out of text entirely. It encodes a document page as an image into roughly 1,024 patch vectors via a SigLIP vision encoder and a language model projection to 128 dimensions, then runs MaxSim against query token vectors. No OCR, no layout parsing, no text extraction pipeline. On the ViDoRe benchmark it substantially outperforms conventional text-extraction retrieval pipelines while being simpler and end-to-end trainable. The lesson is that the interaction operator is modality-agnostic; only the encoders need to change.
Late interaction escapes the rank bound by changing $g$. There is another option: keep $g$ as a plain inner product and make $k$ enormous but sparse.
\[s(q,d) = \langle \phi(q), \psi(d)\rangle, \qquad \psi(d) \in \mathbb{R}^{\lvert V \rvert}, \qquad \lVert \psi(d) \rVert_0 = \bar n_{\text{nz}} \ll \lvert V \rvert \quad \implies \quad \operatorname{rank}(S) \le \lvert V \rvert\]Set $\lvert V \rvert \approx 30{,}000$ and constrain each document to a few hundred non-zero entries $\bar n_{\text{nz}}$. The rank bound is now no constraint in practice, and because the vector is sparse the inner product can be evaluated on a forty-year-old inverted index: the cost of a query depends on the length of the posting lists touched, not on $\lvert V \rvert$.
This is BM25’s representation with learned weights, and the family is called learned sparse retrieval.
doc2query and docTTTTTquery (Nogueira et al., arXiv 2019, not peer-reviewed) do not change the architecture at all. They generate the queries a document would plausibly answer, append them to the document, and index the result with ordinary BM25. Vocabulary mismatch is attacked at the data layer.
DeepCT (Dai and Callan, Context-Aware Term Weighting For First-Stage Passage Retrieval, SIGIR 2020) replaces raw term frequency with a BERT-predicted importance weight, so that a term’s index weight depends on its context rather than its count.
COIL (Gao et al., NAACL 2021) is the bridge between this act and the previous one. It keeps a vector per token, like ColBERT, but gates the interaction on exact lexical match:
\[s_{\text{COIL}}(q,d) \;=\; \sum_{t \in q \cap d} \;\max_{j \,:\, d_j = t} \; \phi(q)_t^\top \psi(d)_j\]Compare this to ColBERT’s MaxSim: the only difference is the restriction $d_j = t$. COIL is ColBERT confined to the diagonal of the vocabulary, which is precisely what makes it implementable as contextualized inverted lists. uniCOIL (Lin and Ma, arXiv 2021, not peer-reviewed) then collapses each vector to a single scalar, recovering a standard inverted index exactly.
DeepImpact (Mallia et al., SIGIR 2021) combines doc2query expansion with learned scalar impact scores in one model.
SPLADE (Formal et al., SIGIR 2021) is the family’s mature form. It reuses BERT’s masked language modeling head, which already maps a hidden state to a distribution over the vocabulary, to project every token into $\lvert V \rvert$-dimensional space, then max-pools with log saturation:
\[w_v(d) \;=\; \sum_{j=1}^{n_d} \log\Big(1 + \text{ReLU}\big(\mathbf{h}_j^\top E_v + b_v\big)\Big), \qquad v \in V\]where $\mathbf{h}_j$ is the contextual state of token $j$ and $E_v$ is the input embedding of vocabulary item $v$. Nothing so far makes the output sparse, so SPLADE adds an explicit regularizer that penalizes the expected posting-list length, which is the actual quantity that determines query latency:
\[\mathcal{L} \;=\; \mathcal{L}_{\text{rank}} \;+\; \lambda_q\, \ell_{\text{FLOPS}}(q) \;+\; \lambda_d\, \ell_{\text{FLOPS}}(d), \qquad \ell_{\text{FLOPS}} \;=\; \sum_{v \in V} \left( \frac{1}{B} \sum_{i=1}^{B} w_v^{(i)} \right)^2\]The squared mean over a batch is a differentiable surrogate for the average number of postings touched per query term. This is a rare and instructive case of a loss function whose second term is literally a latency model.
SPLADE++ (Formal et al., SIGIR 2022) closed the loop by importing the dense retrievers’ training recipe wholesale: cross-encoder distillation, hard negative mining, and better pretrained initialization, showing that sparse models benefit from exactly the same tricks.
Dense and sparse are the same operator at different dimensionalities. Dense has low rank and smooth semantic geometry, so it handles paraphrase; sparse has effectively unbounded rank and exact lexical grounding, so it handles rare entities, identifiers, and out-of-domain terms. Their errors are close to uncorrelated, which is the ideal condition for fusion:
\[s_{\text{hybrid}}(q,d) = \alpha\, \tilde{s}_{\text{dense}}(q,d) + (1-\alpha)\,\tilde{s}_{\text{sparse}}(q,d), \qquad \text{or} \qquad \text{RRF}(d) = \sum_{r \in R} \frac{1}{\kappa + \text{rank}_r(d)}\]where $\tilde{s}$ denotes score-normalized rankers and $\text{RRF}$ is Reciprocal Rank Fusion (Cormack et al., SIGIR 2009) with a smoothing constant conventionally $\kappa = 60$. RRF is usually preferred in production because it needs no score calibration between systems on incompatible scales, which is a real problem when one ranker outputs cosines in $[-1,1]$ and the other outputs unbounded sums of impact weights.
The most important structural idea in this post is that bi-encoder and cross-encoder are not two options; they are the endpoints of a one-parameter family. Take a transformer of $L$ layers, run the first $\ell$ layers on the query and the document separately, then concatenate the hidden states and run the remaining $L - \ell$ layers jointly:
\[s_\ell(q,d) \;=\; f_{\ell+1:L}\Big(\big[\; f_{1:\ell}(q) \;;\; \underbrace{f_{1:\ell}(d)}_{\text{cached offline}} \;\big]\Big)\]At $\ell = L$ this is a bi-encoder, because no joint layer remains. At $\ell = 0$ it is a cross-encoder. Every value in between is a valid, trainable architecture with a monotone cost profile, and the online cost is
\[F_g(\ell) \;\approx\; \lvert \mathcal{C} \rvert \cdot (L - \ell)\,\big(12\,n\,k^2 + 2\,n^2 k\big), \qquad B_d(\ell) \;=\; n_d\, k\, b\]Online compute falls linearly in $\ell$ while index size stays flat at one hidden state per document token, which is the ColBERT storage regime. That asymmetry is the reason this family is less popular than late interaction: you pay ColBERT’s storage bill and only get a fraction of the compute savings.
PreTTR (MacAvaney et al., SIGIR 2020) implements exactly the formula above for reranking, precomputing document term representations up to layer $\ell$ and caching them, with a learned dimensionality-reduction layer to keep storage tolerable. It reports roughly 42 times lower query-time latency at negligible quality cost.
DeFormer (Cao et al., ACL 2020) applies the same decomposition to question answering rather than ranking, adding distillation losses that align the decomposed model’s upper-layer states with the full model’s.
MORES (Gao et al., EMNLP 2020) restructures the transformer explicitly into a document representation module and a separate interaction module, making the split architectural rather than a post-hoc cut.
Poly-encoder (Humeau et al., ICLR 2020) sits very near the bi-encoder end and is the most widely deployed member of the family. The document keeps one vector. The query is summarized into $m$ learned codes, and the final score attends over those codes using the document vector as the attention query:
\[\mathbf{c}_i = \sum_{j=1}^{n_q} a_{ij}\, \mathbf{h}_j, \quad a_i = \text{softmax}\big(\mathbf{v}_i^\top H\big), \qquad s(q,d) = \Big( \sum_{i=1}^{m} \text{softmax}_i\big(\mathbf{c}_i^\top \psi(d)\big)\, \mathbf{c}_i \Big)^{\!\top} \psi(d)\]where $\mathbf{v}_i$ are $m$ learned code vectors and $H$ is the matrix of query token states. Because the document side is still a single cacheable vector, the index is bi-encoder sized. Because the score is a document-conditioned mixture over $m$ query views, the rank bound is relaxed from $k$ to roughly $mk$. This is the cheapest available purchase of extra rank, and it is why Poly-encoders remain common in production dialogue and recommendation retrieval.
Generative retrieval removes $\psi$ altogether. The corpus is memorized in the model parameters, and retrieval is decoding a document identifier:
\[s(q,d) \;=\; \log P_\theta\big(\text{docid}(d) \mid q\big) \;=\; \sum_{t=1}^{T} \log P_\theta\big(\text{id}_t \mid q,\, \text{id}_{<t}\big)\]The Differentiable Search Index (Tay et al., NeurIPS 2022) introduced the framing and showed that semantically structured document IDs, obtained by hierarchically clustering document embeddings, dramatically outperform arbitrary integer IDs; the ID space has to be learnable. NCI (Wang et al., NeurIPS 2022) adds a prefix-aware decoder and heavy synthetic query generation. SEAL (Bevilacqua et al., NeurIPS 2022) sidesteps ID design entirely by generating distinctive n-grams and mapping them back to documents with an FM-index, which keeps the output vocabulary grounded in actual corpus text.
The appeal is that the entire retrieval pipeline collapses into one differentiable model. The problems are severe and structural: adding a document requires updating $\theta$, so corpus dynamics become a continual learning problem; and reported quality degrades sharply as the corpus grows past the hundred-thousand to million document range, because the parameters must store the index.
The second departure keeps the encoders but changes the arity of $g$. Instead of scoring one document at a time, condition on the whole candidate set and emit a permutation:
\[\pi^\star \;=\; \arg\max_{\pi \in \mathfrak{S}_m} \; P_\theta\big(\pi \mid q,\, d_1, \ldots, d_m\big)\]where $\mathfrak{S}_m$ is the symmetric group on $m$ candidates. This lets the model reason about relative quality, novelty, and redundancy, none of which a pointwise scorer can see.
RankGPT (Sun et al., EMNLP 2023) prompts a large language model to output a permutation directly, using a sliding window to handle candidate lists longer than the context. RankZephyr (Pradeep et al., arXiv 2023, not peer-reviewed) distills that behavior into an open 7B model. Both inherit an awkward defect: the output depends on the input order of the candidates, so the same set permuted differently yields different rankings.
Set-Encoder (Schlatt et al., ECIR 2025) fixes this architecturally with permutation-invariant inter-passage attention, letting passages attend to each other through a shared representation while remaining invariant to their input order. It matches state-of-the-art listwise quality while being more efficient and, unlike RankGPT, robust to input permutation. It is particularly stronger when inter-passage information such as novelty actually matters.
This adds a genuine second axis to the taxonomy. Alongside “where does interaction happen,” there is now “how many documents participate in one interaction”: pointwise ($m = 1$), pairwise ($m = 2$), listwise ($m$ arbitrary).
Everything above collapses into two choices, and this is the collectively exhaustive claim of the post. Any relevance scorer is determined by:
The claim is exhaustive because the two choices are not a heuristic list; they are the only two free parameters in the factorization $s(q,d) = g(\phi(q), \psi(d))$, and that factorization is a definition with the cross-encoder as its degenerate case. Any published or future architecture must select a $\mathcal{Z}_d$ and a $g$, and therefore lands in a cell of the grid below.
Two quantities decide whether a cell is deployable at a given corpus size. The index footprint and the online cost per candidate are both fully determined by the row and column choices:
\[B_d = \underbrace{\bar n_{\text{nz}}\,(b + \log_2\lvert V\rvert)}_{\text{sparse}}, \quad \underbrace{kb}_{\text{dense}}, \quad \underbrace{n_d k b}_{\text{multi-vector or layer-}\ell}, \quad \underbrace{0}_{\text{cross, listwise, generative}}\] \[F_g = \underbrace{2k}_{\text{inner product}}, \quad \underbrace{2 n_q n_d k}_{\text{MaxSim}}, \quad \underbrace{(L-\ell)\big(12nk^2 + 2n^2k\big)}_{\text{partial cross-attention}}, \quad \underbrace{L\big(12nk^2 + 2n^2k\big)}_{\text{full cross-attention}}\]| Family | $\mathcal{Z}_d$ | $g$ | Rank bound | Index bytes/doc | Online FLOPs per candidate | ANN? |
|---|---|---|---|---|---|---|
| Learned sparse | sparse $\mathbb{R}^{\lvert V \rvert}$ | inner product | $\lvert V \rvert$ | $\bar n_{\text{nz}}(b + \log_2 \lvert V \rvert)$ | postings lookup | inverted index |
| Bi-encoder | $\mathbb{R}^{k}$ | inner product | $k$ | $kb$ | 0, resolved by index | yes |
| Late interaction | $\mathbb{R}^{n_d \times k}$ | MaxSim | none | $n_d k b$ | $2 n_q n_d k$ | yes, per token |
| Poly-encoder | $\mathbb{R}^{k}$ | $m$-code attention | $\approx mk$ | $kb$ | $2mk$ | approximately |
| Partial cross-attn | $\mathbb{R}^{n_d \times k}$ | $L-\ell$ layers | none | $n_d k b$ | $(L{-}\ell)(12nk^2 + 2n^2k)$ | no |
| Cross-encoder | none | $L$ layers | none | 0 | $L(12nk^2 + 2n^2k)$ | no |
| Listwise | none | set-conditioned | none | 0 | grows with $m$ | no |
| Generative | in $\theta$ | autoregressive | none | 0 | beam search decode | no |
Here $b$ is bytes per scalar, $\bar n_{\text{nz}}$ the mean non-zeros per document, $n = n_q + n_d$, and $m$ the number of codes or candidates. The rank bound column is the single most predictive entry for out-of-domain generalization, and the index bytes column is the single most predictive entry for whether a system survives contact with a production corpus.
The practical answer in 2026 is that these families are stages, not competitors. A production pipeline runs a high-recall, high-rank, cheap retriever first and progressively spends more compute on fewer candidates:
\[\text{Cost} \;=\; F_0 \;+\; \sum_{i=1}^{K} \lvert \mathcal{C}_i \rvert \cdot c_i, \qquad \lvert \mathcal{C}_1 \rvert \gg \lvert \mathcal{C}_2 \rvert \gg \cdots \gg \lvert \mathcal{C}_K \rvert, \qquad c_1 \ll c_2 \ll \cdots \ll c_K\]The critical property is that recall is monotonically non-increasing through the cascade, so stage one imposes a hard ceiling on everything downstream:
\[\text{Recall@}k_K \;\le\; \min_{i=1,\ldots,K} \; \text{Recall@}k_i^{(i)}\]A document dropped by the first-stage retriever cannot be recovered by any reranker, no matter how good. This is why hybrid sparse-plus-dense first stages persist even when a single dense retriever has better mean nDCG: the fused first stage has better recall, which is the only stage-one metric that matters.
The most active frontier is making $g$ arbitrarily expensive on a handful of candidates. Instead of reading a relevance score off a single forward pass, the model samples a reasoning chain $r$ and conditions the judgment on it:
\[s(q,d) \;=\; \log \sum_{r} P_\theta(r \mid q, d)\, P_\theta(\texttt{relevant} \mid q, d, r) \;\approx\; \log P_\theta\big(\texttt{relevant} \mid q, d, \hat{r}\big), \qquad \hat{r} \sim P_\theta(\cdot \mid q, d)\]The sum over chains is intractable, so in practice a single sampled chain $\hat{r}$ stands in for the marginal. This makes $F_g$ depend on the number of generated tokens rather than on the input length, which is a genuinely new cost regime: the interaction operator now has a knob that trades latency for accuracy at inference time, with no retraining. Rank1 (Weller et al., COLM 2025) trains a reranker to emit an explicit reasoning chain before its relevance judgment, reporting state-of-the-art results on the BRIGHT reasoning-intensive retrieval benchmark, the NevIR negation benchmark, and multilingual instruction-following. ReasonRank (arXiv 2508.07050, 2025, not peer-reviewed) extends this to listwise, processing multiple passages under a single reasoning chain and reporting gains of 3 and 5 nDCG points over the prior state of the art on BRIGHT at 7B and 32B scale, while emitting far fewer output tokens than pointwise Rank1.
BRIGHT (Su et al., ICLR 2025) is the benchmark that made this necessary, and its headline finding is a warning about the whole taxonomy: bi-encoders beat BM25 but do not come close to solving it, and the conventional MiniLM cross-encoder reranker actually underperforms the BM25 baseline. When relevance requires multi-step inference rather than semantic similarity, none of the classical interaction operators is expressive enough, regardless of where they place the meeting point. Expressiveness in the sense of the rank bound is necessary but not sufficient; the model also has to be able to reason, and that is a property of scale and training rather than of the factorization.
The complementary direction is making $\mathcal{Z}_d$ smaller without losing quality. Matryoshka representation learning (Kusupati et al., NeurIPS 2022) trains a single embedding whose every prefix is itself a usable embedding, by summing the retrieval loss over a set of nested widths:
\[\mathcal{L}_{\text{MRL}} \;=\; \sum_{k \in \mathcal{K}} c_k \; \mathcal{L}_{\text{rank}}\Big( \big\langle \phi(q)_{1:k},\; \psi(d)_{1:k} \big\rangle \Big), \qquad \mathcal{K} = \{64, 128, 256, 512, 768\}\]where $c_k$ weights each width. One model then serves every dimension in $\mathcal{K}$, and the retrieval system picks a different $k$ per cascade stage without storing separate indexes. Combined with binary or int8 quantization, the standard pattern is to search a 1-bit index for a large candidate set and rescore with full-precision vectors, achieving roughly 32 times index compression with small recall loss. MUVERA generalizes the same move to multi-vector indexes.
The rank bound says a bi-encoder with width $k$ incurs an irreducible error determined by the tail singular values of the true relevance matrix:
\[\mathcal{E}(k) \;=\; \sqrt{\sum_{i > k} \sigma_i^2(S^\star)}\]Nobody has actually measured $\sigma_i(S^\star)$ on a real corpus. Everyone reasons about embedding dimension by ablation, but the quantity that determines the right answer is a spectrum that is estimable, at least approximately, by SVD of a cross-encoder-scored query-document matrix. Until it is measured, “use 768 dimensions” remains folklore.
Cross-encoder to bi-encoder distillation is universal practice, but if the teacher’s score matrix has rank above $k$, the student provably cannot match it. The gap is bounded below:
\[\min_{\text{rank} \le k} \lVert S_T - S_S \rVert_F \;\ge\; \sigma_{k+1}(S_T)\]What is unknown is whether the residual is concentrated in query-document pairs that matter for ranking metrics or spread over pairs nobody sees. If it is the latter, better distillation objectives targeting the head of the ranking could close the practical gap while the Frobenius gap stays open.
Late interaction’s cost is the number of vectors per document. Formally the open problem is a constrained selection:
\[\min_{\mathcal{S} \subseteq \{1,\ldots,n_d\}} \lvert \mathcal{S} \rvert \quad \text{subject to} \quad \mathbb{E}_q\big[\, s_{\mathcal{S}}(q,d) - s_{\text{full}}(q,d) \,\big]^2 \le \epsilon\]ColBERTer and CITADEL attack this heuristically with word aggregation and routing. There is no principled criterion for which token vectors carry ranking-relevant information, and no theory saying how $\lvert \mathcal{S} \rvert$ should scale with document length or corpus size.
MUVERA shows MaxSim can be approximated by an inner product in fixed dimension $D$ with error $\varepsilon$. The dependence is the open question:
\[D \;=\; O\!\left(\frac{f(n_q, n_d, k)}{\varepsilon^2}\right)\]Practitioners need to know how $D$ must grow with document length and desired recall, because that determines whether the technique survives on long documents rather than short passages. The current guarantees are asymptotic and the constants are what decide deployability.
The cascade bound says $\text{Recall@}k_K \le \min_i \text{Recall@}k_i^{(i)}$, but in practice nobody knows how the loss decomposes. The quantity worth measuring is the per-stage marginal loss:
\[\Delta_i \;=\; \text{Recall}^{(i-1)} - \text{Recall}^{(i)}\]subject to $\sum_i \Delta_i = \text{Recall}^{(0)} - \text{Recall}^{(K)}$. Teams routinely spend months improving stage three when $\Delta_1$ dominates, because stage-one recall is measured against shallow judgments that hide the loss.
Modern retrieval is increasingly conditioned on an instruction as well as a query, so the score is $s(q, d, \text{instr})$. The question is whether it factorizes at all:
\[s(q,d,\text{instr}) \;\stackrel{?}{=}\; g\big(\phi(q, \text{instr}),\, \psi(d)\big)\]If relevance depends on the instruction in a way that changes which document properties matter, then $\psi$ would need to depend on the instruction, which destroys cacheability. BRIGHT-style results suggest this is exactly what happens for reasoning-intensive queries, and it is the strongest argument that the offline-encoding paradigm has a genuine ceiling.
Generative retrieval stores the index in $\theta$, so adding a document is a gradient update rather than an append:
\[\theta_{t+1} = \arg\min_\theta \; \mathcal{L}_{\text{new}}(\theta; d_{\text{new}}) + \lambda\, \mathbb{E}_{d \sim \mathcal{D}_{\text{old}}}\big[\lVert P_\theta(\cdot \mid q_d) - P_{\theta_t}(\cdot \mid q_d)\rVert\big]\]The second term is a forgetting penalty over the existing corpus, and evaluating it exactly costs a pass over the whole index. No published method handles corpus churn at the rate a real search system experiences, which is currently the binding constraint on the entire generative retrieval program.
Score a fixed set of 2,000 queries against 20,000 MS MARCO passages with a strong cross-encoder to build $S^\star$, take its SVD, and plot the singular value spectrum. Then train bi-encoders at $k \in {32, 64, 128, 256, 768}$ and check whether the measured nDCG tracks the predicted low-rank approximation error. Report the correlation between the two curves:
\[\rho \;=\; \text{corr}\Big( \text{nDCG}(k), \; -\sqrt{\textstyle\sum_{i>k} \sigma_i^2(S^\star)} \Big)\]Portfolio signal: you can turn a theoretical bound into a measurement and check whether a piece of accepted folklore is actually true.
Take one pretrained encoder and train PreTTR-style rerankers at every split point $\ell \in {0, 2, 4, 6, 8, 10, 12}$ on MS MARCO. Plot the resulting quality against online FLOPs and against index size. Report the Pareto frontier and the knee:
\[\ell^\star = \arg\max_{\ell} \; \frac{\text{nDCG@10}(\ell) - \text{nDCG@10}(L)}{F_g(\ell) / F_g(0)}\]Portfolio signal: you treat architecture as a continuous design variable and can produce a cost-quality frontier rather than a single number.
Implement MaxSim and a token-level index on a corpus of about one million passages. Then ablate storage: full float32, float16, product quantization, and ColBERTv2-style centroid-plus-residual. Report quality per gigabyte:
\[\eta \;=\; \frac{\text{nDCG@10}}{B_{\text{index}} \; [\text{GB}]}\]Portfolio signal: you can implement a non-trivial retrieval operator and reason about the storage-quality trade that decides whether it ships.
Distill one cross-encoder teacher into bi-encoder students at several widths $k$, and into one ColBERT-style student at $k = 128$. Measure how much of the teacher-student gap each recovers:
\[\gamma(k) \;=\; \frac{\text{nDCG}_S(k) - \text{nDCG}_{\text{BM25}}}{\text{nDCG}_T - \text{nDCG}_{\text{BM25}}}\]The interesting result is whether the 128-dimensional multi-vector student beats the 768-dimensional single-vector student, which is a direct empirical test of the rank-bound argument.
Portfolio signal: you design experiments that discriminate between competing explanations rather than just reporting that a model improved.
Given per-stage cost $c_i$ and measured recall curves, choose the cutoffs $k_1, \ldots, k_K$ that maximize final quality within a latency budget $T$:
\[\max_{k_1, \ldots, k_K} \; \text{nDCG@10} \quad \text{subject to} \quad \sum_{i=1}^{K} k_{i-1}\, c_i \;\le\; T\]Solve it by grid search or Lagrangian relaxation, then verify the predicted optimum against measured end-to-end latency on real hardware.
Portfolio signal: you can turn a systems trade-off into a constrained optimization and validate the model against wall-clock reality.
Implement MUVERA-style FDEs, encode a ColBERT index into single vectors of dimension $D \in {1024, 2048, 4096, 8192}$, and measure how well the inner product approximates true MaxSim as $D$ grows:
\[\text{err}(D) \;=\; \mathbb{E}_{q,d}\Big[\big\lvert \langle \text{FDE}_D(q), \text{FDE}_D(d)\rangle - s_{\text{ColBERT}}(q,d) \big\rvert\Big]\]Then plot end-to-end recall against $D$ and find where the approximation stops costing anything measurable.
Portfolio signal: you can read a theory-heavy paper, implement its construction, and empirically characterize constants the paper leaves asymptotic.
Pick a dashed cell from the taxonomy figure and train it. The most promising is sparse representation with partial cross-attention: encode documents into SPLADE-style sparse vectors, retrieve with an inverted index, then feed the sparse activation patterns of query and document into a small joint transformer rather than taking their inner product. Report quality against both parents:
\[\Delta = \text{nDCG@10}_{\text{new}} - \max\big(\text{nDCG@10}_{\text{SPLADE}},\; \text{nDCG@10}_{\text{PreTTR}}\big)\]Portfolio signal: you can read a design space rather than a paper list, identify what is missing, and build it.
On BRIGHT or a similar instruction-following retrieval set, train two systems: one where the instruction is concatenated to the query, so $\phi(q, \text{instr})$, and one where it is concatenated to the document, so $\psi(d, \text{instr})$, which sacrifices cacheability. If the second is substantially better, offline document encoding has a real ceiling for this query class. Report:
\[\delta = \text{nDCG@10}\big(\psi(d,\text{instr})\big) - \text{nDCG@10}\big(\phi(q,\text{instr})\big)\]Portfolio signal: you can design a clean experiment that tests an architectural assumption directly instead of benchmarking around it.
This survey traced a single design axis from Spärck Jones’s inverse document frequency in 1972 to reasoning rerankers in 2025, and the conclusion is that bi-encoder, cross-encoder, and late interaction are three cells in a twenty-five cell grid whose coordinates are what the document encoder is allowed to emit and how expressive the interaction operator is allowed to be. The frontier work appears at SIGIR, NeurIPS, ICLR, ACL, and EMNLP, with the systems-heavy contributions concentrated at SIGIR and CIKM and the newest reasoning-reranker work at COLM. For live signals, watch the BEIR and MTEB leaderboards for the single-vector frontier, BRIGHT for the reasoning-intensive frontier where every classical interaction operator currently fails, and the TREC Deep Learning track for the cascades that actually get deployed.