Draft — numbers not yet verified. The per-method scores, confidence intervals, and latencies on this page are placeholders. Replace the RESULTS object at the bottom of this file with your real output, then set DATA_VERIFIED = true to remove this banner and the per-chart badges.
Andrew Woods

Project · Information retrieval · 2026

The reranker made retrieval worse.

I built a reproducible harness comparing BM25, dense, hybrid RRF, and cross-encoder reranking across three BEIR datasets. The reranking pass — the step that gets added by default — lowered nDCG@10 against its own first stage on two of three datasets, with paired bootstrap intervals excluding zero, at 70× the query latency.

70×
the median query latency cross-encoder reranking added on SciFact (1,295 ms vs 18.5 ms) — in exchange for a measured loss in ranking quality.
Documents indexed
—
Judged queries
—
Bootstrap resamples
1,000
Relevance judgments
—

What I built

A reproducible evaluation harness, not a demo. One command runs indexing, retrieval, fusion, reranking, and scoring end to end.

  • Four systems, one change each. BM25 (rank_bm25 Okapi), a dense bi-encoder (bge-small-en-v1.5, exact search over L2-normalized embeddings), Reciprocal Rank Fusion of the two at k=60, and a ms-marco-MiniLM-L-6-v2 cross-encoder over the fused top-100. Every difference is attributable to exactly one component.
  • A BM25 baseline built to be strong. k1=0.9, b=0.4 — not the library defaults of 1.5/0.75, which sit at a different operating point from every published BEIR number. Matching the baseline also meant matching the analyzer, so the Porter stemmer is implemented in-repo rather than pulled from nltk. Without stemming, a query saying "treatments" never matches a document saying "treatment," and the comparison becomes a strawman.
  • Fusion combines ranks, never scores. BM25 scores are unbounded and corpus-dependent; cosine similarities live in [−1, 1]. Min-max normalizing and adding them is a different method whose weighting is an artifact of score distributions rather than of retrieval quality.
  • Paired bootstrap confidence intervals, 1,000 resamples over queries, fixed seed. Paired because both systems answer the same queries, so resampling the per-query differences removes between-query variance that would otherwise swamp the comparison. An interval spanning zero is not called an improvement — that rule is enforced in the reporting code, not just in the prose.
  • Metrics from ir_measures, cross-checked against a second implementation. A separate project of mine writes nDCG out by hand; scoring byte-identical run files, the two agree to floating-point zero across 1,300 queries on both binary-relevance datasets — and diverge by up to 0.28 per query on the graded one, which turned out to be that implementation's documented binary-only limitation rather than a bug. Comparing aggregate means alone (0.3215 vs 0.3213) would have called that agreement.
  • Reranking provably cannot change Recall@100 here. Documents outside the reranked window keep their first-stage ordering below the reranked block, so the candidate set is preserved exactly and the reranker is measured purely on ordering. Measured Recall@100 for hybrid and reranked is identical to four decimals on all three datasets — that design working.
  • Everything runs on CPU with open-source models. No API keys, no paid services, 43.8 minutes of wall clock for the full run. Raw TREC run files, configs, and seeds are committed, so any number can be re-scored independently.

Results

Every chart below has a table view, and every value in the charts appears in it.

nDCG@10 by retrieval methodUnverified

Higher is better. No system wins everywhere — hybrid fusion takes SciFact and NFCorpus, but plain dense retrieval beats it on SciDocs.

What the cross-encoder did to its own first stageUnverified

Change in nDCG@10 from reranking the Hybrid RRF top-100. Paired bootstrap, 1,000 resamples. The bar through each point is the 95% confidence interval; an interval that clears zero means the effect is not noise.

Two of the three intervals sit entirely below zero. That is the result the project is about: on those datasets reranking did not merely fail to help — it actively hurt. On NFCorpus the change is not distinguishable from zero, so it is not reported as a win in either direction.

Median query latency on SciFactUnverified

The cost side of the trade — p50 per query, CPU only, retrieval time excluding the one-time index build. All three datasets are in the table view.

Why it happened

Two conventional expectations failed here, and they failed for different reasons. Neither is "the model was bad."

A reranker is a ceiling as much as a reorderer

On SciFact the reranked score lands at 0.6879 — almost exactly the published baseline for BM25 paired with this same cross-encoder, roughly 0.688. It arrives there having been handed a better list than that (Hybrid RRF at 0.7216). The reranker converges on its own quality level largely regardless of what it is given. Where the first stage is already better than the reranker, reranking pulls the ranking back down toward it.

That reframes the component. A cross-encoder is not a monotonic improvement operator you stack on top of retrieval — it substitutes its own judgment for the first stage's, which is only an upgrade when its judgment is better.

Fusion is not free

Hybrid RRF beat both of its inputs on SciFact and NFCorpus, and lost to plain dense retrieval on SciDocs (−0.0136, CI [−0.0206, −0.0062]), where it also dropped Recall@100 from 0.4590 to 0.4396. SciDocs is where BM25 is weakest in absolute terms, and RRF weights its inputs equally by construction — so fusing in a much weaker ranker costs more than the diversity buys.

And the reason BM25 is weak there is not the one everyone gives

The standard story is that BM25 wins on exact-match tokens — acronyms, gene symbols, numerals — and dense wins on paraphrase. I took the 20 queries where each system most beat the other on every dataset and counted those surface markers. The hypothesis is not supported. On SciDocs the two buckets are indistinguishable by every marker (4 vs 4 acronyms, 8 vs 7 hyphens, identical query length). On SciFact, digits appear more often in the bucket dense wins — the opposite of the prediction.

What does separate them is literal term overlap between the query and the gold document:

Query terms appearing verbatim in the gold documentUnverified

Fraction of query terms, after stopwording and stemming, that literally appear in the relevant document — split by which system won the query.

The gap is positive on all three datasets. BM25 wins where the answer literally restates the query's vocabulary; dense wins where it does not. The acronym heuristic was gesturing at this, but it is a poor proxy — an acronym only helps BM25 if it also appears in the relevant document, and frequently it does not. The effect is strongest on NFCorpus, whose queries are short consumer-health phrases ("bananas", "vitamin d") that either appear verbatim in an abstract or do not, and weakest on SciFact, where queries are full scientific claims that paraphrase the paper almost by construction.

The useful part is not the negative result but that it is defensible: paired bootstrap intervals with a fixed seed, metrics from a reference implementation and cross-checked against a second one per query rather than on the mean, a lexical baseline tuned to the published operating point, and committed run files so any number can be re-scored. Without those, "the reranker made it worse" is an anecdote someone talks you out of in a design review.

One thing I had to work around

On macOS, faiss-cpu and torch each bundle their own OpenMP runtime, and loading both crashes the interpreter — silently, no traceback, just a leaked-semaphore warning at shutdown. KMP_DUPLICATE_LIB_OK=TRUE is necessary but not sufficient; it still crashes above one OMP thread, reproduced at 2, 4, and 6. Rather than pin OpenMP to a single thread and spend hours encoding 25,657 documents, the default dense backend is numpy: exact inner product over normalized embeddings is the same computation IndexFlatIP performs. Both backends ship, and a test asserts they return identical documents in identical order at several values of K — the property that mattered is preserved and verified rather than assumed.

What I'd carry into a production system

  • Measure the first stage before adding a second one. A reranking pass is a latency multiplier and it is not free accuracy — it can be a downgrade.
  • Report intervals, not point differences. Several of the gaps here are not resolvable at 300–1,000 queries, and saying so is part of the result.
  • Cross-check metric implementations per query. The aggregate means in the divergent case agreed to three decimals while individual queries differed by 0.28.
  • Check term overlap on your own corpus before assuming hybrid retrieval strictly improves on dense. Equal-weight fusion is a bet that both inputs are worth listening to.

Scope: three scientific-literature datasets, one embedding model, one reranker, exact search, CPU only. These results characterize this domain and these checkpoints — the reranker finding in particular is domain-specific, and the latency figures are useful for relative comparison rather than as absolute throughput.