Office Hours — Our RAG system's retrieval accuracy dropped 40% when we scaled from 50K to 2.68M documents. Is this a vector database problem, a ranking problem, or something else? A daily developer question about AI/LLMs, answered with a direct, opinionated take. 2026-11-13T12:00:00.000Z Office Hours Office Hours office-hoursq-and-apractical-ai

Office Hours — Our RAG system's retrieval accuracy dropped 40% when we scaled from 50K to 2.68M documents. Is this a vector database problem, a ranking problem, or something else?

A daily developer question about AI/LLMs, answered with a direct, opinionated take.

Daily One question from the trenches, one opinionated answer.

Our RAG system’s retrieval accuracy dropped 40% when we scaled from 50K to 2.68M documents. Is this a vector database problem, a ranking problem, or something else?

The Real Issue

You’re not looking at a database problem. You’re looking at a fundamental limitation in how retrieval systems degrade under scale. The research is sobering: BM25 and embedding-based retrieval both lose precision catastrophically as context grows, and the position of the answer in your ranked list matters far more than your embeddings suggest. This is reproducible across multiple systems.

When you had 50K documents, your top-5 likely contained the right answer most of the time. At 2.68M, you’re competing against so much noise that even perfect embeddings lose signal.

Why Vector Databases Aren’t the Culprit

Your database is probably fine. Qdrant, Pinecone, pgvector, and Weaviate all handle scale reasonably. The problem is the retrieval algorithm itself. Embedding-based search works by finding cosine similarity in a learned space. That space was optimized on your training data. Once you add 2.6 million documents you didn’t have at training time, you’ve massively expanded the surface area for false positives. A document about, say, “database query performance” sits close to another about “database index tuning” in embedding space, even if they’re not what you need.

BM25 (keyword search) has its own failure mode at scale: it gets drowned in documents that contain all your query terms but are about something else entirely.

What Actually Works: Hybrid Search Done Right

Hybrid search—combining dense (embedding) and sparse (BM25) retrieval—helps, but most people implement it wrong. They treat it like a simple union or average of scores. That doesn’t work.

The correct pattern, per recent production benchmarks, is to generate two good candidate lists independently, then rerank the union:

# Wrong: just average scores
scores = 0.6 * embedding_score + 0.4 * bm25_score

# Right: get top-K from each, then rerank
embedding_candidates = vector_index.search(query, top_k=100)
sparse_candidates = bm25_index.search(query, top_k=100)
combined = deduplicate(embedding_candidates + sparse_candidates)
reranked = rerank_with_cross_encoder(query, combined)
return reranked[:10]

A cross-encoder (a smaller, specialized model) rescore the candidate set is critical. It gives you a unified relevance score across both retrieval methods. Open-source options like ms-marco-MiniLM-L-12-v2 work here—you’re not paying frontier model prices.

The Position Problem

Here’s what the benchmarks found: answer position matters more than you think. If your document is ranked 47th instead of 3rd, the LLM might not see it even if your context window is 128K. Many systems use the top-K straightforwardly, but actually integrating evidence from documents ranked 20–50 helps. Some teams use a “sliding window” approach where they ask the LLM to extract relevant chunks from all ranked results, not just the first few.

Practical Cost: What to Do This Week

Start with a reranker. Add one line to your pipeline:

from sentence_transformers import CrossEncoder

model = CrossEncoder("ms-marco-MiniLM-L-12-v2")
scores = model.predict([(query, doc) for doc in candidates])
reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)

Cost: ~50ms per query on CPU, and you can batch. This often recovers 10–15 points of accuracy without changing your vector database.

If that doesn’t move the needle, you need to audit your retrieval independently. Run a sample of 100 queries, manually check whether the ground-truth document is in your top-50 results at all. If it’s not there, no amount of reranking fixes it. Then your embeddings, chunking strategy, or query expansion is the issue, not scale.

What Not to Do

Don’t blame the vector database and start shopping. Don’t assume you need a larger embedding model (you might, but prove it first). Don’t switch to agentic retrieval expecting it to solve this—agents still call the same ranking algorithms, they just call them more times.

Bottom line: Your accuracy drop is retrieval degradation under scale, not database failure. Add a cross-encoder reranker to your pipeline immediately and measure recall before reranking vs. after. That one addition will likely recover most of your 40-point loss without infrastructure changes.