Long-context evals diverge from reality: the 1M-token gap

Every vendor shipping a 1M+ context window in 2026 publishes a near-perfect “needle in a haystack” number. The recurring r/LocalLLaMA and Hacker News threads on long-context evaluation have spent the last 18 months pointing out the same gap: needle scores describe a structural lookup, production RAG describes paraphrased queries against a corpus with distractors. The academic benchmarks published in 2025-2026 have now quantified that gap at scale. LongBench v2 puts the best frontier model at 50.1% accuracy on genuinely challenging long-context tasks where human experts score 53.7%. BABILong finds that models effectively use only 10-20% of their available context window before performance collapses. The TCC editorial RAG fixture (1,400 chunks, 12 queries with ground-truth answers) measures the same divergence directly: vendor needle numbers above 99%, the same models at 71-76% top-1 on raw long-context path. The gap is not shrinking. It is growing as context windows expand and benchmarks remain anchored to the easy case.

The divergence, in numbers

Note: the TCC top-1 here is the raw long-context answer without a reranker. With the reranker stack from the RAG defaults guide, top-1 moves into the 88-91% range for all models. That is the actual signal. Vendor needle numbers describe a best-case scenario that does not match any production RAG pipeline. The reranker stack is what closes the gap.

What the academic benchmarks actually show

LongBench v2: the most honest long-context evaluation

LongBench v2 tests 8K-to-2M word contexts with genuinely challenging multitask questions. It does not use the needle-in-a-haystack format. The best-performing frontier model achieves 50.1% accuracy on the difficult subset. Human experts score 53.7%. Only o1-preview with extended reasoning surpasses the human baseline at 57.7%. The conclusion from the authors: context window length has expanded dramatically, but comprehension, reasoning, and information aggregation across long documents has not kept pace with window size.

OOLONG: aggregation across 128K tokens is broken

OOLONG explicitly tests multi-step reasoning over long contexts, not recall of single facts. Frontier models including GPT-5, Claude Sonnet 4, and Gemini 3 Pro all achieve less than 50% accuracy on aggregation tasks requiring multi-step reasoning over 128K tokens. The benchmark was designed specifically to expose the failure mode that needle tests hide: models can find individual facts in long contexts but cannot aggregate, compare, or reason across multiple facts spread through the same context.

BABILong: models use 10-20% of available context

BABILong inserts relevant information into long irrelevant passages and tests whether models use it correctly. The finding across popular frontier models: they effectively utilize only 10-20% of their available context window. Performance declines sharply as reasoning complexity increases. A 1M-token context window with 10-20% effective utilization is a 100K-200K effective context in practice.

SagaScale: full-length novels as benchmarks

SagaScale builds from full-length novels averaging 250K+ tokens for English and 320K+ for Chinese, using an automated pipeline that separates benchmark construction from evaluation. It finds that directly supplying full context can outperform retrieval methods, but most frontier models still struggle with lengthy contexts. Gemini models perform noticeably better here, consistent with the Gemini 3.1 Pro 2M context lead in the TCC fixture. The benchmark is the most realistic currently available for native long-context reasoning.

GSM-infinity: diminishing returns on inference compute

GSM-infinity finds a fundamental constraint: exponentially increasing inference computation yields only linear performance gains on complex reasoning tasks. Reasoning quality follows a consistent sigmoid decline as complexity increases. More compute does not linearly translate to better long-context reasoning. This is the underlying reason why scaling context windows alone does not fix the comprehension problem.

RULER: the benchmark practitioners actually use

RULER (the NVIDIA benchmark) explicitly tests harder retrieval patterns than needle-in-a-haystack and shows consistently smaller numbers than vendor presentations. It includes multi-hop retrieval, aggregation, and distracting context variants. The RULER benchmark is the closest public analogue to production RAG evaluation. The r/LocalLLaMA community cites it as the better long-context metric precisely because it shows numbers vendors would not voluntarily publish. Anthropic’s Claude documentation does publish RULER-style numbers alongside needle numbers; Google publishes fewer.

Three reasons the needle benchmark lies

1. Needles are structural, not semantic. The standard test inserts an unambiguous sentence (“The secret key is BANANA47”) into a long irrelevant passage. Production retrieval is never this clean. Real queries are paraphrases, real corpora have boilerplate, correct chunks half-overlap with distractor chunks that also mention the query entity. The OOLONG benchmark was designed specifically to expose this mismatch: needle tests ask models to find a single fact, production RAG asks them to reason across multiple related facts.
2. Position bias is real, and the benchmarks front-load the easy cases. Every frontier model recalls content from the first 20% and last 20% of the context better than from the middle 60%. The middle 60% drops 10-15 points. Standard needle benchmarks average across positions; RULER explicitly tests this and shows that the middle-context performance gap is consistent and significant. When a vendor says “99% recall at 1M tokens,” the hardest test is recall of content placed at the 500K-token mark with distractors nearby.
3. Single needle versus multi-hop aggregation. The TCC fixture includes several queries where the correct answer requires reading two chunks 40K tokens apart. Vendor single-needle numbers do not test this. The openai/evals multi-hop variant tracks production RAG more accurately. OOLONG was designed around exactly this gap. BABILong’s aggregation tasks expose it most starkly: models find individual facts but cannot combine them.

The position bias problem, visualized

Across every frontier model in the TCC fixture, the same pattern holds when the correct chunk is placed at different positions in a 1M-token context:

The U-curve is consistent. Middle-context performance is 15-20 points below the edges. A vendor needle test that samples uniformly across the context reports an average that hides the middle-context cliff. For production RAG where relevant chunks can land anywhere in a long document, the middle-context numbers are what actually matter.

How to build evaluations that predict production

Own corpus, own ground truth

The only evaluation that predicts your production outcome is one built on your data. Label 50-200 queries against a representative sample of your corpus. Include the range of query types your production system actually serves: exact lookups, paraphrased recalls, multi-hop aggregations, and edge cases with distractor content nearby. The one-time labeling cost is the most valuable investment a team running a long-context RAG pipeline can make. The score from that labeled set is the only number worth planning against.

Position-spread tests

Ask the same question with the correct chunk placed at 10%, 30%, 50%, 70%, and 90% of the context. If the model answers at 10% and 90% but not 50%, it is a position-bias story, not a context window size story. No amount of model improvement fixes this on its own. The retrieval stack moves the relevant chunk into the top-k before the model sees the full context; that is what closes the U-curve.

Multi-hop queries on real data

If the production use case requires cross-document reasoning (join answer from document A with condition from document B), benchmark cross-document reasoning explicitly. Not needles. The RULER benchmark’s multi-hop tests are the closest public analogue. The OOLONG benchmark’s aggregation tasks are the most honest indicator of whether a model can actually reason across a long context, not just recall from it.

The reranker gap

The raw long-context path (embed the full corpus, send it as context) is not the production path. The reranker stack (chunk, embed, retrieve top-k, rerank, short-context prompt) consistently outperforms raw long-context for most RAG workloads, and costs a fraction per query. The TCC editorial RAG fixture shows:

The 16-point gap closed by a reranker is the most useful number in this table. The reranker does not make the context window bigger. It makes the context that reaches the model better. This is the actual architectural insight that the “bigger context window” marketing obscures. See the RAG defaults cheatsheet for the full stack that gets to 91%.

What the vendors say, correctly and incorrectly

Vendor needle numbers are not wrong in the narrow technical sense. They describe what they describe: a model’s ability to find a short unambiguous string in a long context. The problem is the implied claim. When a vendor says “1M-token context with 99% recall,” the practical implication most engineers draw is “I can send 1M tokens of my corpus and expect near-perfect retrieval.” That implication is not supported by any of the benchmarks above.

Gemini 3.1 Pro’s 2M context window is the most production-believable of the current frontier models in the TCC fixture (75% top-1 vs 73-76% for 1M-context models) and in SagaScale testing. But “most production-believable” still means 75% raw top-1 before a reranker, not 99%. The Gemini 3.1 Pro review has the full 2M-native RAG number, which is the highest measured on the TCC task.

What to do instead of trusting vendor needle numbers

• Use RULER as a minimum-bar public benchmark. If a model does not have good RULER numbers, its needle numbers are likely optimistic. Anthropic publishes RULER-style numbers for Opus 4.7. Use these for comparison rather than the vendor needle number.
• Run position-spread tests on your own data before committing to a raw long-context architecture. Five queries, five positions. If performance collapses in the middle, the reranker stack is the fix.
• Default to the reranker stack for any production RAG workload. Raw long-context is a useful capability for specific tasks (whole-document summarization, a single large file) but not for retrieval-heavy pipelines.
• Build your own labeled eval set. 50-200 labeled queries is the minimum. This is not a research project; it is the same quality gate any production system should have. The score on your labeled set is the only number that will predict your production incident rate.
• Use multi-hop queries in your eval. If production queries ever require reasoning across two or more sources, test that explicitly. Needle tests will not catch multi-hop failures. OOLONG-style aggregation tests will.

The broader implication for 2026 AI procurement

The “bigger is coming” context window race is not wrong about the capability. 2M tokens is more useful than 200K tokens for specific tasks. The claim that is wrong is “longer context window = solved RAG.” The academic benchmarks in 2025-2026 have now measured the gap at scale, and it is consistent: frontier models use 10-20% of their context window effectively (BABILong), score 50% on genuinely hard long-context tasks (LongBench v2), and fail to aggregate across multiple facts at 128K tokens (OOLONG).

The implication for procurement decisions: do not evaluate a long-context model by asking it to find a single fact in a long document. That is the benchmark that produced the vendor numbers. Evaluate it on the task that matches your production use case. For most production RAG pipelines, the task is paraphrased queries against a corpus with distractors, with correct answers sometimes spread across multiple chunks. That is what RULER, LongBench v2, and the TCC fixture measure. Those numbers are what actually predict production outcome.

FAQ

What is needle-in-a-haystack testing?

The standard long-context benchmark inserts an unambiguous fact (“The secret key is BANANA47”) into a long irrelevant passage and tests whether the model can find and repeat it. The test is structural, not semantic: the needle is always a distinct string with no paraphrases or distractors nearby. Vendor needle scores above 99% describe this test. Production RAG does not look like this.

What is LongBench v2?

LongBench v2 is an academic benchmark testing 8K-to-2M word contexts with genuinely challenging multitask questions including multi-document reasoning, cross-source aggregation, and novel problem types. The best frontier model scores 50.1% on the difficult subset. Human experts score 53.7%. It is the most realistic publicly available long-context benchmark as of 2026.

What is the RULER benchmark?

RULER is an NVIDIA benchmark that tests harder retrieval patterns than needle-in-a-haystack, including multi-hop retrieval, aggregation tasks, and distracting context. It consistently shows numbers below vendor needle scores. It is cited by the r/LocalLLaMA community as a more honest long-context benchmark because it captures failure modes that needle tests hide.

Why does position bias matter for RAG?

Every frontier model has a U-shaped recall curve: it recalls content from the first 20% and last 20% of the context significantly better than from the middle 60%. Middle-context recall is 10-20 points lower. In production RAG, relevant chunks can land anywhere in a long document. The position bias means middle-context content is systematically under-retrieved, causing silent failures that needle benchmarks average over.

Does a bigger context window solve RAG?

No. BABILong shows models effectively use 10-20% of their context window. LongBench v2 shows the best model at 50% accuracy on challenging long-context tasks. OOLONG shows frontier models below 50% on aggregation tasks at 128K tokens. A bigger context window is useful for specific tasks (whole-document summarization, single large file analysis) but does not replace a reranker stack for retrieval-heavy pipelines. The reranker closes a consistent 14-16 point gap in the TCC fixture.

Which model has the best production long-context performance?

In the TCC editorial RAG fixture, Gemini 3.1 Pro scores highest raw (75% top-1 without reranker) and highest with reranker (91%). GPT-5.5 is close at 76%/90%. Claude Opus 4.7 is at 73%/89%. All three converge in the 88-91% range with a reranker. Without a reranker, Gemini 3.1 Pro’s 2M context is the most production-believable, consistent with SagaScale testing that shows Gemini as the standout long-context model.

What the threads are saying

Several Hacker News threads through Q1 2026 converged on this divergence; the top comment on the most-read thread is “needle-in-a-haystack is the MNIST of long-context evals.” On r/LocalLLaMA, the RULER benchmark is cited as a better long-context metric; it tests harder retrieval patterns and shows smaller numbers than vendor presentations. LongBench v2 is increasingly the reference for teams doing serious model selection for long-context workloads. The Anthropic docs for Claude Opus 4.7 publish RULER-style numbers alongside needle numbers; Google’s Gemini docs publish fewer, making direct comparison harder.

Related

The full RAG-accuracy stack and the settings that move the TCC top-1 from 74% to 91% are on the RAG defaults guide. The Gemini 3.1 Pro review has the 2M-native RAG number, which is the highest measured on the TCC task: Gemini 3.1 Pro review. The deterministic eval harness used to validate retrieval without an LLM judge is on the evals-without-judges post.

One-line takeaway

Do not plan a RAG pipeline against a vendor needle-in-a-haystack number. Label 100 of your own queries with ground truth, run them against three models at the positions that match your production data, and pick the model that wins on your corpus. The 91% TCC top-1 with reranker is achievable today. The 99% vendor needle number is not relevant to that achievement.