The switch happened without a formal architecture review. It happened in a planning conversation, or in a postmortem on a retrieval system that had disappointed, or in a developer's read of a model provider's announcement about million-token context windows. The reasoning was available and felt compelling: the documents were there, the context could hold them, and the retrieval layer was the component that broke most often and required the most specialist judgement to tune. Remove the bottleneck. Put the documents in the context.
We have nine production deployments where we can now describe what happened when that reasoning was followed. The deployments span legal review, financial services, insurance underwriting, and internal knowledge retrieval in professional services. Two of the nine are performing as designed and have not required architectural revision. Seven are rebuilding, or carrying cost and latency profiles that do not survive contact with their own volume projections. In five of the seven, the decision to rebuild was not triggered by accuracy failure — the systems found the right information — but by a cost structure that became visible only when user adoption reached the levels the original business case had projected.
This piece is about those seven, what they share, and what the two that are holding have that the seven do not. The difference is not in model quality. It is in task structure, corpus size, and a question that was not asked before the architecture was committed.
#02What the evaluation found, and what it could not measure
In all nine cases, the decision was preceded by a serious evaluation. The teams involved were not cutting corners. They assembled labelled evaluation sets, ran comparisons between the long-context approach and the retrieval system it was replacing, and made the architecture decision on the basis of the results. On the dimensions the evaluations measured — retrieval accuracy, answer quality as assessed by domain reviewers, completeness against a rubric — the long-context approach performed comparably to or better than the retrieval baseline in seven of the nine cases. The recommendation looked well-grounded.
What the evaluations did not measure was cost per query at production volume, latency at production scale, or the operational consequences of a knowledge corpus that cannot be updated incrementally. These were not obvious omissions at evaluation time. They followed from how evaluations are assembled: against a fixed corpus, at a query volume that represents a small fraction of forecast production load, at latencies that are acceptable for a reviewer working through a test suite. An evaluation run on 200 queries over two days does not expose what 8,000 queries per working day looks like when each query carries 400,000 tokens of input context. It is not a failure of the evaluation methodology. It is a failure to extend the evaluation's outputs to the system's operating conditions.
The evaluation corpus was also assembled in ways that systematically favoured the long-context approach. Teams reaching for difficult queries to test the retrieval system's limits selected questions whose answers were distributed across multiple document sections — precisely the query class where chunk-based retrieval struggles and long-context inference holds the full document coherently. Straightforward lookups, single-section extractions, and queries with retrievable answers in the first few paragraphs of the relevant document were underrepresented. The evaluation was honest about what it found. It was not representative of what production would ask.
#03Three failure modes that do not appear in demos
Long-context inference at production scale fails in three ways that retrieval architectures are specifically designed to avoid. All three were present in at least four of the seven struggling deployments. None appeared in the evaluations that preceded the architecture decision.
The first is cost scaling. Retrieval augmentation is designed to minimise the token footprint of each inference call — retrieve the relevant passages, not the full corpus, and respond from a context of 2,000 to 8,000 tokens rather than 400,000. Long-context inference substitutes token processing for retrieval design, and input token cost scales linearly with context length while query volume scales with user adoption. In the two deployments where we have pulled billing records against the original forecasts, the annualised inference cost of the long-context architecture was 4.3 and 5.7 times the projected cost of the retrieval baseline at the same query volume. Neither team had multiplied the per-query token cost by the daily query count before the architecture was committed. The per-query number looked acceptable. The annual total did not.
The second failure mode is prefill latency. Retrieval architectures add latency in the retrieval step — typically 180 to 750 milliseconds for a well-implemented vector retrieval, reranking, and context assembly pipeline. Long-context inference adds latency in the prefill step, during which the model attends to every input token before generating a single output token. At 400,000 input tokens, time-to-first-token on current frontier models under normal production load ranges from approximately four to fourteen seconds, depending on the provider and the inference tier. That is not a configuration problem — it is a property of the architecture at that token count. Usage instrumentation added after launch across three deployments showed that between 34 and 51 per cent of users had navigated away from the interface before receiving a response. They returned for the results. They stopped using the tool for anything that felt time-sensitive.
The third failure mode is knowledge staleness. A long-context inference system's knowledge is bounded by what is in the context at query time. When new documents arrive — a policy revised in April, a regulation updated in May, a supplier contract replaced in June — the context must be regenerated to include them. That regeneration is not a continuous process; it is a scheduled job with a cadence that must be designed, operated, and monitored. Three organisations that chose long-context inference specifically to reduce the operational complexity of their retrieval layer discovered, at the first governance review, that they had traded index management for corpus versioning. The operational discipline required is different in character but comparable in cost.
“The cost of 400,000 input tokens at production query volume is calculable before the architecture is committed. In all five deployments that are now rebuilding, it was not calculated. The evaluation had a per-query cost. Nobody multiplied it.”
#04What the two that held their value have in common
The two deployments where long-context inference replaced retrieval durably share a structural feature: the task required the model to reason across information distributed through the document in ways that chunk-based retrieval cannot preserve, the corpus was bounded in size, and the query volume was low enough that the per-query economics were sustainable. The three conditions together. Not one or two of them.
The first was a transaction due diligence workflow processing long-form corporate documents — share purchase agreements running to 160 to 220 pages, where definitions introduced in section two modify indemnity provisions in schedule seven, and where the practical effect of a liability cap depends on reading clause fourteen against an exception in schedule three that a retrieval system chunked on section boundaries will not present together. The task required holding cross-document relationships across the full text; chunk-based retrieval destroyed the specific context that determined the correct answer. Query volume was low — fewer than 400 per working day across a team of eleven — and the documents were high-stakes enough that a per-query cost of several pounds was plainly justified. Long-context inference was the architecturally correct choice, not because context windows are large, but because the task structure and economics supported it.
The second was an insurance underwriting system processing risk profiles that ran to 50,000 to 110,000 tokens per risk, with temporal relationships across the full profile relevant to the underwriting decision in ways that required the full document to be present. The active corpus was small — fewer than 600 risks under management at any time — and the query volume was under 600 per working day. The cost and latency profile was acceptable for a workflow where the output supported decisions carrying six to eight figures of premium exposure. The system works because the conditions for it to work are present.
In none of the seven struggling deployments were all three conditions met. Query volumes ranged from 3,400 to 14,000 per working day. Corpus sizes were measured in tens of thousands of documents. And retrospective query classification exercises found that between 58 and 79 per cent of actual production queries could have been served from a focused retrieved passage without cross-document reasoning. Those queries were paying for context they did not require.
#05What the alternative actually looks like
The retrieval architectures that long-context inference was replacing were, in many of these cases, genuinely poor. Chunking strategies that split clauses across boundaries. Cosine similarity retrieval without reranking, returning passages that were semantically adjacent to the query but contextually wrong. Prompts that assembled retrieved context without instructing the model in how to use it. The frustration that preceded the switch to long-context inference was honest, and the diagnosis — that the retrieval layer was the source of quality problems — was often correct. The error was in the conclusion: that the right response to a poorly implemented retrieval system is to remove the retrieval layer rather than build a better one.
In four of the seven struggling deployments, we ran a parallel evaluation: the same query set, the same corpus, and a properly implemented retrieval architecture — smaller and more semantically coherent chunks, a cross-encoder reranking step, and structured prompts that explicitly directed the model's use of retrieved context. On 79 to 86 per cent of the query population, the properly implemented retrieval system matched or exceeded the long-context baseline on domain expert quality assessments. Mean latency across the full query population ranged from 590 to 840 milliseconds, against 6.4 to 11.2 seconds for the long-context baseline. Cost per 1,000 queries was 17 to 23 per cent of the long-context equivalent. The accuracy gap the switch had been intended to close had been largely a prompting and chunking gap rather than a retrieval one.
The architecture that serves the residual 14 to 21 per cent of queries — those with genuine cross-document reasoning requirements — is a routing classifier that identifies the query structure before choosing an inference path, directing the minority requiring full-document context to long-context inference while serving the majority from the retrieval path. In the two deployments where this routing layer has been retrofitted, mean query cost fell by 67 and 71 per cent respectively against the all-long-context baseline, with no degradation in output quality as assessed by the domain teams that use the system daily. It is the same architectural discipline we described for the extended thinking question in the piece before this one — a lightweight classifier gating an expensive inference path. It does not require a long-form architecture review to implement. It requires someone to have asked the routing question before the default was committed.
#06A note on the direction of travel
Inference pricing for long contexts is falling, and the prefill latency of very large contexts is improving as model architectures and serving infrastructure advance. The specific numbers described above — the 4.3 and 5.7 cost multipliers, the 6.4 to 11.2 second latency figures — will look different in twelve months. Some of what is currently a hard constraint will become a softer one.
The freshness constraint will not move with pricing. A long-context inference system whose context is assembled from a document corpus cannot incorporate new information without regenerating the context. An organisation that wants its system to know about a regulatory change, a product update, or a revised contractual term must either reprocess or accept stale context. This is a property of the architecture rather than a cost that tracks with provider pricing, and it will remain a material reason to maintain a retrieval layer in any knowledge domain that changes — which is to say, almost every enterprise knowledge domain.
The subtler thing that will not change is the design discipline retrieval requires. To build a retrieval system, a team must specify what information the model needs before it can retrieve it — which demands clarity about what the query class is and what an adequate answer looks like. That specificity tends to produce better prompts, more predictable output distributions, and more debuggable failure modes. Long-context inference, by removing the retrieval constraint, also removes the design accountability it imposed. The teams that have switched back to retrieval in these deployments have, without exception, reported that the process of redesigning the retrieval layer produced a better understanding of the problem than the long-context implementation had done. That is not a finding you can put in an evaluation report. It is worth noting anyway.