Office Hours — How should I handle long-context inference when working with models that have 10M+ token windows?

How should I handle long-context inference when working with models that have 10M+ token windows?

Long context is a real capability now. Gemini 3.7 Flash ships with 1M tokens natively, Anthropic’s Claude Opus 5 handles it, and open models like Kimi K3 and Llama 4 Scout push toward 10M. The problem isn’t whether you can stuff a massive amount of text into the input anymore—it’s whether you should, and how to structure it so the model actually uses the information coherently instead of drowning in it.

The actual bottleneck isn’t context length, it’s coherence

Models don’t uniformly attend to information across 10M tokens. Research from the UK AI Security Institute (AISI, July 2026) measured this directly: success rates on software-engineering tasks rise roughly 25% when test-time token budgets grow tenfold (1M→10M), but that’s not linear. The model pays attention better near the start and end of context (the “lost in the middle” problem is real), and intermediate details get compressed or ignored. Filling a 10M window doesn’t guarantee the model will read it all meaningfully.

More importantly, longer context means longer latency and higher cost. Frontier model pricing is per-token input and output. Running Claude Opus 5 at 10M tokens input is exponentially more expensive than at 200K. Even with cached tokens (where Anthropic charges 10% of the input rate), the economics only work if you’re reusing massive context windows across multiple requests.

Structure context as a compiler problem, not a dump

The most effective teams think about prompt construction as context compilation: deciding what to include, what to exclude, and how to order information to maximize model utilization within whatever window you’re actually using. This often beats waiting for longer contexts.

Before you reach for 10M tokens, ask: Do I actually need all 10M, or do I need the right 500K? Cursor’s agent architecture (separating planning from execution) shows that cheaper models can execute fine-grained tasks when a frontier model plans the work upfront. Apply that thinking to context: use a frontier model’s 10M capacity to reason about what a cheaper model needs to know, then pass the distilled subset.

A concrete pattern: if you’re building an agent over a codebase, don’t dump the entire repo into context. Instead, extract the call graph relevant to the task, rank functions by call distance from the task boundary, and pass the top 50 with their full implementation. That’s maybe 50K tokens for a million-line codebase. The model reasons better on structured, relevant context than on noise.

Cache aggressively when you’re committed to long context

If you are going to use long-context inference (say, processing a full research paper or a year of customer support logs), use prompt caching. Claude Opus 5 caches at the token level with a 10% charge on cached input after the first full pass. Gemini 3.7 Flash has similar caching. If your long context is stable (same document used across multiple queries), caching pays for itself fast.

Example: you’re using an agent to analyze a 2M-token compliance document repeatedly. First request costs full price on all 2M. Requests 2–100 cost 10% of those 2M. After ~12 requests, caching breaks even against not caching. For agents that revisit the same corpus, this is material.

# Claude with prompt caching
response = client.messages.create(
    model="claude-opus-5",
    max_tokens=4096,
    system=[
        {
            "type": "text",
            "text": "You are a code reviewer. Analyze PRs against this architecture."
        },
        {
            "type": "text",
            "text": full_architecture_doc,  # 500K tokens
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": current_pr_diff}]
)

The cache_control directive tells Claude to cache that block. On subsequent requests with the same architecture doc, you pay only 10% of its token cost.

Multi-agent is cheaper than long-context for complex reasoning

If your task is genuinely complex (e.g., reasoning across 10M tokens of documents to synthesize a decision), splitting into multiple agents often costs less than a single long-context call. One agent summarizes documents A–C into 50K tokens. Another summarizes D–F. A third agent synthesizes the two summaries plus the original query. Total tokens might be 800K instead of 10M. Cost: lower. Quality: often higher because each agent has focus.

This is the inverse of the intuition people have. More tokens feels like it should be better, but the token economy says: smaller, focused contexts with explicit handoffs often win.

Watch for diminishing returns on retrieval quality

If you’re using long context for RAG (retrieving 100+ documents to include in the prompt), benchmark whether that actually improves answers versus retrieving 5–10 top-ranked documents. Studies from Towards AI (August 2026) on RAG reranking show that including more documents often increases the token count without proportionally improving output quality. Semantic search followed by a good reranker (like BGE Reranker) often outperforms brute-force context expansion.

Real-world cost math

Claude Opus 5: $5 / 1M input tokens (or $0.50 / 100K), $30 / 1M output. A single 10M-token input request costs $50. If your daily workflow runs 10 such requests, that’s $500/day, or ~$150K/year just on input tokens for one agent. Cached tokens cost $0.50 / 1M after the first pass—orders of magnitude cheaper, but only if your context is reusable.

By comparison, a Claude Sonnet 5 call on 100K tokens costs $0.30 input, $1.50 output. That’s 166x cheaper per token, though it can’t handle the 10M context. The right question isn’t “can I use 10M tokens?” It’s “what’s the cheapest way to answer this question reliably?” That often means smaller contexts, better retrieval, or multi-agent decomposition.

Practical guidance for production

Start with 500K–1M context. That’s enough for most real workflows: a codebase, a document set, a conversation history. Only expand to 10M if:

1. Your task genuinely requires reasoning across that much information as a single unit (rare).
2. You’ve validated that longer context improves output quality on your workload (measure it).
3. You have a cost budget that absorbs 10x higher per-request expenses (or aggressive caching that brings it down).

If you’re building agents, use long context for planning and initial analysis, then decompose into smaller agents for execution. If you’re doing RAG, optimize retrieval before throwing more tokens at the problem. The unsexy truth is that most “long-context” use cases work better as “smart retrieval + modest context + multi-agent coordination.”

Bottom line: Long-context models are a capability, not a default strategy. Use them when your access pattern matches their economics (reusable, cached context) or when your task genuinely requires unified reasoning across massive text. For everything else, structured retrieval and agent decomposition will cost you less and often work better.

Question via Hacker News