Office Hours — What's the most practical way to handle context window limits when building LLM applications with long documents or conversations? A daily developer question about AI/LLMs, answered with a direct, opinionated take. 2026-11-21T12:00:00.000Z Office Hours Office Hours office-hoursq-and-apractical-ai

Office Hours — What's the most practical way to handle context window limits when building LLM applications with long documents or conversations?

A daily developer question about AI/LLMs, answered with a direct, opinionated take.

Daily One question from the trenches, one opinionated answer.

What’s the most practical way to handle context window limits when building LLM applications with long documents or conversations?

Context windows keep growing, but they’re still finite and expensive. The real constraint isn’t token capacity anymore, it’s cost and latency. At 1M tokens, you can fit a lot, but paying for a million-token prompt to answer one question is wasteful. Here’s how teams are actually handling this.

The Three-Tier Strategy

Most production systems use a mix of three approaches depending on task type. First, aggressive pruning: strip out irrelevant context before it reaches the model. Second, semantic filtering: embed and retrieve only what matters. Third, hierarchical summarization: when you must include long context, structure it so the model can skip to relevant sections.

The mistake most teams make is trying to shove everything into the prompt and hoping the model figures it out. That worked at 128K tokens but breaks fast once you hit real-world document counts. At healthcare companies using LLMs to file insurance claims at scale, the winning pattern was to pre-process PDFs into structured sections, then retrieve only the forms the agent actually needs to complete, rather than loading entire patient records.

Chunking and Retrieval

If you’re building with documents, you need retrieval. Period. Don’t treat the context window as a cache for “everything the model might need.” Use it strategically.

For conversations, the story is different. If you’re maintaining a chat thread, most teams keep the last N messages in context (maybe 10-20), plus a summary of the earlier thread. When the summary gets stale, regenerate it. Claude Opus 5.5 can summarize a 50K-token conversation into 2-3K tokens without losing critical details.

For code, the pattern is simpler: keep the files being modified in context, plus imports and dependencies. Use git diffs instead of full file content when possible. GitHub Copilot’s multi-model approach uses Sol for understanding large codebases and cheaper models for inline completions, which is a good template if you’re building your own stack.

Practical Cost Math

Here’s where this actually matters. Using GPT-6 Astra ($10/$50 per M input/output), a million-token prompt costs $10 just for the input. If you’re making 100 of these calls a day, you’re spending $1,000 daily on input tokens alone. That’s $30K a month for context you might not need.

Compare that to a retrieval-based approach: embed your documents once (a one-time cost), then do semantic search over embeddings (cheap, microseconds), and retrieve the top 5-10 chunks. Total prompt size drops to 20-50K tokens, cutting input cost to $0.20-$0.50 per call. At scale, that’s a 20x cost reduction.

The tradeoff: retrieval can miss context if your embedding model doesn’t capture semantic relationships well, or if the document is poorly structured. Run evals on your own data. Don’t assume generic RAG benchmarks apply to your domain.

When Long Context Actually Wins

There are real cases where shoving everything in works better than retrieval. Code review on a large codebase: loading the entire modified diff (even at 500K tokens) often beats doing retrieval because the model needs global context to understand architecture implications. Scientific papers with complex interdependencies. Legal contracts where clause relationships matter.

For these, use long context directly, but be intentional. Structure the input so the model can quickly identify relevant sections. Use markdown headings, numbered sections, explicit markers. A well-formatted long prompt at 500K tokens often performs better than a poorly formatted 100K prompt because the model can navigate it.

Caching: The Actual Game-Changer

If you’re making multiple requests with overlapping context, use prompt caching. Claude’s cached input reads at $0.10 (vs. $2/$10 for uncached input/output) make this economical. Set up caching for:

  • System prompts that don’t change
  • Large reference documents (style guides, API docs, code libraries)
  • Base conversation history that you’re extending

Example: an AI coding agent working on a single codebase all day. Load the entire repo structure once into cache, then each subsequent request to the agent only pays for the cache read ($0.10 per M) plus new input tokens for the current task. Over 50 agent interactions, cached context saves 95% of the input cost.

Multi-Model Routing

If you need flexibility, route tasks by context size. Use Claude Opus 5.5 for requests under 100K tokens (maximum quality). For anything larger, use Claude Fable 5.2 with its full 1M context, since you’re paying the long-context premium anyway. For retrieval-based work (where the prompt is semantically simpler), use Claude Haiku 5.5 at the smaller pricing tier.

This pattern—pick the cheapest model that works for the task—is how Linear scaled to $100M ARR with 50%+ agent-written support tickets. They weren’t using the most capable model for everything. They were matching model tier to task complexity.

Real Example: Document Q&A

Say you’re building a Q&A system over a 10,000-page manual. Here’s what actually works:

  1. Split the manual into 1-2K token sections.
  2. Embed each section once (one-time cost, ~$1 for 10,000 sections).
  3. When a user asks a question, embed the question (microseconds, negligible cost).
  4. Retrieve the top 3-5 sections by similarity.
  5. Send those sections + the question to Claude Opus 5.5 (50K tokens, $0.50 input cost).

Total cost per query: ~$0.50-$1. At a million-token prompt: $10. That’s a 10x savings and often better quality because the model isn’t drowning in irrelevant text.

Bottom line:

Don’t use long context as a substitute for retrieval. Use it as a last resort when the task genuinely requires global context, or when you’re caching expensive reference material across many requests. For everything else, retrieve aggressively and keep prompts lean. The cheapest token is the one you never send.

Question via Hacker News