Office Hours — How do you effectively manage and evict old tokens from an LLM's context window to stay within token limits while preserving important conversation history?

How do you effectively manage and evict old tokens from an LLM’s context window to stay within token limits while preserving important conversation history?

This is the wrong framing of the problem. Most teams don’t need sophisticated token eviction strategies. They need better architectural decisions upstream.

The Real Problem Isn’t Eviction, It’s Design

The token limit question usually surfaces because you’ve built a system that accumulates conversation history linearly—every message gets appended to the context, and at some point you hit the ceiling. This design is brittle because:

Token limits aren’t hard walls anymore. GPT-6 Astra has 1.05M tokens. Claude Opus 5.5 has 1M. Gemini 4 Argon has 1M with up to 1M output. Even Claude Haiku 5.5 at the cheapest tier gives you 100K before pricing jumps. For most applications, the limit is cost, not availability.

When you do need to prune, you’re usually solving the wrong problem at the application level instead of thinking about what information actually matters for your task.

Practical Token Management Patterns

Summarization is your first lever. Before the context gets unwieldy, summarize old turns. This isn’t “lose information”—it’s “compress information into denser tokens.” After every 5-10 turns, generate a brief summary of decisions made and context established, then replace the raw conversation with that summary. You go from 4,000 tokens of back-and-forth down to 300 tokens of structured facts.

Here’s a concrete pattern:

# After N turns, generate and replace
if turn_count % 10 == 0:
    summary_prompt = f"""
    Summarize this conversation in 150 tokens, capturing:
    - Key decisions made
    - Current task state
    - Assumptions about the user's goals
    
    Original conversation:
    {conversation_history}
    """
    summary = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=200,
        messages=[{"role": "user", "content": summary_prompt}]
    )
    
    # Replace old history with summary + last 3 turns
    conversation_history = [
        {"role": "system", "content": f"Context summary:\n{summary.content[0].text}"},
        *conversation_history[-6:]  # Keep last 3 exchanges
    ]

This costs you one extra API call per 10 turns, but cuts context bloat by 60-80%. For Claude at $2/$10 per M input/output, you’re spending maybe $0.01 per summary to save 3,700 tokens on the next call. The math is almost always positive.

Separate episodic from semantic memory. Not all context belongs in the prompt. Conversations tend to mix three things: what just happened (episodic), what we know to be true (semantic), and how to proceed (procedural). Your prompt should carry the last turn or two of episodic context, facts that are actively relevant to the current task, and standing instructions. Everything else lives in a database.

# Episodic: last 2 turns, always in context
episodic = conversation_history[-4:]  # 2 exchanges

# Semantic: relevant facts only
relevant_facts = vector_search(query=current_task, k=3)

# Procedural: standing system instructions
system_msg = "You are a data analyst. Always validate assumptions."

# Build minimal prompt
prompt_messages = [
    {"role": "system", "content": system_msg},
    {"role": "system", "content": f"Relevant context:\n{format_facts(relevant_facts)}"},
    *episodic
]

This keeps your prompt lean. The vector search is doing the real work—retrieving only what matters—instead of blindly accumulating history.

Use Claude’s cache for static context. If you have large, static reference material (documentation, code context, policy docs), put that in the cache and reference it across calls. Claude’s cache reads cost $0.10/M vs. $2/$10 for regular input, a 95% discount. This is brutal leverage if your context includes things that don’t change per request.

# Static reference material cached once, reused
system_with_cache = [
    {
        "type": "text",
        "text": "You are an expert code reviewer."
    },
    {
        "type": "text",
        "text": "Our codebase style guide:\n" + load_large_style_guide(),
        "cache_control": {"type": "ephemeral"}  # Cache this
    }
]

# Per-request, only new conversation and the task
messages = [
    *system_with_cache,
    {"role": "user", "content": current_request}
]

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=2000,
    system=system_with_cache,
    messages=messages
)

The style guide gets cached for 5 minutes across requests. You pay the full token cost once, then 95% off for follow-ups.

When Token Limits Actually Matter

Multi-turn agent workflows with long-running tasks. An agent executing 50+ steps accumulates context naturally—tool calls, results, intermediate reasoning. Here, sliding-window retention beats aggressive eviction. Keep the last 10-15 steps and one system summary. Discard the middle, which is less likely to be relevant than the immediate past or the overall goal.

For truly long-running agents (the kind Claude Code or Cursor handle), use structured memory checkpoints. Every 20 steps, write a checkpoint: current state, completed goals, next steps. Then reset the context window and load only the latest checkpoint plus the last 3-5 steps. You trade one extra write for a clean restart.

if step_count % 20 == 0:
    checkpoint = {
        "step": step_count,
        "state": current_state,
        "completed": goals_completed,
        "next": upcoming_tasks,
        "context_summary": summarize_recent_actions(last_steps=5)
    }
    save_checkpoint(checkpoint)
    # Reset prompt to checkpoint + minimal recent history

Bottom line:

Don’t build a token eviction system. Build systems where you’re explicit about what context matters, summarize instead of accumulating, and separate episodic memory (this conversation) from semantic memory (what we know) and procedural memory (how we work). Caching static material and checkpointing long workflows handles 90% of the real problem without custom eviction logic.

Question via Hacker News