Office Hours — How do you optimize token usage when your AI agent needs to process long documents like Wikipedia pages?

How do you optimize token usage when your AI agent needs to process long documents like Wikipedia pages?

Long documents kill your token budget fast. A Wikipedia page on quantum mechanics can easily run 10K tokens. If your agent makes five passes through it (one to understand scope, one to extract facts, one to validate, etc.), you’ve burned 50K tokens on a single task. That math breaks production economics quickly.

The core problem isn’t that documents are long, it’s that you’re treating them as monolithic blobs instead of structured data.

Chunk, Index, Then Retrieve Instead of Reading Whole Documents

Stop feeding entire Wikipedia pages to your model. Use a retrieval step first. Split the document into semantic chunks (roughly 500-800 tokens each, with overlap), embed them, and only retrieve the chunks relevant to your agent’s actual question.

A Wikipedia page on “Machine Learning” has sections on history, algorithms, applications, and criticism. If your agent only needs to answer “What are the applications of machine learning in medicine?” you retrieve maybe 2-3 chunks, not the whole 15K-token page.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS

# Split document into chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n\n", "\n", ".", " "]
)
chunks = splitter.split_text(long_document)

# Embed and store
embeddings = OpenAIEmbeddings()
vector_store = FAISS.from_texts(chunks, embeddings)

# Later: retrieve only relevant chunks
query = "What applications does this technology have in medicine?"
relevant_chunks = vector_store.similarity_search(query, k=3)
# Pass only these 3 chunks to the agent, not the full document

This cuts token usage from 10K+ down to ~2-3K for a single query. The tradeoff: you need embedding infrastructure and a retrieval latency hit (usually 200-500ms). Worth it.

Use Document Summarization as a Preprocessing Step

If your agent needs broad context about a document before diving into details, don’t have it read everything. Summarize first, then retrieve details as needed.

Run a one-time, batch summarization job on new documents. Store the summary alongside the full text. When an agent asks a question, it first reads the summary (200-400 tokens), then decides whether to retrieve specific chunks from the full document.

Cost: one expensive summarization call per document (a 15K-token Wikipedia page might cost $0.30 to summarize once). Benefit: every subsequent agent query is cheaper because it reads a 300-token summary instead of the full 15K text.

Implement Smart Context Pruning in the Agent Loop

Agents often keep redundant context across multiple reasoning steps. If your agent makes ten API calls, tool invocations, or reasoning loops, each one might include the full document context by default.

Add a context manager that tracks what information the agent has already extracted and used. If the agent already pulled out “the founding date of this technology” in step 2, don’t pass that information back in step 5’s context.

class ContextManager:
    def __init__(self, full_document):
        self.full_document = full_document
        self.extracted_facts = {}
        self.context_budget = 4000  # tokens
    
    def get_context_for_step(self, step_type, previous_extractions):
        # Don't repeat facts already extracted
        new_context = self.full_document
        for fact, value in previous_extractions.items():
            new_context = new_context.replace(
                # Remove redundant passages we already processed
                self._find_passage(fact), 
                f"[Already extracted: {fact} = {value}]"
            )
        
        # Truncate to token budget
        return self._truncate_to_tokens(new_context, self.context_budget)

This prevents context bloat from accumulating as the agent reasons through a task.

Choose the Right Model Tier for the Task

Not every agent step needs Claude Opus 5 or GPT-5.6 Sol. Use cheaper, faster models (Claude Sonnet 5 or Gemini 3.5 Flash) for retrieval, filtering, and validation steps. Reserve expensive models for the reasoning that actually needs depth.

Cursor’s agent architecture does this: frontier models (Opus, GPT-5.6) plan the work; cheaper models (Sonnet, Flash) execute it. A Wikipedia-processing agent could:

Use Claude Sonnet 5 to chunk and summarize the raw document ($0.30 per task).
Use Gemini 3.5 Flash to retrieve relevant sections given a query ($0.05 per query).
Use Claude Opus 5 only for nuanced reasoning or synthesis steps where accuracy matters ($0.20 per step).

This hybrid approach can cut total costs 60-70% versus running everything through the flagship model.

Measure Hidden Token Costs in Your Agent Loop

Most teams don’t actually know where tokens are burning. You think your agent is cheap because a single call to Claude Sonnet costs $0.02, but you’re making five hidden calls per user interaction: one to retrieve chunks, one to validate the retrieved context, one to rerank results, one to generate the response, and one to check for hallucinations.

Log every LLM call with its token count.

import anthropic

client = anthropic.Anthropic()
total_tokens = 0

for step in agent_steps:
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1000,
        messages=[{"role": "user", "content": step["prompt"]}]
    )
    step_tokens = response.usage.input_tokens + response.usage.output_tokens
    total_tokens += step_tokens
    print(f"Step {step['name']}: {step_tokens} tokens")

print(f"Total: {total_tokens} tokens for this agent task")

You’ll often find that step 3 (validation) alone burns more tokens than your initial retrieval. Once you see it, you can fix it—maybe validation doesn’t need the full document context, or maybe you can batch validations into a single call.

Bottom line:

Never pass a long document whole to an agent. Retrieve only the relevant chunks, summarize once and reuse, use cheaper models for most steps, and measure token burn across the entire agent loop. This typically cuts token usage 70-80% while actually improving latency and reliability—because agents reason better on focused context anyway.

Question via Hacker News