Office Hours — What are the practical considerations for tokenization and memory management when building production LLM systems?
A daily developer question about AI/LLMs, answered with a direct, opinionated take.
What are the practical considerations for tokenization and memory management when building production LLM systems?
Tokenization looks simple until you deploy. You pick a model, trust its tokenizer is correct, and suddenly you’re bleeding tokens on production traffic because you didn’t account for how different models count the same text. Memory management compounds the problem. Cache every response and you’ll eventually run out of RAM. Don’t cache and you’ll recompute the same context a thousand times. The real question isn’t whether these matter, it’s where the actual pain points hide.
Tokenization Variance Is Real and Costs Money
Claude Sonnet 5’s new tokenizer emits roughly 30% more tokens than Sonnet 4.6 for the same text. That’s not a bug. That’s a fundamental difference in how Anthropic redesigned their token boundaries. If you’re using Sonnet 5 without re-benchmarking, your real per-task costs are approximately 40% higher than the list price suggests, because the introductory pricing doesn’t account for the higher token count. GPT-5.6 Sol and Claude Opus 5 use different tokenizers entirely. A migration isn’t free.
The cost impact compounds at scale. If you’re processing 10 million tokens daily at $0.05 per million input tokens, a 30% tokenization variance costs you $1,500 extra per month. That’s not trivial. But the bigger trap is that you won’t discover this until you’re already in production, comparing actual API bills to your projections.
Concrete example: You benchmark an extraction task using GPT-5.4 Nano (fast, cheap). The average query is 2,000 tokens, response is 500 tokens. Cost per call: $0.11. You scale to 100,000 calls monthly, budgeting $11,000. You then migrate to Claude Sonnet 5 without re-running the benchmarks. Same logical prompts now consume 2,600 input tokens and 650 output tokens due to the tokenizer change. Cost per call climbs to $0.16. Your actual monthly spend is now $16,000—45% over budget. This happens because tokenizer variance is invisible until you measure.
Context Window and Cache Trade-Offs
Longer context windows feel like a free win. GPT-5.6 Sol, Claude Opus 5, and Gemini 3.7 Flash all support 1M+ context. But longer context creates two hard choices: you either cache aggressively (eating RAM and requiring cache invalidation logic), or you recompute on every request (burning tokens and latency).
Prompt caching is powerful if your workload has stable context. Claude’s cache mechanism ($0.90 per 1M tokens cached vs. $3.00 per 1M input tokens normally) creates 67% savings on cached tokens, but only if you’re reusing the exact same context prefix across multiple requests. The moment your context drifts (user adds a new document, retrieval results change), the cache misses and you recompute the full context. Most real systems have high cache miss rates because context is rarely perfectly stable.
vLLM’s PagedAttention architecture (now standard in many production inference servers) helps here by breaking context into manageable pages that can be reused without recomputing. But you need to understand that your inference layer actually implements this. Many teams run vLLM and assume they’re getting PagedAttention benefits when they’re running older configurations. Check your vLLM version and verify continuous batching is enabled.
Memory Management: The Hidden Killer
Running multiple concurrent inference requests is where memory management stops being theoretical. Claude Code Auto Mode running in parallel on macOS and Linux can now message each other and share context—which is elegant until both instances consume overlapping memory trying to process the same document.
A practical rule: assume 10-15KB of memory per cached token on typical GPU inference infrastructure. If you’re keeping 1M tokens cached per user in a 100-user system, you’re allocating 1-1.5TB of VRAM just for caches. Most teams discover this by accident when their inference server runs out of memory and starts swapping to disk (which is catastrophic for latency).
The fix is conscious cache invalidation. Set a TTL on cached context. Implement an LRU eviction policy. Monitor actual cache hit rates and only cache when the hit rate exceeds 60-70%. Below that threshold, you’re wasting memory on low-utilization cache entries.
Token Budgeting and Cost Blowouts
Building AI agents without explicit token budgeting is how you end up with runaway costs. Claude Code and similar autonomous agents can consume tokens geometrically if they’re allowed to retry failed operations with full context re-inclusion each time.
A concrete safeguard: set a per-task token budget. If you’re extracting data from a document, cap input tokens at 50K and output at 5K. When you hit the cap, fail fast instead of allowing the agent to prompt retry loops that reconsume the entire context. This means implementing token counting at every step, not just at the API call level.
Token counting itself is often wrong. Many teams use approximate tokenizer implementations (running the tokenizer locally) that diverge from the actual API tokenizer by 5-15%. The only reliable approach is to use the actual provider’s tokenizer for pre-flight accounting. OpenAI provides tiktoken. Anthropic provides their tokenizer. Alibaba’s Qwen models include their tokenizer. Use the real thing, not approximations.
Practical Memory Patterns in Multi-Agent Systems
Multi-agent architectures can silently triple token costs if token economics aren’t explicitly modeled. One agent retrieves context (consumes tokens), another agent processes it (consumes tokens again), a third synthesizes results (consumes a third time). If you’re not deduplicating shared context, you’re paying for the same information three times.
Use a shared context cache at the agent orchestration layer, not at individual agent layers. This means one retrieval step feeds multiple agents without re-tokenizing. Track which agents have seen which context segments and avoid redundant processing.
Bottom line:
Benchmark tokenization for every model migration before production deployment, because real per-token costs diverge from listed pricing by 30-50% depending on tokenizer changes. Set explicit token budgets on tasks, implement proper cache invalidation with TTLs, and monitor actual cache hit rates—if you’re caching without measuring hits, you’re trading RAM for marginal latency gains that probably aren’t worth it.
Question via Hacker News