Where randomness meets reason

Paper of the Week — Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents Silent and Costly: LLM agents silently waste money through mis-integrated prompt caching — this paper shows how to detect and fix the bug class. 2026-10-29T12:00:00.000Z Paper of the Week Paper of the Week researchpapersarxivpractical-ai

Paper of the Week — Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents

Silent and Costly: LLM agents silently waste money through mis-integrated prompt caching — this paper shows how to detect and fix the bug class.

researchpapersarxivpractical-ai
Weekly One research paper, broken down for people who build things.

Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents

Ying Zhang, Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Yepang Liu, Leo Zhang. Published 2026-10-26. arXiv:2610.34817

One sentence summary

Prompt caching is a deceptively fragile optimization — this paper systematically characterizes the bug patterns that silently eliminate its cost and latency benefits, and provides detection tooling to catch them.

Why this paper

Every team running agentic workloads at scale has enabled prompt caching. But almost nobody has audited whether their integration is actually working — and with providers like Anthropic, OpenAI, and Google all shipping subtly different caching semantics, the surface area for mis-integration is large and growing.

What they did

The authors analyzed how LLM agent frameworks integrate prompt caching across heterogeneous providers (each with different rules around prefix structure, cache-key construction, and TTLs) and catalogued the distinct ways these integrations silently fail. They then built a static/dynamic detection approach to identify mis-integration patterns in agent codebases without requiring access to billing data.

Key findings

  • Prompt caching mis-integrations form a distinct bug class with recurring patterns — not one-off mistakes, but systematic errors reproducible across frameworks
  • Cache invalidation from minor prompt perturbations (role-label reordering, whitespace, metadata injection) is the most common silent failure mode
  • Provider-specific caching rules (e.g., Anthropic’s prefix-pinning requirements, OpenAI’s turn-boundary semantics) are frequently violated even in popular open-source agent frameworks
  • Mis-integrated caching doesn’t raise errors — it simply bills at full token price while appearing functional, making it invisible without dedicated instrumentation
  • The detection approach flags mis-integration sites with high precision on a set of real-world agent repositories

Why it matters for practitioners

If you’re using Claude Haiku 5.5, Claude Sonnet 5.5, or GPT-6 Luna for high-volume agentic tasks, prompt caching is one of your primary cost levers — cache reads on Haiku 5.5 drop to a fraction of standard input pricing. A single mis-integration pattern (say, a dynamic system-prompt timestamp invalidating your cache prefix on every turn) can silently wipe out those savings at scale, and your cost dashboard won’t tell you why.

What you can use today

  • Audit your agent’s system prompt construction: anything injected before the stable prefix (timestamps, request IDs, user metadata) will bust the cache on every call — move dynamic content after the static prefix or into the user turn
  • Check provider-specific requirements before assuming caching is active: Anthropic requires cache_control markers, OpenAI caches by prefix automatically but only above a token threshold, and Google’s Gemini 4 Argon uses a separate context caching API — each has different invalidation semantics
  • If the paper’s detection tooling is released, run it against your agent’s prompt-assembly code before your next infrastructure review; even catching one high-frequency cache miss pattern can meaningfully reduce per-task cost on long-running workflows