Technical breakdown of semantic caching for LLM apps: exact vs embedding-based matching, similarity thresholds, cache invalidation, and the real latency/cost math.

Semantic Caching for LLM Applications

Every LLM API call costs tokens and time. Semantic caching intercepts calls that are “close enough” to previous ones and returns a stored response instead of hitting the model again. The idea is simple — the implementation is full of sharp edges around similarity thresholds, cache invalidation, embedding costs, and failure modes that can silently degrade output quality.

Traditional caching (exact string match on the prompt) catches duplicates but misses the vast majority of reusable responses. “What’s the capital of France?” and “what is the capital of france?” are different strings. Semantic caching uses embeddings to match on meaning instead of characters, which dramatically increases hit rates — but introduces an entirely new class of problems.

Exact Match Caching: The Baseline
Semantic Caching Architecture
Embedding Models for Cache Matching
Similarity Thresholds: The Central Tradeoff
Cache Key Design
Storage Backends
Cache Invalidation Strategies
The Latency and Cost Math
Failure Modes
Implementation Patterns
When Semantic Caching Doesn’t Work
Production Checklist
Summary
Further Reading

Exact Match Caching: The Baseline

Exact match caching hashes the full request (prompt text, model name, temperature, and other parameters) and checks a key-value store. It works, it’s deterministic, and it never returns a wrong cached result for a different question.

import hashlib
import json
import redis

r = redis.Redis()

def cache_key(request: dict) -> str:
    # Normalize: sort keys, strip whitespace
    normalized = json.dumps(request, sort_keys=True).strip()
    return f"llm:exact:{hashlib.sha256(normalized.encode()).hexdigest()}"

def cached_completion(request: dict) -> str | None:
    key = cache_key(request)
    cached = r.get(key)
    if cached:
        return cached.decode()
    
    response = call_llm(request)
    r.setex(key, 3600, response)  # 1-hour TTL
    return response

Hit rates for exact match caching depend entirely on the application. Chatbots with free-form user input see 2-8% hit rates. Structured workflows where the same prompts recur (classification, extraction, templated generation) can reach 30-60%.

The ceiling is low for conversational applications because humans rarely type the same thing twice. Normalization helps — lowercasing, stripping punctuation, collapsing whitespace — but catches only trivial variations.

Hybrid caching checks exact match first (sub-millisecond), then falls back to semantic similarity search.

Semantic Caching Architecture

A semantic cache replaces string hashing with embedding-based similarity search. The flow:

Incoming prompt is embedded using an embedding model
The embedding vector is searched against a vector index of previously cached prompts
If the nearest neighbor exceeds a similarity threshold, the cached response is returned
Otherwise, the LLM is called, and both the prompt embedding and response are stored

The embedding step adds 5-20ms of latency per request. The vector search adds another 1-10ms depending on index size and backend.

The critical difference from exact caching: semantic caching is probabilistic. Two prompts with cosine similarity of 0.96 might mean the same thing. Or they might not. The threshold determines the false-positive rate, and there is no universally correct value.

Embedding Models for Cache Matching

The embedding model choice affects hit rate, latency, accuracy, and cost. Cache matching has different requirements than retrieval for RAG — cache embeddings need to distinguish between prompts that look similar but require different answers.

Model	Dimensions	Latency (p50)	Cost per 1M tokens	Notes
OpenAI text-embedding-3-small	1536	~15ms	$0.02	Good balance for caching
OpenAI text-embedding-3-large	3072	~25ms	$0.13	Higher precision, higher cost
Cohere embed-v4	1024	~12ms	$0.10	Strong multilingual support
Voyage voyage-3	1024	~18ms	$0.06	Good at code similarity
Jina jina-embeddings-v3	1024	~10ms	$0.02	Fast, cheap, self-hostable
BGE bge-m3 (self-hosted)	1024	~5ms (local)	Compute only	No API cost, full control

For semantic caching specifically, smaller models (text-embedding-3-small, jina-embeddings-v3) are often better choices than large ones. The cache lookup happens on every request, so embedding latency is on the critical path. A 25ms embedding call on every request adds up; a 5ms local BGE call is nearly free.

Self-hosted embedding models (BGE-M3, Jina v3) eliminate per-token embedding costs entirely, which matters at scale. At 10M requests/day with average 200-token prompts, text-embedding-3-small costs ~$40/day just for cache lookups. BGE-M3 on a single L4 GPU handles the same load for a fixed ~$15/day in compute.

Dimensionality Reduction

OpenAI’s text-embedding-3 models support Matryoshka representations — truncating the vector to fewer dimensions while preserving most of the similarity structure. For cache matching, 512 or 256 dimensions from text-embedding-3-small often provide sufficient discrimination while cutting vector storage and search time roughly in half.

from openai import OpenAI

client = OpenAI()

def get_cache_embedding(text: str, dimensions: int = 512) -> list[float]:
    response = client.embeddings.create(
        input=text,
        model="text-embedding-3-small",
        dimensions=dimensions  # Matryoshka truncation
    )
    return response.data[0].embedding

Testing on internal query logs suggests 512 dimensions retain ~97% of the pairwise similarity ranking compared to full 1536 dimensions for English text. For multilingual or code-heavy workloads, the degradation is steeper — probably best to stay at 768+ dimensions.

Similarity Thresholds: The Central Tradeoff

The similarity threshold is the most consequential parameter in a semantic cache. Too high (0.99) and the cache rarely hits. Too low (0.85) and the cache returns wrong answers for genuinely different questions.

Cosine similarity between embeddings captures semantic closeness, but the relationship between cosine similarity and “same question” is non-linear and model-dependent. For text-embedding-3-small:

Cosine Similarity	Typical Relationship
0.99-1.00	Near-identical phrasing, trivial rewording
0.96-0.99	Same question, different wording
0.93-0.96	Similar topic, probably same answer
0.90-0.93	Related questions, answer may differ
0.85-0.90	Same domain, different questions
< 0.85	Unrelated

These ranges are approximate and vary by embedding model, domain, and prompt structure. The only reliable way to set a threshold is empirical: log prompt pairs with their similarity scores, have humans label whether the cached response would be acceptable, and plot precision/recall curves.

Start with 0.97 and tune down based on measured false-positive rates in your specific workload.

Adaptive Thresholds

A single global threshold is a blunt instrument. Some query types tolerate looser matching (factual lookups, classification) while others require tight matching (personalized generation, multi-turn conversation).

One pattern: maintain per-category thresholds. Route queries through a lightweight classifier first, then apply the appropriate threshold.

THRESHOLDS = {
    "factual_lookup": 0.94,
    "classification": 0.93,
    "summarization": 0.96,
    "creative_writing": 0.99,  # Almost never cache
    "code_generation": 0.97,
}

def should_use_cache(query_type: str, similarity: float) -> bool:
    threshold = THRESHOLDS.get(query_type, 0.96)
    return similarity >= threshold

Cache Key Design

The cache key for semantic matching isn’t just the user prompt. Two identical prompts with different system prompts, temperatures, or model versions should not share a cache entry.

The composite cache key needs to account for:

Prompt text → embedded for semantic matching
System prompt → hashed, used as a partition key
Model identifier → different models produce different outputs
Temperature → temperature=0 and temperature=1 shouldn’t share cache entries; temperature=0 entries are safe to cache, higher temperatures arguably shouldn’t be cached at all
Other sampling parameters → top_p, max_tokens, response format

import hashlib

def composite_cache_key(request: dict) -> dict:
    """Returns both the embedding text and the partition key."""
    prompt_text = request["messages"][-1]["content"]
    
    # Everything except the prompt content becomes a partition key
    partition_parts = [
        request.get("model", ""),
        str(request.get("temperature", 1.0)),
        str(request.get("top_p", 1.0)),
        request.get("system", ""),
    ]
    partition_hash = hashlib.sha256(
        "|".join(partition_parts).encode()
    ).hexdigest()[:16]
    
    return {
        "embed_text": prompt_text,
        "partition": partition_hash,
    }

The vector search is then scoped to the partition — only comparing against prompts that share the same model, system prompt, and sampling parameters. This prevents cross-contamination between different configurations.

Partitioning the vector index by non-semantic parameters prevents returning cached responses generated with different configurations.

Multi-Turn Conversations

Caching multi-turn conversations is harder. The same user message in different conversation contexts should produce different responses. Options:

Cache only on the full conversation — embed the concatenation of all messages. Hit rates drop because conversation histories are rarely similar enough.
Cache the last N turns — embed the last 2-3 turns. Better hit rates, risk of missing important context from earlier in the conversation.
Don’t cache multi-turn — only cache single-turn requests. The pragmatic choice for most applications.

Option 3 is usually correct. The hit rate on multi-turn semantic caching is low enough that the complexity isn’t justified.

Storage Backends

The vector index and response store can be co-located or separate. Common patterns:

Backend	Vector Search	Response Storage	Latency (p50)	Notes
Redis + RedisVL	HNSW in Redis	Redis strings	1-3ms	Single-node simplicity
pgvector	IVFFlat or HNSW	Same Postgres table	3-10ms	Good if already on Postgres
Qdrant	HNSW	Built-in payload	1-5ms	Purpose-built, good filtering
Pinecone	Proprietary	Metadata fields	5-15ms	Managed, higher latency
In-memory (FAISS/hnswlib)	HNSW	Application memory	<1ms	Fast but not persistent
SQLite + sqlite-vss	IVFFlat	Same SQLite DB	2-5ms	Single-file, zero infra

For semantic caching specifically, latency matters more than in RAG because the cache lookup is on the critical path of every request. Redis with the RedisVL extension or an in-memory HNSW index with periodic persistence are the lowest-latency options.

# Example: semantic cache with Qdrant
from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, PointStruct, Filter,
    FieldCondition, MatchValue
)
import uuid

client = QdrantClient(url="http://localhost:6333")

# Create collection with cosine similarity
client.create_collection(
    collection_name="llm_cache",
    vectors_config=VectorParams(size=512, distance=Distance.COSINE),
)

def cache_lookup(embedding: list[float], partition: str, threshold: float = 0.96):
    results = client.query_points(
        collection_name="llm_cache",
        query=embedding,
        query_filter=Filter(must=[
            FieldCondition(key="partition", match=MatchValue(value=partition))
        ]),
        limit=1,
        score_threshold=threshold,
    )
    if results.points:
        return results.points[0].payload["response"]
    return None

def cache_store(embedding: list[float], partition: str, prompt: str, response: str):
    client.upsert(
        collection_name="llm_cache",
        points=[PointStruct(
            id=str(uuid.uuid4()),
            vector=embedding,
            payload={
                "partition": partition,
                "prompt": prompt,
                "response": response,
                "created_at": int(time.time()),
            }
        )]
    )

Index Size Considerations

Semantic caches grow continuously. At 10K requests/day with a 30% miss rate (3K new entries/day), the index reaches 1M vectors in about a year. HNSW indexes with 512-dimensional vectors at 1M points consume roughly 2-4 GB of memory and search in under 5ms. This is manageable on a single node.

At 10M+ vectors, consider sharding by partition key or applying TTL-based eviction to keep the working set small.

Cache Invalidation Strategies

LLM cache invalidation is different from traditional caching because there’s no authoritative “source of truth” that changes. The underlying model’s knowledge doesn’t change between API calls. Invalidation is driven by:

Time-based TTL — Responses become stale as the world changes. A cached answer about “current stock price” is wrong within seconds. A cached answer about “what is photosynthesis” is valid indefinitely.
Model version changes — When the provider updates the model (even minor versions), cached responses may no longer match what the model would produce. Partition keys that include the model version handle this automatically.
System prompt changes — Application updates that modify the system prompt invalidate all cached responses for that configuration. Again, partition keys handle this.
Explicit invalidation — Application-level events (data updates, policy changes) that make cached responses incorrect.

Most production systems combine TTL with version-aware partition keys, plus application-level invalidation for specific events.

TTL Tiering

Different query types warrant different TTLs:

TTL_CONFIG = {
    "real_time_data": 300,        # 5 minutes
    "news_summary": 3600,         # 1 hour
    "factual_qa": 86400,          # 24 hours
    "code_explanation": 604800,   # 7 days
    "math_computation": 2592000,  # 30 days
    "translation": 2592000,       # 30 days
}

Classify queries into TTL tiers using a lightweight classifier or keyword matching. Over-caching (too-long TTLs) causes stale responses; under-caching (too-short TTLs) wastes the cache investment.

LRU and Usage-Based Eviction

Beyond TTL, evict based on access patterns. Track hit counts and last-access timestamps. Entries that haven’t been hit in 7 days are candidates for eviction regardless of TTL. Entries with high hit counts should have their TTLs extended — if a query is asked frequently, the cached response is probably still valid.

The Latency and Cost Math

The economic case for semantic caching depends on four variables: LLM cost per call, embedding cost per call, cache hit rate, and cache infrastructure cost.

Per-Request Cost Breakdown

Assume a typical request: 500 input tokens, 300 output tokens.

Without caching (GPT-5.6 Sol):

Input: 500 × $5.00/1M = $0.0025
Output: 300 × $30.00/1M = $0.009
Total: $0.0115 per request

With semantic caching (text-embedding-3-small, 512-dim):

Embedding call: 500 × $0.02/1M = $0.00001
Vector search: ~$0.000001 (amortized infrastructure)
Cache hit (no LLM call): $0.00001
Cache miss (LLM call + store): $0.0115 + $0.00001 = $0.01151

Net savings at different hit rates:

Hit Rate	Cost per Request	Savings vs No Cache
10%	$0.01036	10%
25%	$0.00864	25%
40%	$0.00691	40%
60%	$0.00461	60%
80%	$0.00231	80%

The embedding cost ($0.00001) is negligible compared to LLM costs. The cache pays for itself at any hit rate above ~2%, which covers the infrastructure overhead.

With a cheaper model (GPT-5.6 Luna at $0.20/$1.20 per M):

Without cache: $0.00046 per request
The absolute savings per cached hit are much smaller ($0.00046 vs $0.0115)
Cache infrastructure costs (embedding API, vector DB) represent a larger fraction

Semantic caching is most valuable for expensive models. For budget models like GPT-4.1 Nano ($0.10/$0.40 per M), the savings from caching are often smaller than the operational complexity cost.

Latency Impact

Caching adds latency on misses and removes it on hits:

Scenario	Added Latency	Saved Latency
Cache hit (API embedding)	+15-25ms (embed) + 1-5ms (search)	-500-3000ms (LLM call)
Cache hit (local embedding)	+3-8ms (embed) + 1-5ms (search)	-500-3000ms (LLM call)
Cache miss (API embedding)	+15-25ms (embed) + 1-5ms (search)	None
Cache miss (local embedding)	+3-8ms (embed) + 1-5ms (search)	None

On hits, the latency improvement is massive — 10-100x faster. On misses, the overhead is small enough to be acceptable (15-30ms on a 1-3 second LLM call). Local embeddings keep the miss overhead under 10ms.

Time to first token (TTFT) for streaming responses is where caching shines most. A cached response can start streaming from storage immediately, with TTFT under 10ms instead of the typical 200-800ms from an LLM API.

Failure Modes

Semantic caching fails in specific, often subtle ways.

False Positives (Wrong Cache Hits)

The most dangerous failure mode. Two queries have high cosine similarity but require different answers:

“What is the population of Paris?” vs “What is the population of Paris, Texas?”
“How do I delete a file in Python?” vs “How do I delete a file in Rust?”
“Summarize this document” (with different documents in the system prompt)

False positives serve the user a confidently wrong answer with no indication it came from cache. Mitigation:

Higher thresholds (0.97+) reduce false positives but cut hit rates
Post-match verification — after finding a cache candidate, do a quick LLM call to verify the cached response still applies (defeats the cost purpose but preserves latency gains)
Entity extraction — extract key entities from both the query and cached query; reject if entities don’t match
User feedback loops — let users flag bad responses; evict cache entries that get flagged

def verify_cache_match(query: str, cached_query: str, cached_response: str) -> bool:
    """Lightweight verification that the cache match is valid."""
    # Extract entities from both queries
    query_entities = extract_entities(query)  # e.g., spaCy NER
    cached_entities = extract_entities(cached_query)
    
    # If key entities differ, reject the match
    if query_entities != cached_entities:
        return False
    
    return True

Stale Responses

Cached responses go stale when the underlying truth changes. A response about “the latest version of Python” cached three months ago is wrong today. TTLs are the primary defense, but setting the right TTL requires knowing which queries have time-sensitive answers — which is itself a classification problem.

Embedding Drift

If the embedding model is updated or swapped, existing cache vectors become incompatible. Cosine similarity between vectors from different models is meaningless. Migration requires re-embedding all cached prompts, which can be expensive for large caches.

Pin embedding model versions explicitly. When upgrading, run both models in parallel during a migration window, or simply flush the cache and let it rebuild.

Cold Start

A new cache has zero entries and zero hits. The cache only pays off after a warm-up period. For applications with high query diversity and low repetition, the cache may never warm up enough to justify its cost.

Measure time-to-breakeven: how many requests before cumulative savings exceed cumulative cache infrastructure costs. If this number exceeds the expected request volume during the cache’s lifetime, don’t bother.

Implementation Patterns

Middleware Pattern

The cleanest implementation wraps the LLM client in a caching middleware that’s transparent to application code.

import time
from dataclasses import dataclass
from typing import Optional

@dataclass
class CacheResult:
    response: str
    cache_hit: bool
    similarity: Optional[float]
    latency_ms: float

class SemanticCache:
    def __init__(self, embed_fn, vector_store, threshold=0.96):
        self.embed_fn = embed_fn
        self.store = vector_store
        self.threshold = threshold
    
    def get_or_call(self, request: dict, llm_fn) -> CacheResult:
        start = time.monotonic()
        
        # Build composite key
        prompt = request["messages"][-1]["content"]
        partition = compute_partition(request)
        
        # Skip cache for high-temperature requests
        if request.get("temperature", 1.0) > 0.3:
            response = llm_fn(request)
            elapsed = (time.monotonic() - start) * 1000
            return CacheResult(response, False, None, elapsed)
        
        # Embed and search
        embedding = self.embed_fn(prompt)
        cached = self.store.search(embedding, partition, self.threshold)
        
        if cached:
            elapsed = (time.monotonic() - start) * 1000
            return CacheResult(
                cached["response"], True, cached["similarity"], elapsed
            )
        
        # Cache miss — call LLM
        response = llm_fn(request)
        self.store.insert(embedding, partition, prompt, response)
        elapsed = (time.monotonic() - start) * 1000
        return CacheResult(response, False, None, elapsed)

Two-Tier Cache

Combine exact match (fast, precise) with semantic fallback (slower, broader):

The exact match layer catches identical queries at sub-millisecond cost. Semantic matching handles paraphrased queries. The LLM is the last resort.

class TwoTierCache:
    def __init__(self, redis_client, semantic_cache):
        self.redis = redis_client
        self.semantic = semantic_cache
    
    def lookup(self, request: dict) -> Optional[str]:
        # Tier 1: exact match
        exact_key = compute_exact_hash(request)
        cached = self.redis.get(exact_key)
        if cached:
            return cached.decode()
        
        # Tier 2: semantic match
        prompt = request["messages"][-1]["content"]
        partition = compute_partition(request)
        embedding = self.semantic.embed(prompt)
        result = self.semantic.search(embedding, partition)
        
        if result:
            # Promote to exact cache for faster future lookups
            self.redis.setex(exact_key, 3600, result["response"])
            return result["response"]
        
        return None

The promotion step is important — when a semantic match is found, store it under the exact hash of the current query. Subsequent identical queries skip the embedding step entirely.

Async Pre-Warming

For applications with predictable query patterns (scheduled reports, daily batch jobs), pre-warm the cache by running expected queries during off-peak hours.

async def prewarm_cache(expected_queries: list[str], cache: SemanticCache):
    """Run expected queries through the LLM and cache results."""
    for query in expected_queries:
        request = build_request(query)
        await cache.get_or_call(request, llm_fn=call_llm)

When Semantic Caching Doesn’t Work

Semantic caching is not universally applicable. It fails or provides negative value in several common scenarios:

High-creativity tasks. Code generation, creative writing, brainstorming — tasks where variety in responses is desirable. Returning the same cached creative story for a similar prompt is a bug, not a feature.

Personalized responses. If the response depends on user-specific context (profile data, conversation history, uploaded documents), two semantically similar prompts from different users should produce different responses. The partition key can include user ID, but this reduces hit rates to near-zero for most applications.

Rapidly changing data. Queries about current events, stock prices, live sports scores. TTLs can mitigate this, but short TTLs (seconds to minutes) make the cache nearly useless.

Low-volume applications. With fewer than ~1K requests/day, the cache never accumulates enough entries to achieve meaningful hit rates. The infrastructure overhead exceeds the savings.

Multi-modal inputs. Queries that include images, audio, or documents alongside text. The semantic matching would need to embed all modalities, which is computationally expensive and the similarity metrics are less reliable for non-text inputs.

Semantic caching works best for repetitive, deterministic, high-volume workloads. Measure before committing.

Production Checklist

Building a semantic cache that works in production requires more than the core search logic:

Observability. Track these metrics continuously:

Cache hit rate (overall and per-partition)
False positive rate (requires sampling and human review)
Embedding latency (p50, p95, p99)
Vector search latency
Cache size (number of entries, memory usage)
Cost savings (estimated from hit rate × model pricing)

import prometheus_client as prom

cache_hits = prom.Counter("semantic_cache_hits_total", "Cache hits", ["partition"])
cache_misses = prom.Counter("semantic_cache_misses_total", "Cache misses", ["partition"])
cache_latency = prom.Histogram("semantic_cache_lookup_seconds", "Lookup latency")
similarity_scores = prom.Histogram(
    "semantic_cache_similarity", "Similarity scores for hits",
    buckets=[0.90, 0.92, 0.94, 0.96, 0.98, 1.0]
)

Cache-Control headers. Give callers the ability to bypass the cache when they need a fresh response:

def handle_request(request: dict, headers: dict) -> str:
    if headers.get("Cache-Control") == "no-cache":
        return call_llm(request)
    if headers.get("Cache-Control") == "no-store":
        response = call_llm(request)
        # Don't store in cache
        return response
    return cache.get_or_call(request, call_llm)

Graceful degradation. If the embedding service or vector store is down, fall through to the LLM directly. The cache is an optimization, not a required component.

def cached_completion_safe(request: dict) -> str:
    try:
        result = cache.get_or_call(request, call_llm)
        return result.response
    except (EmbeddingServiceError, VectorStoreError) as e:
        logger.warning(f"Cache unavailable, falling through: {e}")
        return call_llm(request)

Response attribution. Include metadata indicating whether a response came from cache, so downstream systems can handle cached responses differently if needed (e.g., not using cached responses for evaluation data collection).

Periodic cache quality audits. Sample cached query-response pairs weekly, compute what the current model would produce for those queries, and measure drift. If the model has been updated and cached responses diverge from what the model now produces, accelerate eviction.

Existing Tools and Libraries

Several libraries implement semantic caching with varying levels of sophistication:

GPTCache — open-source semantic caching library supporting multiple embedding backends and vector stores. Handles cache key construction, similarity matching, and eviction policies out of the box.
LangChain CacheBackedEmbeddings + InMemoryCache/RedisCache — basic caching at the chain level, more exact-match than semantic.
LiteLLM — includes a caching layer that can be configured for semantic matching using Redis.
Portkey — managed LLM gateway with built-in semantic caching as a feature.

For most teams, starting with GPTCache or building a thin wrapper around Qdrant/Redis (as shown above) provides sufficient control without taking on a large dependency.

The build-vs-buy spectrum for semantic caching. Most teams should start with GPTCache or a managed gateway, then build custom only if they need specific control over thresholds, eviction, or partitioning.

Summary

Semantic caching reduces LLM costs and latency by returning stored responses for queries that are semantically similar to previous ones. The core mechanism is embedding-based similarity search with a cosine threshold.

Key takeaways:

Threshold is everything. Start at 0.97, tune empirically. False positives (returning wrong cached answers) are worse than cache misses.
Partition cache by non-semantic parameters (model, system prompt, temperature) to prevent cross-contamination.
Self-hosted embedding models eliminate per-request embedding costs and reduce lookup latency to under 10ms.
Two-tier caching (exact match first, semantic fallback) captures both identical queries and paraphrased ones.
ROI scales with model cost. Semantic caching saves real money on GPT-5.6 Sol ($5/$30 per M) but probably isn’t worth the complexity for budget models ($0.10-0.20/$0.40-1.20 per M).
Don’t cache creative, personalized, or time-sensitive queries. The false positive risk outweighs the savings.
Monitor continuously. Track hit rates, false positive rates, and similarity score distributions. A cache that silently serves wrong answers is worse than no cache.
Temperature > 0.3 should probably skip the cache. The same prompt at high temperature is expected to produce varied responses.

The ideal semantic cache candidate is a high-volume application making repetitive, deterministic queries to expensive models — classification pipelines, FAQ systems, structured extraction, and templated generation workflows.