Office Hours — What's the best strategy for controlling costs and enforcing rate limits on LLM API calls in production?

What’s the best strategy for controlling costs and enforcing rate limits on LLM API calls in production?

Cost control and rate limiting are two separate problems that many teams conflate, and that mistake costs them money and uptime. Rate limits protect your service from thundering herd and quota breaches. Cost controls protect your budget from runaway inference. You need both, and they require different architecture.

The Cost Problem Is Worse Than You Think

Your actual per-request costs are higher than the per-token price suggests. Coinbase cut AI spending in half by routing to Chinese models with intelligent caching, which tells you something important: the easy wins come from architectural decisions, not negotiating better rates. Every retry, every failed parse, every hallucinated tool call that triggers a re-run multiplies your true cost. A single bad agent loop can consume thousands of tokens trying to recover from a model error.

Track three metrics separately: token cost (tokens × rate), failure cost (cost of retry loops), and opportunity cost (money spent on tasks that didn’t deliver business value). Most teams only measure the first one.

Hard Limits, Not Soft Guidance

Set a hard monthly budget cap in your LLM provider’s dashboard. Not a “warning at 80%” alert that gets ignored. A cap that stops new requests when hit. This sounds brutal, but it’s the only reliable way to prevent a bug from costing you ten thousand dollars in four hours. Anthropic, OpenAI, and other major providers support this. Use it.

Within that, implement per-request cost budgets. If a single request is going to cost more than a threshold you set, fail loudly before making the call. This is especially critical for agents that can chain multiple model calls. Your agent shouldn’t be allowed to call GPT-5.5 fifteen times in sequence if you’ve only budgeted for three. Decide that before the request starts.

# Pseudo-code, not a real library
class CostGuard:
    def __init__(self, monthly_budget=1000, per_request_max=50):
        self.monthly_budget = monthly_budget
        self.per_request_max = per_request_max
        self.month_spent = get_month_spent_from_provider()
    
    def can_afford_request(self, estimated_tokens, model):
        estimated_cost = estimate_cost(estimated_tokens, model)
        remaining = self.monthly_budget - self.month_spent
        
        if estimated_cost > self.per_request_max:
            raise BudgetError(f"Single request ${estimated_cost} exceeds limit")
        if estimated_cost > remaining:
            raise BudgetError(f"Insufficient monthly budget: ${remaining} left")
        
        return True

Rate Limiting Is About Protecting Your Service, Not Just Your Wallet

Rate limits serve two purposes. First, they prevent your service from melting when a customer’s code goes haywire and starts hammering your endpoint. Second, they enforce fair allocation when multiple services share the same API quota. Conflating these leads to bad decisions.

Use a token-bucket algorithm for rate limiting. It’s standard, well-understood, and handles burst traffic cleanly. Libraries like Redis or in-process libraries handle this. The bucket fills at your target rate (say, 10k tokens per second across all customers), and requests draw from it. When the bucket is empty, requests wait or fail depending on your SLA.

Set your bucket size to handle realistic spikes in traffic, not to prevent all variance. A three-second burst capacity is reasonable for most services. Longer than that and you’re just delaying an inevitable failure.

# Using a Redis-backed token bucket
import redis
import time

class RateLimiter:
    def __init__(self, redis_client, key, rate_per_second, burst_capacity):
        self.redis = redis_client
        self.key = key
        self.rate = rate_per_second
        self.capacity = burst_capacity
    
    def consume(self, tokens):
        now = time.time()
        pipe = self.redis.pipeline()
        
        # Get last refill time and current tokens
        pipe.get(f"{self.key}:last_refill")
        pipe.get(f"{self.key}:tokens")
        results = pipe.execute()
        
        last_refill = float(results[0] or now)
        current = float(results[1] or self.capacity)
        
        # Refill based on elapsed time
        elapsed = now - last_refill
        current = min(self.capacity, current + elapsed * self.rate)
        
        if current >= tokens:
            current -= tokens
            pipe.set(f"{self.key}:tokens", current)
            pipe.set(f"{self.key}:last_refill", now)
            pipe.execute()
            return True
        
        return False

Separate Cost Control From Request Throttling

Don’t use rate limiting as a proxy for cost control. They solve different problems. You can have a service that’s rate-limited (requests per second) but still blowing budget (because each request calls expensive models). Conversely, you can have high throughput with low cost if you route to cheaper models intelligently.

Implement a separate cost-aware router that decides which model to call before the request is made. If you have a task that works with Claude Haiku 4.5 or Claude Opus 4.8, compute the expected token cost of both, and pick the cheaper one if latency allows. This is routing, not rate limiting. They’re different layers.

Coinbase’s approach of routing to Chinese models with different cost profiles shows that the money is in model selection, not in rate limits. Their 50% cost reduction came from choosing cheaper models plus aggressive caching, not from slowing down requests.

Caching Is Your Biggest Lever

If you’re not caching LLM responses, you’re leaving thousands on the table. Both OpenAI and Anthropic support prompt caching. Cache your system prompts, cached contexts, and any static data you’re including in requests. Cached tokens cost 10% of normal token costs (at OpenAI and Anthropic).

For a typical RAG system where you’re including a large corpus, caching the corpus alone can cut costs by 40%. The first request to a user is expensive; the second and third are cheap.

Set your cache TTL based on how stale your data can be. If you’re doing Q&A over internal docs that change weekly, a 24-hour cache is fine. If you’re pulling live data, cache for five minutes. Most teams don’t cache at all, which is a self-inflicted wound.

Monitoring and Alerting

Track cost per unit of business value, not total cost. Cost per successful request completion. Cost per customer interaction. Cost per correct answer. If your cost per successful request is trending up while your model doesn’t change, your failure rate is increasing, and that’s a hidden signal that the system is degrading.

Alert on cost velocity, not absolute cost. If your daily spend doubles in 24 hours, that’s worth investigating immediately, even if the absolute number is still in budget. Use anomaly detection (comparing to a rolling baseline) rather than static thresholds.

Log which model, which customer, which task type, and which outcome every request. When you inevitably need to understand where costs went, you’ll be grateful for this.

The Tradeoff You Can’t Avoid

Better models cost more. Cheaper models fail more often. More aggressive caching increases latency. Tighter rate limits improve fairness and stability but can frustrate power users. You’re not optimizing for one thing. You’re balancing cost, latency, reliability, and fairness.

Most teams should start with this ordering: first, set a hard monthly budget cap. Second, implement per-request cost budgets. Third, add caching for static content. Fourth, implement rate limiting to prevent cascading failures. Fifth, add a cost-aware model router if you have multiple models available.

Don’t skip step one and jump to step five. The order matters.

Bottom line: Set a hard monthly budget cap that actually stops requests, implement per-request cost limits before agents spiral, and cache aggressively. Rate limiting and routing are secondary optimizations; the real leverage is in preventing unnecessary retries and choosing cheaper models when they work.

Question via Hacker News