How token-aware rate limiting works for LLM APIs: sliding windows, token budgets, adaptive throttling, and implementation patterns across API gateways and proxy layers.

Token-Based Rate Limiting for LLM APIs

Request-count rate limiting — the kind every web framework ships with — breaks down the moment you point it at an LLM API. A single request might consume 500 tokens or 500,000. Billing is per-token. Quota exhaustion is per-token. But most rate limiters still count requests.

This post covers why token-aware throttling exists, how it works mechanically, the algorithms that make it practical, and concrete implementation patterns for API gateways, proxy layers, and application code.

Why Request-Count Limits Fail for LLM Workloads
The Token Budget Model
Estimation: Counting Tokens Before You Spend Them
Sliding Window Algorithms for Token Budgets
Adaptive Throttling and Backpressure
Provider Rate Limit Behavior
Implementation Patterns
Gateway-Level Token Rate Limiting
Multi-Tenant Token Quotas
Handling Streaming Responses
Failure Modes and Edge Cases
Summary
Further Reading

Why Request-Count Limits Fail for LLM Workloads

Traditional rate limiting assumes requests are roughly equal in cost. An HTTP API serving JSON payloads might vary by 10x between its cheapest and most expensive endpoint. LLM API calls vary by 10,000x. A classification prompt with a one-sentence input and a max_tokens of 5 costs fractions of a cent. A code-generation request stuffing 200K tokens of context into Claude Fable 5.1 at $10/M input tokens costs $2.00 in a single call.

Request-count limits create two failure modes simultaneously:

Under-protection: A single user sends 10 requests per minute, each consuming 100K tokens. At 10 RPM this looks well-behaved. At 1M TPM this blows through a $50/hour budget in minutes.
Over-restriction: A high-volume classification service sends 1,000 requests per minute, each consuming 200 tokens. At 200K TPM this is cheap, but a 100 RPM rate limit blocks 90% of its traffic.

The mismatch extends to provider-side limits. OpenAI, Anthropic, and Google all enforce token-per-minute (TPM) quotas alongside request-per-minute (RPM) quotas. Hitting either one triggers a 429. An application that only tracks RPM will encounter seemingly random 429s whenever its token consumption spikes — and the retry logic won’t know why.

The Token Budget Model

Token-based rate limiting replaces “requests per window” with “tokens per window.” The unit of accounting shifts from the HTTP request to the token, which aligns with how providers bill and how costs accumulate.

A token budget has three dimensions:

Dimension	Description	Typical range
Input TPM	Tokens sent to the model per minute	10K – 10M
Output TPM	Tokens generated by the model per minute	5K – 2M
Combined TPM	Sum of input + output per minute	20K – 12M
Cost per window	Dollar ceiling per time window	$0.10/min – $50/min

Most production systems track input and output separately because they have different per-token prices. GPT-6 Astra charges $10/M input and $50/M output — a 5:1 ratio. Claude Fable 5.1 charges $10/M input and $50/M output. Treating all tokens equally underestimates output cost by 5x.

The cost-based variant converts everything to dollars:

cost = (input_tokens × input_price) + (output_tokens × output_price)

This is more accurate for multi-model setups where the same gateway routes to GPT-6 Luna ($0.10/$0.50 per M) and Claude Fable 5.1 ($10/$50 per M). A token-count limit would need to be set conservatively enough for the expensive model, wasting capacity on the cheap one. A dollar-denominated budget handles heterogeneous model routing naturally.

Token budget check flow: estimate first, then enforce against per-window limits.

Estimation: Counting Tokens Before You Spend Them

The hard problem with token-based rate limiting: you need to know the token count before the request completes, but output token count isn’t known until the model finishes generating.

Input token estimation

Input tokens can be counted exactly using the model’s tokenizer. For OpenAI models, tiktoken with the appropriate encoding gives exact counts. For Anthropic’s Claude models from the 4.7 generation onward, the tokenizer produces ~30% more tokens for the same text compared to earlier models, which matters for budget calculations.

import tiktoken

# OpenAI models (GPT-6 family uses cl200k_base)
enc = tiktoken.get_encoding("cl200k_base")
input_tokens = len(enc.encode(prompt_text))

# For system prompt + messages, count each component
total_input = sum(len(enc.encode(msg["content"])) for msg in messages)
total_input += 4 * len(messages)  # per-message overhead tokens

For models without a publicly available tokenizer, a character-based heuristic works as a fallback: English text averages roughly 4 characters per token for GPT-family tokenizers, though this varies by language (CJK text can be 1.5–2 characters per token). The heuristic is wrong often enough that it should only be used for pre-check estimation, not for billing reconciliation.

Output token estimation

Output tokens are unknown ahead of time. Three strategies:

1. Use max_tokens as the upper bound. If the request sets max_tokens: 4096, reserve 4096 output tokens from the budget. After the response completes, reconcile the actual count and release the unused reservation. This is conservative — most responses use 20–60% of max_tokens — but it prevents over-commitment.

2. Use historical averages per endpoint/use-case. A summarization endpoint that consistently produces 200–400 token outputs can reserve 500 tokens with high confidence. This requires tracking actual output distributions per route.

3. Hybrid: reserve at max_tokens, reconcile on completion. The budget ledger temporarily holds the full max_tokens reservation. When the response finishes (or streams its final chunk), the ledger credits back unused tokens. This creates brief periods where the effective limit is tighter than intended, but prevents budget overruns.

Reserve-then-reconcile: debit the full estimate upfront, credit back the difference on completion.

Sliding Window Algorithms for Token Budgets

Fixed-window counters (reset every 60 seconds) have a well-known edge case: a burst at the window boundary can consume 2x the limit in rapid succession. For token budgets, this means a brief period where the effective spend rate is double the intended ceiling.

Fixed window

Simplest implementation. A counter per window (e.g., minute) tracks cumulative token consumption. Resets to zero at the start of each window.

import time
import threading

class FixedWindowTokenLimiter:
    def __init__(self, tokens_per_minute: int):
        self.limit = tokens_per_minute
        self.window_seconds = 60
        self.current_window = int(time.time()) // self.window_seconds
        self.consumed = 0
        self.lock = threading.Lock()

    def try_consume(self, token_count: int) -> tuple[bool, float]:
        with self.lock:
            now_window = int(time.time()) // self.window_seconds
            if now_window != self.current_window:
                self.current_window = now_window
                self.consumed = 0

            if self.consumed + token_count > self.limit:
                retry_after = self.window_seconds - (time.time() % self.window_seconds)
                return False, retry_after

            self.consumed += token_count
            return True, 0.0

Sliding window log

Stores a timestamped log of every consumption event. To check the budget, sum all entries within the last 60 seconds. Accurate but memory-intensive — a high-throughput system making 1,000 requests/minute stores 1,000 entries per window per tenant.

Sliding window counter

A practical compromise. Track the current and previous window’s totals, then weight the previous window by the fraction of overlap:

effective_count = prev_window_count × (1 - elapsed_fraction) + current_window_count

This smooths bursts at window boundaries without the memory cost of a full log.

class SlidingWindowTokenLimiter:
    def __init__(self, tokens_per_minute: int):
        self.limit = tokens_per_minute
        self.window = 60
        self.prev_count = 0
        self.curr_count = 0
        self.curr_window_start = int(time.time()) // self.window
        self.lock = threading.Lock()

    def try_consume(self, token_count: int) -> tuple[bool, float]:
        with self.lock:
            now = time.time()
            window_start = int(now) // self.window

            if window_start != self.curr_window_start:
                if window_start == self.curr_window_start + 1:
                    self.prev_count = self.curr_count
                else:
                    self.prev_count = 0
                self.curr_count = 0
                self.curr_window_start = window_start

            elapsed = now - (window_start * self.window)
            fraction = elapsed / self.window
            weighted = self.prev_count * (1 - fraction) + self.curr_count

            if weighted + token_count > self.limit:
                # Estimate when enough budget frees up
                needed = weighted + token_count - self.limit
                retry_after = (needed / max(self.prev_count, 1)) * self.window
                return False, min(retry_after, self.window)

            self.curr_count += token_count
            return True, 0.0

Token bucket (leak bucket variant)

A token bucket refills at a constant rate (e.g., 10,000 tokens per second for a 600K TPM limit). Each request drains the bucket by its estimated token count. If the bucket is empty, the request waits or is rejected.

This algorithm naturally allows bursts up to the bucket capacity while enforcing a sustained rate. For LLM workloads, the burst capacity should be set to at least the largest expected single-request token count — otherwise, a single large-context request will always be rejected.

Algorithm	Burst handling	Memory per tenant	Implementation complexity
Fixed window	Poor (2x at boundary)	O(1)	Low
Sliding window counter	Good	O(1)	Low
Sliding window log	Exact	O(n) per window	Medium
Token bucket	Configurable burst	O(1)	Medium

For most LLM gateway use cases, sliding window counter or token bucket provides the right tradeoff. Fixed windows are acceptable for internal services where boundary bursts are tolerable.

Adaptive Throttling and Backpressure

Static token budgets assume a constant capacity. Real LLM APIs don’t behave this way. Provider-side rate limits change based on account tier, time of day, and system load. A gateway that blindly enforces a local budget will either waste available capacity or overshoot and eat 429s.

Reading provider headers

All major providers return rate limit state in response headers:

Header	OpenAI	Anthropic	Google
Remaining requests	`x-ratelimit-remaining-requests`	`anthropic-ratelimit-requests-remaining`	`x-ratelimit-remaining-requests`
Remaining tokens	`x-ratelimit-remaining-tokens`	`anthropic-ratelimit-tokens-remaining`	varies by endpoint
Reset time	`x-ratelimit-reset-requests`	`anthropic-ratelimit-requests-reset`	`x-ratelimit-reset-requests`
Token limit	`x-ratelimit-limit-tokens`	`anthropic-ratelimit-tokens-limit`	varies

An adaptive limiter reads these headers on every response and adjusts its local budget to track the provider’s actual remaining capacity:

def update_from_headers(self, headers: dict):
    remaining = int(headers.get("x-ratelimit-remaining-tokens", self.limit))
    reset_at = headers.get("x-ratelimit-reset-tokens")

    # Adjust local budget to match provider state
    if remaining < self.get_local_remaining():
        # Provider is more restrictive — tighten local budget
        self.set_remaining(remaining)

    if reset_at:
        self.set_reset_time(parse_reset(reset_at))

Adaptive throttling synchronizes the local budget with the provider’s actual remaining capacity.

Backpressure with queuing

Instead of rejecting requests immediately when the budget is exhausted, a queue-based approach holds requests and drains them as budget becomes available. This smooths bursty workloads and improves throughput utilization.

The queue needs a priority mechanism. In a multi-tenant system, high-priority tenants (or latency-sensitive endpoints) should drain first. A weighted fair queue works:

import heapq
import asyncio
from dataclasses import dataclass, field

@dataclass(order=True)
class PendingRequest:
    priority: int
    estimated_tokens: int = field(compare=False)
    future: asyncio.Future = field(compare=False)

class TokenAwareQueue:
    def __init__(self, limiter):
        self.limiter = limiter
        self.queue: list[PendingRequest] = []
        self.drainer_task = None

    async def enqueue(self, estimated_tokens: int, priority: int = 5) -> bool:
        future = asyncio.get_event_loop().create_future()
        heapq.heappush(self.queue, PendingRequest(priority, estimated_tokens, future))
        if not self.drainer_task:
            self.drainer_task = asyncio.create_task(self._drain())
        return await future

    async def _drain(self):
        while self.queue:
            pending = self.queue[0]
            allowed, retry_after = self.limiter.try_consume(pending.estimated_tokens)
            if allowed:
                heapq.heappop(self.queue)
                pending.future.set_result(True)
            else:
                await asyncio.sleep(retry_after)
        self.drainer_task = None

Queue depth should be bounded. An unbounded queue in front of a rate-limited LLM API creates unbounded latency. A practical upper bound: if the queue depth exceeds 2x the per-window budget, reject new arrivals with a 503 and an estimated wait time.

Provider Rate Limit Behavior

Each provider enforces token limits differently, and the specifics matter for building a limiter that works across them.

OpenAI

OpenAI enforces both RPM and TPM limits. The TPM limit applies to the sum of input and output tokens. Limits vary by model and tier — GPT-6 Astra has different limits than GPT-6 Luna. When you exceed either limit, you get a 429 with a Retry-After header. Batch API requests have separate, higher quotas and are not subject to real-time TPM limits (they’re queued and processed within 24 hours).

Cached input tokens (via prompt caching) still count against TPM for rate-limiting purposes, even though they’re billed at a discount. This is a common surprise: caching saves money but doesn’t save rate limit headroom.

Anthropic

Anthropic enforces three limits simultaneously: RPM, input TPM, and output TPM — tracked separately. Exceeding any one triggers a 429. The headers are anthropic-ratelimit-tokens-remaining for the combined view and separate input/output counters. Claude Haiku 5.5’s tiered pricing (different rates above and below 100K prompt tokens) doesn’t affect rate limits, only billing.

Google

Gemini API limits are per-project and per-model. Gemini 4 Argon and Flash models have per-minute and per-day quotas. The Gemini API returns rate limit information in response headers, though the header names and formats have changed across API versions.

Practical implications

A gateway routing across all three providers needs three different header-parsing strategies, three different budget tracking models (combined TPM vs. split input/output TPM), and three different retry strategies. This is one of the strongest arguments for using a dedicated LLM gateway layer rather than building rate limit awareness into application code.

Implementation Patterns

Pattern 1: Middleware in the application layer

The simplest approach. A middleware in the application’s HTTP stack estimates tokens before each outbound LLM call and enforces a local budget.

// Express middleware for token-aware rate limiting
import { encoding_for_model } from "tiktoken";

const enc = encoding_for_model("gpt-6-astra");
const BUDGET_TPM = 500_000;
let windowStart = Date.now();
let consumed = 0;

function tokenRateLimiter(req, res, next) {
  const now = Date.now();
  if (now - windowStart > 60_000) {
    windowStart = now;
    consumed = 0;
  }

  const inputTokens = enc.encode(JSON.stringify(req.body.messages)).length;
  const maxOutput = req.body.max_tokens || 4096;
  const estimated = inputTokens + maxOutput;

  if (consumed + estimated > BUDGET_TPM) {
    const retryAfter = Math.ceil((60_000 - (now - windowStart)) / 1000);
    res.status(429).json({
      error: "Token budget exceeded",
      retry_after: retryAfter,
      consumed_tpm: consumed,
      budget_tpm: BUDGET_TPM,
    });
    return;
  }

  consumed += estimated;

  // After response, reconcile actual usage
  const origEnd = res.end;
  res.end = function (...args) {
    const actualOutput = res.getHeader("x-actual-output-tokens");
    if (actualOutput) {
      consumed -= (maxOutput - parseInt(actualOutput));
    }
    origEnd.apply(res, args);
  };

  next();
}

This pattern works for single-instance applications. It breaks in multi-instance deployments because each instance has its own counter with no shared state.

Pattern 2: Redis-backed distributed limiter

For horizontally scaled services, the budget state lives in Redis. A Lua script makes the check-and-increment atomic:

-- token_rate_limit.lua
-- KEYS[1] = rate limit key (e.g., "trl:{tenant}:{model}:{window}")
-- ARGV[1] = token count to consume
-- ARGV[2] = max tokens per window
-- ARGV[3] = window duration in seconds

local key = KEYS[1]
local tokens = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local window = tonumber(ARGV[3])

local current = tonumber(redis.call("GET", key) or "0")

if current + tokens > limit then
    local ttl = redis.call("TTL", key)
    if ttl < 0 then ttl = window end
    return {0, ttl, current, limit}
end

local new_total = redis.call("INCRBY", key, tokens)
if new_total == tokens then
    -- First entry in this window, set expiry
    redis.call("EXPIRE", key, window)
end

return {1, 0, new_total, limit}

import redis
import time

class RedisTokenLimiter:
    def __init__(self, redis_client: redis.Redis, limit_tpm: int):
        self.r = redis_client
        self.limit = limit_tpm
        self.window = 60
        self.script = self.r.register_script(open("token_rate_limit.lua").read())

    def try_consume(self, tenant: str, model: str, tokens: int) -> tuple[bool, float]:
        window_id = int(time.time()) // self.window
        key = f"trl:{tenant}:{model}:{window_id}"
        result = self.script(keys=[key], args=[tokens, self.limit, self.window])
        allowed = bool(result[0])
        retry_after = float(result[1])
        return allowed, retry_after

    def reconcile(self, tenant: str, model: str, tokens_to_return: int):
        window_id = int(time.time()) // self.window
        key = f"trl:{tenant}:{model}:{window_id}"
        self.r.decrby(key, tokens_to_return)

Redis-backed limiting adds ~1ms of latency per check. For LLM calls that take 500ms–30s, this is negligible.

Redis-backed distributed token limiting across multiple application instances.

Pattern 3: Token-aware API gateway

The most robust approach places token rate limiting in the API gateway itself, before traffic reaches application instances. This centralizes enforcement, handles multi-tenant isolation, and works across different backend services hitting the same LLM provider.

Kong, Envoy, and NGINX can all be extended with custom plugins for token-aware limiting. A purpose-built LLM gateway (like LiteLLM’s proxy mode) often has this built in.

# LiteLLM proxy config with token-based budget
model_list:
  - model_name: gpt-6-astra
    litellm_params:
      model: openai/gpt-6-astra
      api_key: sk-...
      tpm_limit: 1000000
      rpm_limit: 500
      max_budget: 100.0  # USD per day

  - model_name: claude-opus-5-5
    litellm_params:
      model: anthropic/claude-opus-5-5
      api_key: sk-ant-...
      tpm_limit: 800000
      rpm_limit: 400
      max_budget: 80.0

general_settings:
  database_url: postgresql://...
  master_key: sk-master-...

litellm_settings:
  max_budget: 500.0  # Global daily budget USD
  budget_duration: 24h

Gateway-Level Token Rate Limiting

A dedicated LLM gateway inserts a full token-accounting layer between clients and providers. The flow:

Gateway request flow: authenticate, estimate, enforce, then route.

The gateway’s responsibilities:

Authenticate and identify the tenant — API key, OAuth token, or workload identity
Estimate input tokens using the appropriate tokenizer for the target model
Reserve output tokens using max_tokens from the request or a configured default
Check per-tenant, per-model budget against sliding window counters
Forward or reject with appropriate headers
Reconcile on response completion, crediting unused output token reservations

After the response completes, the gateway also reads the provider’s usage headers (most providers return usage.prompt_tokens and usage.completion_tokens in the response body) and reconciles against the reservation. If the actual usage was lower than estimated, the difference is credited back.

Response body reconciliation

Provider APIs return exact token counts in the response:

{
  "usage": {
    "prompt_tokens": 1247,
    "completion_tokens": 892,
    "total_tokens": 2139
  }
}

The gateway parses this on every response to:

Correct estimation errors
Credit back over-reserved output tokens
Feed accurate data into cost tracking

For streaming responses, the usage block typically appears only in the final chunk (the one with finish_reason: "stop"). The gateway must buffer or accumulate chunks to catch it.

Multi-Tenant Token Quotas

Production systems serving multiple teams, customers, or applications need per-tenant token budgets. The data model:

@dataclass
class TenantQuota:
    tenant_id: str
    model: str
    input_tpm: int        # Max input tokens per minute
    output_tpm: int       # Max output tokens per minute
    daily_cost_usd: float # Max spend per 24h
    burst_multiplier: float = 1.5  # Allow 1.5x burst for 10s

# Example quotas
quotas = {
    "team-ml": TenantQuota("team-ml", "gpt-6-astra",
                           input_tpm=2_000_000, output_tpm=500_000,
                           daily_cost_usd=200.0),
    "team-support": TenantQuota("team-support", "claude-haiku-5-5",
                                input_tpm=5_000_000, output_tpm=1_000_000,
                                daily_cost_usd=50.0),
}

The hierarchy of limits typically looks like:

Level	Scope	Purpose
Global	All tenants combined	Don’t exceed provider account limits
Per-tenant	One team/customer	Cost isolation and fairness
Per-model	One model within a tenant	Prevent expensive model overuse
Per-user	Individual API key	Prevent single-user abuse

Enforcement order matters. Check global limits first (cheapest to check, protects everyone), then per-tenant, then per-model, then per-user. Short-circuit on the first rejection to avoid unnecessary Redis round-trips.

Hierarchical budget enforcement: global → tenant → model.

Fairness across tenants

When multiple tenants share a provider account, one tenant’s burst can starve others. Two mitigations:

Weighted fair sharing: Each tenant gets a guaranteed minimum allocation (their “reserved” budget) plus a share of any unreserved capacity. If tenant A’s reserved budget is 200K TPM and tenant B’s is 800K TPM, but tenant B is idle, tenant A can temporarily use up to the global limit.

Borrowing with payback: A tenant can briefly exceed their per-minute budget by “borrowing” from the next minute’s allocation. The next window starts with a reduced budget. This smooths legitimate bursts (e.g., a batch job starting up) without allowing sustained overspend.

Handling Streaming Responses

Streaming complicates token accounting because the output token count isn’t known until the stream ends. Three strategies:

Strategy 1: Reserve max, reconcile on stream end

Debit max_tokens from the output budget when the request starts. When the stream emits its final chunk, read the actual completion token count and credit back the difference. This is the simplest correct approach.

The drawback: during streaming (which can last 30+ seconds for long generations), the budget shows less available capacity than actually exists. For high-concurrency systems, this phantom reservation can cause unnecessary rejections.

Strategy 2: Incremental accounting

Count tokens as they stream. Each SSE chunk contains one or more tokens. Increment the output counter as chunks arrive.

async def stream_with_accounting(response_stream, limiter, tenant):
    output_tokens = 0
    async for chunk in response_stream:
        if chunk.choices and chunk.choices[0].delta.content:
            # Approximate: 1 chunk ≈ 1 token for most providers
            output_tokens += 1
            limiter.increment_output(tenant, 1)
        yield chunk

    # Final reconciliation using actual usage from last chunk
    if hasattr(chunk, 'usage') and chunk.usage:
        actual = chunk.usage.completion_tokens
        diff = output_tokens - actual
        if diff != 0:
            limiter.adjust(tenant, diff)

The approximation of “one chunk ≈ one token” is mostly correct for OpenAI and Anthropic (they typically emit one token per SSE event), but not guaranteed. Some providers batch multiple tokens per event, especially at high throughput.

Strategy 3: Hybrid with early termination

Reserve a moderate estimate (e.g., 50th percentile of historical output lengths for this endpoint). If the stream exceeds the reservation, either:

Allow it to complete (soft limit) and debit the overage from the next window
Terminate the stream by closing the connection (hard limit)

Hard termination is aggressive. It wastes the tokens already generated and returns an incomplete response. But for cost-protection scenarios (a runaway generation consuming a model with $50/M output pricing), it might be the only safe option.

Token accounting through a streaming response lifecycle.

Failure Modes and Edge Cases

Race conditions in distributed limiters

Even with Redis Lua scripts, there’s a window between checking the budget and the request actually consuming tokens at the provider. If two requests arrive simultaneously, both pass the budget check, but their combined consumption exceeds the limit. The provider returns a 429 for one of them.

Mitigation: use pessimistic reservation (debit before sending) rather than optimistic accounting (debit after receiving). The Lua script in the Redis pattern above does this correctly — it increments the counter atomically as part of the check.

Tokenizer mismatch

Using the wrong tokenizer produces wrong estimates. GPT-6 Astra uses a different vocabulary size than Claude Opus 5.5. A gateway routing to multiple providers needs multiple tokenizers or a fast approximation layer.

A practical approach: maintain exact tokenizers for the top 2–3 models by volume, and use a character-based heuristic (÷ 4 for English, ÷ 2 for CJK) for the rest. Track estimation error rates and alert when they exceed 15%.

System prompt and tool definition overhead

Requests with function/tool definitions can consume substantial hidden tokens. A set of 20 tool definitions can easily add 2,000–5,000 tokens to input that aren’t visible in the messages array. The token estimator must include tool schemas and system prompts in its count.

def estimate_total_input(messages, tools=None, system=None):
    total = 0
    if system:
        total += count_tokens(system) + 4  # system prompt overhead
    for msg in messages:
        total += count_tokens(msg["content"]) + 4
    if tools:
        # Tool definitions are serialized to JSON-ish format in the prompt
        total += count_tokens(json.dumps(tools))
    total += 2  # end-of-turn overhead
    return total

Prompt caching interactions

OpenAI and Anthropic both support prompt caching (automatic for OpenAI, explicit cache_control blocks for Anthropic). Cached input tokens are billed at a discount (e.g., $0.20/M for cached reads on Claude Opus 5.5 vs. $4/M for uncached). But they still count toward the provider’s TPM rate limit.

This creates a counterintuitive situation: prompt caching saves money but doesn’t increase throughput. A token rate limiter that factors in caching discounts for cost budgeting but not for TPM budgeting handles this correctly.

Long-context requests hitting provider thresholds

Some providers have tiered pricing based on prompt size. OpenAI’s GPT-6 Astra bills at 2x for prompts over 272K input tokens. Claude Haiku 5.5 charges 5x for prompts over 100K tokens ($0.50/$2.50 vs. $0.10/$0.50 per M). A cost-based budget must account for these tiers:

def estimate_cost_anthropic_haiku_55(input_tokens: int, output_tokens: int) -> float:
    if input_tokens <= 100_000:
        input_cost = input_tokens * 0.10 / 1_000_000
        output_cost = output_tokens * 0.50 / 1_000_000
    else:
        input_cost = input_tokens * 0.50 / 1_000_000
        output_cost = output_tokens * 2.50 / 1_000_000
    return input_cost + output_cost

Budget exhaustion near window boundaries

If a budget window resets every 60 seconds and a large request arrives at second 59, it might consume most of the next window’s budget before any other requests can use it. The sliding window counter algorithm mitigates this, but doesn’t eliminate it for very large single requests.

For models supporting 1M+ context windows, a single request can consume the entire per-minute budget. The limiter should have a per-request maximum (e.g., no single request can consume more than 50% of the window budget) to prevent complete starvation.

Summary

Token-based rate limiting replaces request counts with token counts (or dollar costs) as the unit of accounting, aligning enforcement with how LLM APIs actually bill and throttle.

Key takeaways:

Estimate input tokens exactly using the correct tokenizer for the target model. Estimate output tokens by reserving max_tokens and reconciling on completion.
Sliding window counters or token buckets handle burst smoothing with O(1) memory per tenant. Fixed windows work for simple cases but allow 2x bursts at boundaries.
Adaptive throttling reads provider rate limit headers and tightens local budgets to avoid 429s. This is essential for any system that can’t afford retry latency.
Distributed enforcement requires atomic check-and-increment in Redis (or equivalent). Single-instance counters break under horizontal scaling.
Streaming responses need reserve-then-reconcile accounting. The output token count isn’t known until the final chunk.
Cost-based budgets handle multi-model routing naturally. A $1/minute budget works whether traffic goes to GPT-6 Luna at $0.10/$0.50 or Claude Fable 5.1 at $10/$50.
Per-tenant hierarchies (global → tenant → model → user) prevent one consumer from starving others. Check in order, short-circuit on first rejection.
Include tool definitions and system prompts in token estimates. They’re invisible in the messages array but consume real tokens.