How token-aware rate limiting works for LLM APIs: sliding windows, token budgets, adaptive throttling, and implementation patterns across API gateways and proxy layers.

Token-Based Rate Limiting for LLM APIs

Traditional rate limiting counts requests. A GET to /users/123 costs roughly the same as any other GET to /users/456. LLM APIs break this assumption entirely — a single request can consume 1,000 tokens or 1,000,000 tokens, with cost and latency varying by three orders of magnitude. Request-count limits are the wrong abstraction for AI workloads.

Token-based rate limiting replaces the request counter with a token budget, throttling based on actual resource consumption rather than request frequency. Every major LLM provider now enforces token-per-minute (TPM) limits alongside request-per-minute (RPM) limits, and any serious production system needs to do the same internally.

Why Request-Count Limits Fail
The Token Budget Model
Sliding Windows vs Fixed Windows vs Token Buckets
Estimating Token Consumption Before Completion
Provider Rate Limit Headers
Architecture: Where Token Limiting Lives
Implementation Patterns
Adaptive Throttling and Backpressure
Multi-Tenant Token Budgeting
Failure Modes and Edge Cases
Summary
Further Reading

Why Request-Count Limits Fail

A request-per-minute limit of 100 RPM is meaningless when one request sends a 200-token prompt and another sends a 500,000-token document with a 50,000-token response. The second request consumes ~2,750x more compute, but both count as “1 request.”

The mismatch creates two problems simultaneously. Low-token requests get throttled too early — a chatbot sending quick back-and-forth messages hits RPM limits long before it stresses the model. High-token requests slip through unchecked — a single RAG pipeline stuffing 800K tokens of context into a Claude Fable 5.1 request consumes enormous GPU-seconds while counting as just one tick on the RPM counter.

Request-count limits block the lightweight workload while allowing the expensive one through.

The economics make this worse. At GPT-6 Astra pricing ($10/$50 per M tokens), a single 500K-input, 50K-output request costs $13.75 (the over-272K surcharge applies). A hundred 200-token-input, 500-token-output requests cost about $2.70 total. An RPM limit treats these as equivalent load.

Every major provider enforces both RPM and TPM limits for this reason, and the TPM limit is almost always the binding constraint for production workloads.

The Token Budget Model

Token-based rate limiting assigns each client (or tenant, or API key, or organization) a token budget — typically expressed as tokens per minute (TPM) — and tracks consumption against that budget.

The budget has two components that must be tracked separately:

Component	Why separate
Input tokens	Known before the request completes (or estimable from the prompt)
Output tokens	Unknown until generation finishes; varies with `max_tokens`, sampling, and stop sequences
Total tokens	Input + output; this is what most TPM limits count

Some providers split their limits. OpenAI enforces separate input TPM and output TPM limits on certain tiers. Anthropic’s rate limits are expressed as combined input+output TPM. Google’s Gemini API enforces TPM on total tokens.

The core challenge: you need to make an admission decision (allow or reject the request) before you know the output token count. This is the fundamental difference from traditional rate limiting, where the cost of a request is known at admission time.

Token admission requires estimating total consumption before the LLM generates output.

Sliding Windows vs Fixed Windows vs Token Buckets

Three algorithms dominate token-based rate limiting, each with different tradeoff profiles.

Fixed Window

Divide time into 60-second buckets. Each bucket has a TPM budget. Reset to zero at the boundary.

Problem: burst at the boundary. A client can use its full budget in the last second of window N and the first second of window N+1, effectively doubling throughput over a 2-second span.

# Fixed window — simple but allows boundary bursts
import time

class FixedWindowTokenLimiter:
    def __init__(self, tpm_limit: int, window_seconds: int = 60):
        self.tpm_limit = tpm_limit
        self.window_seconds = window_seconds
        self.current_window = 0
        self.tokens_used = 0

    def _get_window(self) -> int:
        return int(time.time()) // self.window_seconds

    def try_consume(self, tokens: int) -> bool:
        window = self._get_window()
        if window != self.current_window:
            self.current_window = window
            self.tokens_used = 0
        if self.tokens_used + tokens > self.tpm_limit:
            return False
        self.tokens_used += tokens
        return True

Sliding Window Log

Track every request’s timestamp and token count. On each new request, sum tokens from all requests within the last 60 seconds. No boundary burst problem, but memory usage grows linearly with request count.

Sliding Window Counter

A hybrid: maintain counters for the current and previous fixed windows, then interpolate based on where you are in the current window. If you’re 40% through the current window, count 60% of the previous window’s tokens plus 100% of the current window’s tokens. Good approximation of the sliding log with O(1) memory.

Token Bucket

The token bucket algorithm is probably the best fit for LLM rate limiting. Tokens (the rate-limiting kind, not the NLP kind — the naming collision is unfortunate) refill at a constant rate. Requests drain from the bucket. If the bucket is empty, the request is rejected or queued.

import time

class TokenBucketLimiter:
    def __init__(self, tpm_limit: int, burst_multiplier: float = 1.5):
        self.capacity = int(tpm_limit * burst_multiplier)
        self.tokens = self.capacity  # start full
        self.refill_rate = tpm_limit / 60.0  # tokens per second
        self.last_refill = time.monotonic()

    def try_consume(self, token_count: int) -> tuple[bool, float]:
        now = time.monotonic()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now

        if self.tokens >= token_count:
            self.tokens -= token_count
            return True, 0.0
        else:
            wait_seconds = (token_count - self.tokens) / self.refill_rate
            return False, wait_seconds

The burst_multiplier matters. Setting it to 1.0 means no burst tolerance — a client using tokens at exactly the sustained rate has zero headroom. 1.5x allows short bursts (e.g., a batch of parallel requests at startup) without exceeding the average rate over a minute.

Algorithm	Memory	Burst handling	Accuracy	Best for
Fixed window	O(1)	Boundary bursts possible	Low	Simple internal limits
Sliding window log	O(n)	No bursts	Exact	Low-volume, audit-heavy
Sliding window counter	O(1)	Approximate	Good enough	Most production systems
Token bucket	O(1)	Configurable burst	Tunable	LLM workloads

Evolution path: most production systems land on token bucket or sliding window counter.

Estimating Token Consumption Before Completion

The admission control problem: you receive a request with a prompt and a max_tokens parameter. Input tokens can be counted exactly (run the tokenizer). Output tokens are unknown — the model might generate 10 tokens or 100,000.

Three strategies:

1. Reserve max_tokens pessimistically

When a request arrives, deduct input_tokens + max_tokens from the budget. When the request completes, refund the difference between max_tokens and actual_output_tokens.

class ReservationLimiter:
    def __init__(self, tpm_limit: int):
        self.bucket = TokenBucketLimiter(tpm_limit)
        self.reservations: dict[str, int] = {}

    def admit(self, request_id: str, input_tokens: int, max_tokens: int) -> bool:
        reserved = input_tokens + max_tokens
        allowed, _ = self.bucket.try_consume(reserved)
        if allowed:
            self.reservations[request_id] = max_tokens
        return allowed

    def complete(self, request_id: str, actual_output_tokens: int):
        reserved_output = self.reservations.pop(request_id, 0)
        refund = reserved_output - actual_output_tokens
        if refund > 0:
            self.bucket.tokens = min(
                self.bucket.capacity,
                self.bucket.tokens + refund
            )

Tradeoff: safe but wasteful. If clients set max_tokens: 128000 but typically generate 2,000 tokens, the system reserves 64x more budget than needed, severely underutilizing capacity.

2. Estimate output from historical data

Track the average output/input ratio per client, model, or endpoint. Use that ratio to estimate output tokens. More accurate than reserving max, but wrong for bursty workloads.

3. Hybrid: estimate with a safety cap

Use historical averages for the estimate but cap at some fraction of max_tokens (e.g., min(historical_estimate * 1.3, max_tokens * 0.5)). Refund overestimates, charge underestimates against the next request.

The reservation-refund cycle is the standard pattern for handling unknown output costs.

Tokenizer choice matters

Counting input tokens requires running the same tokenizer the model uses. For OpenAI models, tiktoken with the correct encoding (e.g., o200k_base for recent models). For Anthropic models from Claude 4.7 onward, the tokenizer emits ~30% more tokens for the same text than the pre-4.7 tokenizer — using the wrong tokenizer produces estimates that are off by 30%.

For proxy/gateway layers that support multiple providers, maintaining multiple tokenizers is overhead. A reasonable approximation: len(text) / 3.5 gives a rough byte-to-token estimate for English text across most modern tokenizers. It’s wrong by ±20%, but for rate limiting (not billing), that’s often acceptable.

Provider Rate Limit Headers

Every major LLM provider returns rate limit state in response headers. These are essential for client-side token-aware throttling.

OpenAI

x-ratelimit-limit-requests: 10000
x-ratelimit-limit-tokens: 2000000
x-ratelimit-remaining-requests: 9999
x-ratelimit-remaining-tokens: 1985432
x-ratelimit-reset-requests: 6ms
x-ratelimit-reset-tokens: 432ms

OpenAI provides separate requests and tokens limits. The reset headers indicate when the next unit becomes available. Note that OpenAI’s TPM limits vary by model tier and organization tier — GPT-6 Luna has higher TPM limits than GPT-6 Astra for the same organization level, reflecting the compute cost difference.

Anthropic

anthropic-ratelimit-requests-limit: 4000
anthropic-ratelimit-requests-remaining: 3999
anthropic-ratelimit-requests-reset: 2026-11-03T14:30:00Z
anthropic-ratelimit-tokens-limit: 400000
anthropic-ratelimit-tokens-remaining: 398500
anthropic-ratelimit-tokens-reset: 2026-11-03T14:30:00Z
anthropic-ratelimit-input-tokens-limit: 400000
anthropic-ratelimit-output-tokens-limit: 80000

Anthropic splits input and output token limits explicitly in headers. This is more useful than a combined TPM limit because it lets clients optimize independently — a workload that sends large prompts with short completions hits different constraints than one generating long outputs.

Google Gemini

Google’s Gemini API returns standard x-ratelimit-* headers with both request-count and token-count limits. The limits vary by model — Gemini 4 Argon has different TPM limits than Gemini 3.8 Flash.

Building a client-side limiter from headers

import asyncio
from dataclasses import dataclass
from datetime import datetime, timezone

@dataclass
class ProviderLimits:
    tokens_remaining: int
    tokens_reset: datetime
    requests_remaining: int
    requests_reset: datetime

def parse_openai_headers(headers: dict) -> ProviderLimits:
    return ProviderLimits(
        tokens_remaining=int(headers.get("x-ratelimit-remaining-tokens", 0)),
        tokens_reset=_parse_reset(headers.get("x-ratelimit-reset-tokens", "0ms")),
        requests_remaining=int(headers.get("x-ratelimit-remaining-requests", 0)),
        requests_reset=_parse_reset(headers.get("x-ratelimit-reset-requests", "0ms")),
    )

class HeaderAwareThrottler:
    def __init__(self):
        self.limits: ProviderLimits | None = None
        self._lock = asyncio.Lock()

    async def wait_if_needed(self, estimated_tokens: int):
        async with self._lock:
            if self.limits and self.limits.tokens_remaining < estimated_tokens:
                wait = (self.limits.tokens_reset - datetime.now(timezone.utc)).total_seconds()
                if wait > 0:
                    await asyncio.sleep(min(wait, 60))

    def update_from_response(self, headers: dict):
        self.limits = parse_openai_headers(headers)

Client-side throttling uses provider headers to avoid sending requests that will be rejected.

Architecture: Where Token Limiting Lives

Token-based rate limiting can be implemented at four different layers, each with different visibility and control:

Layer	Visibility	Control	Latency impact
Client SDK	Sees own requests only	Can delay/queue locally	None (pre-send)
API Gateway	Sees all requests for a tenant	Can reject at edge	<1ms per check
LLM Proxy/Router	Sees all requests across providers	Can route around limits	1-5ms
Provider-side	Provider enforces their limits	429 responses	N/A (reaction only)

The most robust architecture implements limiting at both the gateway and proxy layers.

Two-layer limiting: the gateway enforces tenant budgets, the proxy respects provider limits.

The gateway layer handles multi-tenant fairness — ensuring one customer doesn’t consume the entire organization’s TPM budget. The proxy layer handles provider-specific limits — if OpenAI’s remaining TPM is low, route to Anthropic instead. These are complementary, not redundant.

Implementation Patterns

Pattern 1: Redis-Based Distributed Token Bucket

For multi-instance deployments, the token bucket state needs to be shared. Redis is the standard choice.

-- token_bucket.lua (Redis Lua script for atomic token bucket)
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])  -- tokens per second
local requested = tonumber(ARGV[3])
local now = tonumber(ARGV[4])

local bucket = redis.call('hmget', key, 'tokens', 'last_refill')
local tokens = tonumber(bucket[1]) or capacity
local last_refill = tonumber(bucket[2]) or now

-- Refill
local elapsed = math.max(0, now - last_refill)
tokens = math.min(capacity, tokens + elapsed * refill_rate)

if tokens >= requested then
    tokens = tokens - requested
    redis.call('hmset', key, 'tokens', tokens, 'last_refill', now)
    redis.call('expire', key, 120)
    return {1, tokens}  -- allowed, remaining
else
    redis.call('hmset', key, 'tokens', tokens, 'last_refill', now)
    redis.call('expire', key, 120)
    local wait_ms = math.ceil((requested - tokens) / refill_rate * 1000)
    return {0, wait_ms}  -- denied, retry-after in ms
end

import redis
import time

class DistributedTokenLimiter:
    def __init__(self, redis_client: redis.Redis, tpm_limit: int):
        self.redis = redis_client
        self.capacity = int(tpm_limit * 1.5)  # burst allowance
        self.refill_rate = tpm_limit / 60.0
        self._script = self.redis.register_script(open("token_bucket.lua").read())

    def try_consume(self, tenant_id: str, tokens: int) -> tuple[bool, int]:
        result = self._script(
            keys=[f"tpm:{tenant_id}"],
            args=[self.capacity, self.refill_rate, tokens, time.time()],
        )
        allowed = bool(result[0])
        remaining_or_wait = int(result[1])
        return allowed, remaining_or_wait

The Lua script executes atomically on Redis — no race conditions between checking and decrementing across instances. The expire call ensures dead keys get cleaned up.

Pattern 2: Request Queue with Token-Aware Scheduling

Instead of rejecting requests that exceed the budget, queue them and dispatch when tokens become available. This is better for batch workloads where latency tolerance is higher.

import asyncio
import heapq
from dataclasses import dataclass, field

@dataclass(order=True)
class QueuedRequest:
    priority: int
    estimated_tokens: int = field(compare=False)
    future: asyncio.Future = field(compare=False)

class TokenAwareQueue:
    def __init__(self, tpm_limit: int):
        self.bucket = TokenBucketLimiter(tpm_limit)
        self.queue: list[QueuedRequest] = []
        self._drain_task: asyncio.Task | None = None

    async def submit(self, estimated_tokens: int, priority: int = 0) -> None:
        """Block until tokens are available."""
        allowed, wait = self.bucket.try_consume(estimated_tokens)
        if allowed:
            return

        future = asyncio.get_event_loop().create_future()
        heapq.heappush(
            self.queue,
            QueuedRequest(priority, estimated_tokens, future),
        )
        if not self._drain_task:
            self._drain_task = asyncio.create_task(self._drain_loop())
        await future

    async def _drain_loop(self):
        while self.queue:
            item = self.queue[0]
            allowed, wait_seconds = self.bucket.try_consume(item.estimated_tokens)
            if allowed:
                heapq.heappop(self.queue)
                item.future.set_result(None)
            else:
                await asyncio.sleep(min(wait_seconds, 1.0))
        self._drain_task = None

The priority queue ensures high-priority requests (e.g., real-time chat) drain before low-priority ones (e.g., batch embedding jobs) even when both are waiting for token budget.

Pattern 3: Nginx/OpenResty Token Limiting at the Edge

For teams using Nginx or OpenResty as their API gateway, token-aware limiting can be implemented in Lua at the edge:

# nginx.conf snippet
lua_shared_dict token_limits 10m;

server {
    location /v1/chat/completions {
        access_by_lua_block {
            local limiter = require("token_limiter")
            local body = ngx.req.get_body_data()
            local estimated = limiter.estimate_tokens(body)
            local key = ngx.var.http_authorization

            local ok, remaining = limiter.consume(key, estimated)
            if not ok then
                ngx.header["Retry-After"] = tostring(remaining / 1000)
                ngx.status = 429
                ngx.say('{"error":"token rate limit exceeded"}')
                return ngx.exit(429)
            end
        }
        proxy_pass http://llm_backend;
    }
}

This requires a fast token estimation function — running a full tokenizer in the Nginx request path adds latency. The len(body) / 3.5 heuristic is acceptable here; accuracy matters less than speed at the admission point.

Adaptive Throttling and Backpressure

Static TPM limits don’t account for provider-side conditions. The provider might be experiencing load, returning elevated latencies, or throttling in ways that don’t manifest as 429s.

Adaptive throttling adjusts local limits based on observed conditions:

class AdaptiveTokenLimiter:
    def __init__(self, base_tpm: int):
        self.base_tpm = base_tpm
        self.current_tpm = base_tpm
        self.consecutive_429s = 0
        self.consecutive_successes = 0

    def record_response(self, status_code: int, latency_ms: float):
        if status_code == 429:
            self.consecutive_429s += 1
            self.consecutive_successes = 0
            # Multiplicative decrease
            self.current_tpm = max(
                self.base_tpm * 0.1,  # floor at 10% of base
                self.current_tpm * 0.5,
            )
        elif latency_ms > 30_000:  # response took >30s — provider under stress
            self.current_tpm = max(
                self.base_tpm * 0.1,
                self.current_tpm * 0.8,
            )
        else:
            self.consecutive_successes += 1
            self.consecutive_429s = 0
            # Additive increase
            if self.consecutive_successes > 10:
                self.current_tpm = min(
                    self.base_tpm,
                    self.current_tpm + self.base_tpm * 0.1,
                )

This implements AIMD (additive increase, multiplicative decrease), the same congestion control principle behind TCP. On a 429, cut the limit in half. On sustained success, gradually increase by 10% of the base rate.

AIMD throttling: fast backoff on errors, slow recovery on success.

Backpressure signals beyond 429s

A 429 is the explicit signal, but implicit signals matter too:

Latency spikes: if P99 latency doubles, the provider is probably overloaded. Reduce sending rate preemptively.
Partial responses / timeouts: a request that times out consumed GPU-seconds without producing tokens. Count it against the budget at max_tokens (worst case).
Streaming stalls: if a streaming response stops sending tokens for >10 seconds mid-generation, the provider may be shedding load. Consider canceling and retrying at a lower rate.
Response header changes: a suddenly lower x-ratelimit-remaining-tokens value indicates the provider lowered your limit (some providers adjust limits dynamically based on system load).

Multi-Tenant Token Budgeting

In a platform serving multiple customers through shared LLM API keys, token budgets must be subdivided across tenants. This is where token-based limiting intersects with billing and fairness.

Hierarchical budgets

Hierarchical budgets: fixed allocations per team with a shared overflow pool.

Each team gets a guaranteed minimum TPM allocation. A shared pool absorbs burst demand — if Team A is idle, Team B can borrow from the shared pool. When Team A comes back, it reclaims its guaranteed allocation. This mirrors how bandwidth allocation works in network QoS.

class HierarchicalBudget:
    def __init__(self, total_tpm: int, tenant_configs: dict[str, int]):
        self.shared_pool = TokenBucketLimiter(
            total_tpm - sum(tenant_configs.values())
        )
        self.tenant_buckets = {
            tid: TokenBucketLimiter(tpm)
            for tid, tpm in tenant_configs.items()
        }

    def try_consume(self, tenant_id: str, tokens: int) -> bool:
        bucket = self.tenant_buckets.get(tenant_id)
        if not bucket:
            return False

        # Try tenant's own budget first
        allowed, _ = bucket.try_consume(tokens)
        if allowed:
            return True

        # Fall back to shared pool
        allowed, _ = self.shared_pool.try_consume(tokens)
        return allowed

Per-tenant cost tracking

Token budgets and cost budgets are related but not identical. Different models have different per-token costs, and input vs output tokens are priced differently. A tenant’s TPM budget limits throughput; a separate cost budget limits spend.

Budget type	Unit	Controls	Example
TPM	tokens/minute	Throughput, fairness	500K TPM for Team A
Daily cost	dollars/day	Spend control	$50/day per team
Monthly cost	dollars/month	Billing cap	$5,000/month per tenant

Implement these as separate limiters that both must pass. A request is allowed only if it passes both the TPM check and the cost-budget check.

Failure Modes and Edge Cases

Streaming responses and token accounting

With streaming (SSE), tokens arrive incrementally. The limiter reserved an estimated output budget at admission time, but the actual output unfolds over seconds or minutes. Two problems:

Over-reservation: if the model generates fewer tokens than reserved, the budget is unnecessarily constrained until the stream completes and the refund is issued.
Under-reservation: if the model generates more tokens than estimated (possible when max_tokens is set high and the historical estimate was low), the budget goes negative.

The practical solution: reserve at admission, but run a background task that periodically reconciles reservations against actual streaming progress. If a stream has emitted 5,000 tokens against a 50,000-token reservation and is 80% through (based on elapsed time relative to typical generation speed), release some of the excess reservation early.

Tokenizer divergence

The tokenizer you use for admission control might disagree with the provider’s tokenizer. If your estimate says 10,000 tokens but the provider bills 13,000 (because you’re using the wrong tokenizer or the provider’s tokenizer changed), your budget tracking drifts. Over time, this drift accumulates.

Mitigation: on every completed request, compare your estimate against the usage field in the provider’s response. Track the error ratio and apply a correction factor:

class CalibratedEstimator:
    def __init__(self):
        self.error_ratios: list[float] = []

    def record(self, estimated: int, actual: int):
        if estimated > 0:
            self.error_ratios.append(actual / estimated)
            if len(self.error_ratios) > 1000:
                self.error_ratios = self.error_ratios[-1000:]

    @property
    def correction_factor(self) -> float:
        if not self.error_ratios:
            return 1.0
        return sum(self.error_ratios) / len(self.error_ratios)

    def estimate(self, raw_count: int) -> int:
        return int(raw_count * self.correction_factor)

Prompt caching and token limits

Prompt caching (supported by OpenAI, Anthropic, and Google) changes the economics but not the rate limit accounting. Cached input tokens are cheaper ($0.10/M cached vs $2/M input for Claude Sonnet 5.5) but they still count against TPM limits at most providers. A common misconception is that cached tokens are “free” for rate limiting purposes — they’re cheaper for billing but consume the same TPM budget.

Long-running requests and budget starvation

A Claude Fable 5.1 request with 1M context and 128K max output could run for minutes. During that time, the reserved token budget is locked. If the bucket capacity is close to the request size, all other requests for that tenant are blocked until the large request completes.

Solutions:

Set the bucket capacity (burst multiplier) high enough to accommodate at least 2-3 concurrent max-size requests.
Implement request-level timeouts and cancel requests that exceed a latency budget, freeing their reservation.
Route large requests to a separate limiter pool so they don’t starve small interactive requests.

Separate pools prevent large requests from starving interactive traffic.

The retry amplification trap

When a 429 is returned, naive clients retry immediately with exponential backoff. But if the retry includes the same large prompt, it consumes the same token budget on admission — even if the request is ultimately rejected again. A client retrying a 500K-token request 5 times before it succeeds has “consumed” 2.5M tokens of admission checks even though only one request actually ran.

The fix: don’t count rejected requests against the budget. Only deduct tokens on admission success. This seems obvious, but some gateway implementations deduct-then-check rather than check-then-deduct.

Rate limit arbitrage across providers

With an LLM gateway that routes across providers, a sophisticated limiter can exploit the fact that different providers have different remaining capacity. If OpenAI’s TPM budget is nearly exhausted but Anthropic’s is full, route the next request to Anthropic — assuming the models are comparable for the use case.

class MultiProviderRouter:
    def __init__(self, limiters: dict[str, DistributedTokenLimiter]):
        self.limiters = limiters
        self.preference_order = ["openai", "anthropic", "google"]

    def route(self, estimated_tokens: int) -> str | None:
        for provider in self.preference_order:
            limiter = self.limiters[provider]
            allowed, _ = limiter.try_consume("shared", estimated_tokens)
            if allowed:
                return provider
        return None  # all providers exhausted

This is the intersection of token-based rate limiting and LLM gateway architecture — the limiter becomes a routing signal, not just an admission control mechanism.

Summary

Request-count limits are the wrong abstraction for LLM workloads. A 200-token chat message and a 500K-token document analysis request are not the same “1 request.”
Token bucket algorithms with reservation-refund cycles handle the core challenge of unknown output token counts. Reserve pessimistically at admission, refund the surplus on completion.
Every major provider returns token-aware rate limit headers. Client-side throttlers should read these headers and adjust sending rate before hitting 429s.
Distributed implementations need atomic operations — Redis Lua scripts are the standard pattern for shared token bucket state across instances.
Adaptive throttling (AIMD) responds to implicit signals (latency spikes, timeouts) not just explicit 429s.
Multi-tenant systems need hierarchical budgets: guaranteed per-tenant allocations with a shared overflow pool, plus separate cost budgets for spend control.
Separate large and small request pools prevent budget starvation from long-running max-context requests.
Calibrate tokenizer estimates against actual usage fields to prevent drift.