How token-aware rate limiting works for LLM APIs: algorithms, implementation patterns, sliding windows, and production architectures across API gateways.

Token-Based Rate Limiting for LLM APIs

Request-count rate limiting — the kind that’s protected REST APIs for decades — breaks down when applied to LLM workloads. A single API call can consume anywhere from 50 tokens to 500,000 tokens, meaning one request might cost 10,000x more compute than another. Treating them equally is like charging the same toll for a bicycle and a freight train.

Token-based rate limiting is the response: throttling based on the actual computational cost of each request, measured in tokens consumed rather than requests made. Every major LLM provider now enforces token-level limits, and any production system sitting in front of these APIs needs to understand and implement them.

Why Request-Count Limits Fail for LLM Workloads
The Token Accounting Problem
Rate Limiting Algorithms Adapted for Tokens
Provider Rate Limit Models Compared
Pre-Request Token Estimation
Sliding Window Token Counters
Implementation Patterns
Gateway-Level Token Rate Limiting
Handling Streaming Responses
Multi-Tenant Token Budgets
Failure Modes and Edge Cases
Summary
Further Reading

Why Request-Count Limits Fail

A standard API rate limiter counts requests per time window. 100 requests per minute, 1,000 per hour. This works when requests have roughly uniform cost — a GET to /users/123 and a GET to /users/456 consume similar server resources.

LLM API calls have wildly non-uniform cost. The compute required scales with both input token count (prefill) and output token count (decode). A 200-token prompt generating a 50-token response might take 100ms of GPU time. A 100,000-token prompt generating 4,000 tokens of output might take 30 seconds. Both count as one request under traditional rate limiting.

Request-count limiting treats a 250-token call and a 104K-token call identically, despite a 400x compute difference.

The consequences are concrete:

Unfair resource allocation. A tenant sending 10 requests with 100K-token contexts consumes more GPU time than a tenant sending 1,000 requests with 200-token contexts, but the first tenant uses only 1% of a 1,000 RPM limit.
Capacity planning failures. A sudden shift from short to long requests can overwhelm GPU memory (KV cache scales with sequence length) without triggering any rate limit.
Cost model mismatch. LLM providers charge per token, not per request. A rate limiter that doesn’t track tokens can’t enforce cost budgets.
Burst vulnerability. Ten concurrent requests with 200K-token contexts can exhaust GPU memory even if the request rate is well under the RPM cap.

The Token Accounting Problem

Token-based rate limiting requires knowing how many tokens a request will consume before the response completes. This is harder than it sounds, because LLM requests have two components with different observability characteristics.

Input tokens are knowable before the request reaches the model. The prompt text can be tokenized client-side or at the gateway. The count is deterministic for a given tokenizer.

Output tokens are unknowable in advance. The model generates tokens autoregressively until it hits a stop condition — an EOS token, a max_tokens limit, or a stop sequence. A request with max_tokens: 4096 might generate 12 tokens or 4,096 tokens.

This creates a fundamental design tension: do you rate-limit based on what you know (input tokens), what you budget for (max possible output), or what actually happened (post-response accounting)?

The three phases of token accounting: pre-request reservation, in-flight streaming, and post-response reconciliation.

Most production systems use a reserve-and-reconcile approach:

Reserve input tokens + max_tokens against the budget before sending the request.
Reconcile after the response completes by releasing the difference between reserved and actual output tokens.

This is conservative — it temporarily over-counts — but it prevents budget overruns. The alternative (only counting after completion) allows bursts that exceed the actual limit.

Rate Limiting Algorithms Adapted for Tokens

The classic rate limiting algorithms — fixed window, sliding window, token bucket, leaky bucket — all need modification when the unit of measurement shifts from “1 request” to “N tokens.”

Fixed Window Token Counter

The simplest approach: maintain a counter of tokens consumed in the current time window (e.g., 1 minute). Each request adds its token count. Reject when the counter exceeds the limit.

import time
from dataclasses import dataclass

@dataclass
class FixedWindowTokenLimiter:
    max_tokens_per_window: int
    window_seconds: int = 60
    _current_window: int = 0
    _tokens_used: int = 0

    def try_consume(self, token_count: int) -> bool:
        now = int(time.time())
        window = now // self.window_seconds
        if window != self._current_window:
            self._current_window = window
            self._tokens_used = 0
        if self._tokens_used + token_count > self.max_tokens_per_window:
            return False
        self._tokens_used += token_count
        return True

The problem is the same as with request-count fixed windows: boundary bursts. A client can consume max_tokens at the end of window N and again at the start of window N+1, effectively doubling throughput over a short period.

Sliding Window Log

Track every token consumption event with a timestamp. To check the limit, sum all tokens consumed in the trailing window. More accurate but memory-intensive — each event must be stored until it ages out.

For high-throughput systems, this is impractical without bucketing. A common compromise: use sub-windows (e.g., 1-second buckets within a 60-second window) and sum the buckets.

Token Bucket (Adapted)

The classic token bucket adds tokens at a steady rate and removes them on each request. For LLM rate limiting, each API call removes a variable number of tokens from the bucket rather than a fixed count.

import time

class TokenBucketLimiter:
    def __init__(self, capacity: int, refill_rate: float):
        """
        capacity: max tokens in bucket
        refill_rate: tokens added per second
        """
        self.capacity = capacity
        self.tokens = capacity
        self.refill_rate = refill_rate
        self.last_refill = time.monotonic()

    def try_consume(self, count: int) -> bool:
        now = time.monotonic()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now
        if self.tokens >= count:
            self.tokens -= count
            return True
        return False

This naturally handles bursts (the bucket can accumulate tokens during quiet periods) while enforcing a sustained rate. The capacity parameter controls maximum burst size — important because a single LLM request can consume a large fraction of the budget.

A token bucket drains variable amounts per request, with the bucket capacity controlling burst tolerance.

Dual-Dimension Limiting

In practice, most production systems enforce both request-count and token-count limits simultaneously. A request must pass both checks. This prevents edge cases where many tiny requests overwhelm request-processing overhead, or where a few enormous requests exhaust GPU memory.

Algorithm	Burst Handling	Memory	Accuracy	Complexity
Fixed window	Poor (boundary bursts)	O(1)	Low	Trivial
Sliding window log	Good	O(n) per client	High	Moderate
Sliding window counter	Good	O(k) for k sub-windows	Medium-high	Moderate
Token bucket	Good (configurable)	O(1)	High	Low
Dual-dimension	Good	2× base algorithm	High	Moderate

Provider Rate Limit Models Compared

Every major LLM provider enforces token-level rate limits, but the specifics vary. These limits are typically expressed as tokens per minute (TPM) and vary by model tier and pricing plan.

Provider	Rate Limit Dimensions	Typical TPM (Flagship)	Separate Input/Output Limits	Retry Header
OpenAI	RPM + TPM + RPD	800K–30M (varies by tier)	No (combined)	`x-ratelimit-remaining-tokens`
Anthropic	RPM + input TPM + output TPM	Varies by model/tier	Yes (separate)	`anthropic-ratelimit-tokens-remaining`
Google (Gemini)	RPM + TPM	Varies by model	Combined	Standard HTTP 429
Amazon Bedrock	TPM per model (provisioned or on-demand)	Varies by provisioned throughput	Yes	`x-amzn-bedrock-*` headers

Key differences worth noting:

Anthropic separates input and output token limits. This is more precise because output tokens are more expensive computationally (each requires a full forward pass, while input tokens are processed in parallel during prefill). A system that only tracks combined tokens can’t distinguish between a 100K-input / 100-output request and a 100-input / 100K-output request, even though the latter uses far more GPU time.
OpenAI returns detailed rate limit headers including x-ratelimit-limit-tokens, x-ratelimit-remaining-tokens, and x-ratelimit-reset-tokens, making client-side tracking straightforward.
Bedrock uses provisioned throughput as the primary mechanism, where you pay for a fixed token-per-minute allocation per model. This is closer to reserved capacity than dynamic rate limiting.

Pre-Request Token Estimation

To rate-limit on tokens before sending a request to the LLM provider, the gateway needs to estimate token count from the raw prompt text. There are three approaches, each with different accuracy-latency tradeoffs.

Exact Tokenization

Run the provider’s actual tokenizer on the input text. OpenAI publishes tiktoken; many open models use SentencePiece or the Hugging Face tokenizers library. This gives an exact count but requires maintaining tokenizer binaries and keeping them synchronized with model updates.

import tiktoken

# For OpenAI models
enc = tiktoken.encoding_for_model("gpt-4.1-nano")
token_count = len(enc.encode(prompt_text))

Tokenization is fast — typically microseconds for short prompts, low milliseconds for long ones. The overhead is negligible compared to inference latency. The real cost is operational: you need the correct tokenizer for each model, and tokenizer changes between model versions can shift counts.

Note that Claude Sonnet 5’s new tokenizer emits ~30% more tokens for the same text compared to earlier Claude models. A gateway serving multiple providers needs per-model tokenizer logic, or at least per-model correction factors.

Character-Based Estimation

A rough heuristic: 1 token ≈ 4 characters for English text (or equivalently, ~0.75 words per token). This is fast and requires no tokenizer binary, but accuracy varies by language, code content, and whitespace patterns.

def estimate_tokens_chars(text: str) -> int:
    return len(text) // 4  # rough, English-centric

For rate limiting purposes, a ±15% error margin is often acceptable — you’re enforcing a budget, not billing. Overestimation is preferable to underestimation for safety.

Hybrid: Estimate Then Reconcile

Use character-based estimation for the pre-request check (fast path), then reconcile with the actual token count from the provider’s response headers (accurate post-hoc adjustment). This keeps latency low while maintaining accurate long-term accounting.

Hybrid estimation: fast approximation for gating, exact reconciliation from response metadata.

Sliding Window Token Counters

For distributed systems handling thousands of concurrent requests, the rate limiter state must be shared across gateway instances. Redis is the standard backing store.

Redis Sliding Window Implementation

A sliding window counter using Redis sorted sets, where each entry is a token consumption event scored by timestamp:

import time
import redis

class RedisTokenRateLimiter:
    def __init__(self, client: redis.Redis, window_seconds: int = 60):
        self.r = client
        self.window = window_seconds

    def try_consume(self, key: str, tokens: int, limit: int) -> bool:
        now = time.time()
        window_start = now - self.window
        pipe = self.r.pipeline()
        # Remove expired entries
        pipe.zremrangebyscore(key, 0, window_start)
        # Get current token sum
        pipe.zrangebyscore(key, window_start, now, withscores=True)
        results = pipe.execute()
        entries = results[1]
        current_total = sum(score for _, score in entries)

        if current_total + tokens > limit:
            return False

        # Add new consumption — member must be unique
        member = f"{now}:{id(object())}"
        pipe2 = self.r.pipeline()
        pipe2.zadd(key, {member: now})
        # Store token count in a parallel hash
        pipe2.hset(f"{key}:counts", member, tokens)
        pipe2.expire(key, self.window + 10)
        pipe2.expire(f"{key}:counts", self.window + 10)
        pipe2.execute()
        return True

This approach has a problem: the sorted set stores timestamps as scores but doesn’t encode the token count per entry directly. A cleaner alternative uses a Lua script to atomically check and update:

-- KEYS[1]: sorted set key
-- ARGV[1]: window start timestamp
-- ARGV[2]: current timestamp  
-- ARGV[3]: token count for this request
-- ARGV[4]: token limit
-- ARGV[5]: unique member ID

redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, ARGV[1])

local members = redis.call('ZRANGEBYSCORE', KEYS[1], ARGV[1], ARGV[2])
local total = 0
for _, m in ipairs(members) do
    local count = tonumber(string.match(m, ':(%d+)$'))
    total = total + (count or 0)
end

if total + tonumber(ARGV[3]) > tonumber(ARGV[4]) then
    return 0
end

-- Encode token count in the member name
local member = ARGV[2] .. ':' .. ARGV[5] .. ':' .. ARGV[3]
redis.call('ZADD', KEYS[1], ARGV[2], member)
redis.call('EXPIRE', KEYS[1], tonumber(ARGV[2]) - tonumber(ARGV[1]) + 10)
return 1

Encoding the token count in the sorted set member name avoids a secondary data structure. The Lua script runs atomically in Redis, preventing race conditions between check and update.

Multiple gateway instances share rate limit state through Redis, using atomic Lua scripts to prevent races.

Memory Considerations

Each sorted set entry for a sliding window consumes roughly 80-100 bytes in Redis. At 1,000 requests per minute per client with 100 clients, that’s 100K entries × 100 bytes = 10MB — manageable. But if you’re tracking per-user limits across thousands of users, memory can add up. Setting TTLs on the sorted set keys (slightly longer than the window) ensures automatic cleanup.

Implementation Patterns

Pattern 1: Gateway-Side Pre-Check with Provider Passthrough

The gateway estimates tokens, checks the budget, and forwards to the provider. The provider’s own rate limits serve as a second layer. After the response, the gateway reconciles actual usage.

Gateway-side pre-check: the gateway acts as the first line of defense, with the provider’s limits as backstop.

This is the most common pattern. The gateway can enforce per-tenant budgets, per-model limits, and cost caps that the provider knows nothing about.

Pattern 2: Response-Header Feedback Loop

Instead of estimating tokens locally, the gateway relies on rate limit headers from the provider’s response to adjust its own state. This avoids tokenizer maintenance but only works reactively — it can’t prevent the first request that exceeds the limit.

def update_from_headers(headers: dict, limiter: RedisTokenRateLimiter, key: str):
    remaining = int(headers.get("x-ratelimit-remaining-tokens", 0))
    limit = int(headers.get("x-ratelimit-limit-tokens", 0))
    used = limit - remaining
    # Sync local state with provider's view
    limiter.sync_used(key, used)

This works well as a correction mechanism layered on top of local estimation, catching drift between estimated and actual token counts.

Pattern 3: Budget Pools with Hierarchical Limits

For multi-tenant platforms, token budgets often have a hierarchy: organization → project → user. Each level has its own limit, and a request must pass all levels.

Hierarchical token budgets: each request must pass organization, project, and user-level checks.

Implementation requires atomic multi-key operations. A Lua script can check and decrement all three levels in a single Redis round-trip, rolling back if any level rejects:

-- Check all three levels atomically
local levels = {KEYS[1], KEYS[2], KEYS[3]}
local tokens = tonumber(ARGV[1])

for i, key in ipairs(levels) do
    local remaining = tonumber(redis.call('GET', key) or '0')
    if remaining < tokens then
        return -i  -- return which level rejected
    end
end

-- All passed — decrement all
for _, key in ipairs(levels) do
    redis.call('DECRBY', key, tokens)
end
return 1

Gateway-Level Token Rate Limiting

Several API gateway and proxy solutions have added token-aware rate limiting, either natively or through plugins.

Kong / Kong Gateway

Kong’s rate-limiting plugin supports custom cost functions. A plugin can inspect the request body, estimate token count, and pass a weight to the rate limiter:

-- Kong plugin: token-weighted rate limiting
local body = kong.request.get_body()
local prompt = body and body.messages or ""
local estimated_tokens = math.ceil(#cjson.encode(prompt) / 4)
local max_tokens = body and body.max_tokens or 1024
local total_weight = estimated_tokens + max_tokens

kong.ctx.shared.rate_limit_weight = total_weight

Envoy / Istio

Envoy’s rate limit service supports descriptors, which can include custom values extracted by Lua or Wasm filters. A Wasm filter can parse the request body, estimate tokens, and set a descriptor value that the rate limit service uses as the cost.

Cloudflare AI Gateway

Cloudflare’s AI Gateway product provides token-based rate limiting as a built-in feature, tracking both input and output tokens across supported providers. It sits between the client and the LLM provider, handles provider-specific auth, and exposes a unified analytics dashboard showing token consumption by model and endpoint.

Custom Middleware (Node.js Example)

For applications that proxy LLM calls through their own backend:

import { Ratelimit } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";
import { encoding_for_model } from "tiktoken";

const tokenLimiter = new Ratelimit({
  redis: Redis.fromEnv(),
  limiter: Ratelimit.slidingWindow(100_000, "1 m"), // 100K tokens/min
});

async function rateLimitMiddleware(req, res, next) {
  const enc = encoding_for_model("gpt-4.1-nano");
  const inputTokens = enc.encode(JSON.stringify(req.body.messages)).length;
  const maxOutput = req.body.max_tokens ?? 1024;
  const reserveTokens = inputTokens + maxOutput;

  const { success, remaining, reset } = await tokenLimiter.limit(
    `user:${req.userId}`,
    { rate: reserveTokens }  // consume variable amount
  );

  if (!success) {
    res.status(429).json({
      error: "Token rate limit exceeded",
      remaining_tokens: remaining,
      reset_at: reset,
    });
    return;
  }

  next();
}

Note: Upstash’s Ratelimit supports variable cost via the rate parameter in the limit call, making it suitable for token-based limiting without custom sorted-set logic.

Handling Streaming Responses

Streaming responses (SSE) complicate token accounting. The full output token count isn’t known until the stream ends, but the budget reservation was made at the start. Several patterns address this.

Reserve Max, Reconcile on Stream End

Reserve max_tokens worth of output budget before the request starts. When the stream completes, read the final usage object (most providers include it in the last SSE event or a separate [DONE] message) and release the difference.

Reserve-and-reconcile for streaming: over-reserve at start, release surplus when the stream completes.

The downside: during streaming, the reserved tokens reduce the available budget for other requests. For long-running streams (common with large max_tokens values), this can cause unnecessary rejections. A mitigation is to use a “soft” reservation that allows slight over-subscription, with a hard check only at reconciliation.

Progressive Counting

Count tokens as they stream through the gateway. This requires the gateway to tokenize each SSE chunk, which adds per-chunk overhead. For most tokenizers, this is sub-millisecond per chunk, but it adds complexity.

class StreamingTokenCounter:
    def __init__(self):
        self.output_tokens = 0

    def on_chunk(self, chunk: str):
        # Approximate: SSE chunks are usually 1-3 tokens
        # Most providers include token deltas in chunk metadata
        if hasattr(chunk, 'usage') and chunk.usage:
            self.output_tokens = chunk.usage.output_tokens
        else:
            # Fallback: estimate from text
            self.output_tokens += max(1, len(chunk) // 4)

OpenAI’s streaming API includes a usage field in the final chunk when stream_options: {"include_usage": true} is set. Anthropic’s message_delta event includes output_tokens in the usage block. Relying on provider-reported counts is more accurate than local tokenization of chunks.

Connection Drops and Partial Streams

If the client disconnects mid-stream, the gateway may never receive the final usage event. In this case, the reserved tokens remain consumed until the window expires. For long windows (hourly or daily budgets), this creates a phantom usage problem.

Solutions:

Track in-flight streams and time them out after a maximum duration.
On disconnect, estimate actual usage from chunks received so far and reconcile immediately.
Use the provider’s usage API (if available) to query actual consumption after the fact.

Multi-Tenant Token Budgets

Production LLM platforms serve multiple tenants — internal teams, API customers, or application components — each with different token budgets. The design challenge is isolation without waste.

Fixed Allocation

Each tenant gets a fixed TPM allocation. Simple, predictable, but wasteful when tenants are idle. A tenant with a 100K TPM limit that only uses 10K leaves 90K unusable by anyone else.

Shared Pool with Guaranteed Minimums

Each tenant has a guaranteed minimum allocation, with the remainder in a shared pool. Tenants can burst above their minimum if the shared pool has capacity. This is how network QoS (weighted fair queuing) works, adapted for tokens.

Shared pool with guaranteed minimums: tenants can burst into shared capacity when available.

Cost-Based Budgets

Instead of token counts, allocate dollar budgets. This accounts for the fact that different models have different per-token costs. A request to GPT-6 Astra ($10/$50 per M) costs roughly 100x more than the same token count on GPT-4.1 Nano ($0.10/$0.40 per M).

MODEL_COSTS = {
    "gpt-6-astra": {"input": 10.00, "output": 50.00},  # per million tokens
    "gpt-4.1-nano": {"input": 0.10, "output": 0.40},
    "claude-sonnet-5": {"input": 2.00, "output": 10.00},
    "gemini-3.8-flash": {"input": 0.75, "output": 3.75},  # introductory
}

def estimate_cost(model: str, input_tokens: int, max_output_tokens: int) -> float:
    costs = MODEL_COSTS.get(model, {"input": 5.0, "output": 15.0})
    input_cost = (input_tokens / 1_000_000) * costs["input"]
    output_cost = (max_output_tokens / 1_000_000) * costs["output"]
    return input_cost + output_cost

Dollar-based budgets are more intuitive for business stakeholders and naturally handle model routing decisions. A $100/day budget means the same thing regardless of which model mix is used.

Failure Modes and Edge Cases

The Max Tokens Trap

If a client sends max_tokens: 128000 but the model typically generates 200 tokens for this type of request, the reserve-and-reconcile approach wastes 99.8% of the reserved budget during the request. Mitigations:

Cap max_tokens at the gateway. Override unreasonably high values with a per-model or per-tenant default.
Use historical p95 output length for the client/model combination as the reservation amount, rather than max_tokens.
Allow over-subscription of the budget pool by a configurable factor (e.g., 2x), accepting that actual usage will probably be far below reservations.

Tokenizer Version Drift

If the gateway uses tiktoken version X but the provider has quietly updated to version X+1 with a different vocabulary, input token estimates will be wrong. This is uncommon but has happened — OpenAI changed the tokenizer between GPT-3.5 and GPT-4, and again for the o-series models. The impact on rate limiting is usually small (a few percent drift), but it can compound for cost tracking.

Mitigation: periodically validate gateway estimates against provider-reported usage and alert on sustained divergence above a threshold.

Prompt Caching and Double Counting

OpenAI and Anthropic both support prompt caching, where repeated prefixes are processed from cache rather than re-computed. Cached tokens consume less compute and cost less (e.g., GPT-4.1 Nano cached reads are $0.025/M vs $0.10/M for uncached). A token rate limiter that doesn’t account for caching will over-count resource usage.

Provider response headers typically include both total and cached token counts. The reconciliation step should use uncached tokens for compute-based limits, or apply a discount factor to cached tokens.

Reasoning Tokens

Models with extended thinking (Claude’s adaptive thinking, OpenAI’s thinking models, Gemini’s tunable thinking) generate “reasoning tokens” that count toward output but may not appear in the response. These tokens consume GPU compute and are billed, but clients may not realize they exist when setting max_tokens.

For example, a request to a thinking model might generate 5,000 reasoning tokens internally plus 500 visible output tokens, but the client set max_tokens: 1000 expecting at most 1,000 output tokens. The total billed output is 5,500 tokens.

Rate limiters must include reasoning tokens in the output count. Most providers report them separately in usage metadata (thinking_tokens, reasoning_tokens, or similar), and this number should be added to the output token count for budget purposes.

Reasoning tokens are invisible to the user but billed as output — rate limiters must account for them.

Race Conditions in Distributed Systems

Two gateway instances checking the same client’s budget simultaneously can both see “enough remaining” and both allow their requests, causing an overrun. The Lua script approach (atomic check-and-decrement in Redis) handles this for single-Redis deployments. For Redis Cluster or multi-region setups, eventual consistency means brief overruns are possible.

Accept that distributed rate limiting is best-effort with a small margin of error. Set limits slightly below the hard ceiling to absorb races.

429 Backoff and Provider Limits

When the LLM provider returns HTTP 429, the response typically includes Retry-After or provider-specific headers indicating when to retry. The gateway should:

Propagate the 429 to the client with the retry timing.
Temporarily reduce the local rate limit to avoid hammering the provider.
Not count the rejected request’s tokens against the client’s budget (no tokens were consumed).

A common mistake is retrying 429s immediately with exponential backoff at the gateway level. This can cause thundering herds when many clients hit the provider’s limit simultaneously and all back off to similar retry times. Jitter is essential.

import random

def backoff_with_jitter(attempt: int, base: float = 1.0, max_delay: float = 60.0) -> float:
    exp_delay = min(base * (2 ** attempt), max_delay)
    return random.uniform(0, exp_delay)  # full jitter

Summary

Token-based rate limiting is a necessary adaptation of traditional API rate limiting for LLM workloads, where request cost varies by orders of magnitude. The core components:

Dual-dimension limits (RPM + TPM) are the minimum viable approach. Token-only limits miss request-processing overhead; request-only limits miss compute cost variation.
Reserve-and-reconcile is the most robust pattern for pre-request gating: reserve input + max_output tokens before the call, reconcile with actual usage from response headers.
Token bucket or sliding window counters backed by Redis provide the distributed state management needed for multi-instance gateways. Lua scripts ensure atomicity.
Pre-request estimation using the provider’s tokenizer (tiktoken, SentencePiece) is fast enough for inline use. Character-based heuristics work as a fallback with ~15% error.
Streaming responses require special handling: reserve at stream start, reconcile at stream end, and handle mid-stream disconnects gracefully.
Multi-tenant systems benefit from dollar-based budgets rather than raw token counts, since per-token costs vary 100x across models.
Reasoning tokens, prompt caching, and tokenizer drift are the edge cases that break naive implementations. Account for all three in the reconciliation step.

The gap between “rate limiting that works in development” and “rate limiting that works in production” is mostly about these edge cases — and about accepting that distributed rate limiting is inherently approximate. Build for ±5% accuracy, not exactness.