How LLM gateways work: provider abstraction, fallback routing, load balancing, cost tracking, and credential management across OpenAI, Anthropic, Google, and open-source backends.

LLM Gateway Architecture: Managing Multiple AI Providers Through a Unified API Layer

Every production LLM application starts with one provider and one model. Within six months, it uses three providers, five models, and a routing layer held together with if-else chains and environment variables. The LLM gateway exists to replace that mess with something durable.

An LLM gateway sits between application code and upstream AI providers, presenting a single API surface while handling provider translation, failover, cost tracking, credential management, and traffic shaping. It is the load balancer of the LLM era — except the backends have different schemas, different rate limit semantics, different streaming formats, and different failure modes.

Why a Gateway Instead of Direct Calls
Core Architecture
Provider Abstraction and Schema Translation
Routing Strategies
Fallback and Retry Logic
Load Balancing Across Providers and Keys
Cost Tracking and Token Accounting
Credential Management
Caching at the Gateway Layer
Streaming Translation
Observability and Logging
Open-Source and Commercial Gateways Compared
Building vs Buying
Reference Architecture
Summary
Further Reading

Why a Gateway Instead of Direct Calls

Direct provider SDK calls work fine for prototypes. They break down in production for specific, enumerable reasons:

Provider outages are routine. OpenAI, Anthropic, and Google all experience periodic degradation. A 30-minute outage on a single provider shouldn’t take down a production feature. Without a gateway, adding failover means wrapping every call site in try/catch logic with provider-specific error handling.

Rate limits differ per provider, per model, per key. OpenAI rate limits are per-organization and per-model. Anthropic limits are per-workspace. Google limits are per-project. Distributing load across multiple API keys or accounts requires centralized tracking.

Cost attribution is invisible. When five services call the same OpenAI key, the monthly invoice is a single number with no breakdown by feature, team, or request type. The gateway is the natural place to tag requests and aggregate spend.

Model routing is a product decision. Sending simple classification tasks to GPT-4.1 Nano ($0.10/$0.40 per M tokens) instead of GPT-5.6 Sol ($5/$30 per M tokens) saves 50x on those requests. That routing logic shouldn’t live in application code scattered across repositories.

The gateway presents a single API surface while managing multiple upstream providers.

Core Architecture

An LLM gateway has six core responsibilities, each implemented as a middleware layer or sidecar component:

Schema translation — convert between the gateway’s canonical request/response format and each provider’s API
Routing — decide which provider and model handles each request
Failover — detect errors, apply retry policies, fall back to alternate providers
Rate management — track token and request budgets per key, per provider, per consumer
Cost accounting — count tokens, apply pricing, attribute spend
Credential handling — store, rotate, and scope API keys

These compose as a pipeline. A request enters, gets routed, translated to the target provider’s schema, sent upstream, and the response is translated back into the canonical format while cost and observability data are emitted asynchronously.

Request lifecycle through the gateway pipeline.

Provider Abstraction and Schema Translation

The hardest part of building a gateway is making incompatible APIs look the same. The differences are deeper than field names.

Message Format Differences

OpenAI uses a messages array with role and content fields. Anthropic separates the system prompt from the messages array and requires alternating user/assistant turns. Google’s Gemini API uses contents with parts arrays and a separate systemInstruction field.

A minimal translation layer for a chat completion request:

def to_openai(canonical: dict) -> dict:
    return {
        "model": canonical["model"],
        "messages": [
            {"role": m["role"], "content": m["content"]}
            for m in canonical["messages"]
        ],
        "max_tokens": canonical.get("max_tokens", 4096),
        "temperature": canonical.get("temperature", 1.0),
    }

def to_anthropic(canonical: dict) -> dict:
    system = next(
        (m["content"] for m in canonical["messages"] if m["role"] == "system"),
        None,
    )
    messages = [
        {"role": m["role"], "content": m["content"]}
        for m in canonical["messages"]
        if m["role"] != "system"
    ]
    return {
        "model": canonical["model"],
        "system": system,
        "messages": messages,
        "max_tokens": canonical.get("max_tokens", 4096),
    }

def to_gemini(canonical: dict) -> dict:
    system = next(
        (m["content"] for m in canonical["messages"] if m["role"] == "system"),
        None,
    )
    contents = []
    for m in canonical["messages"]:
        if m["role"] == "system":
            continue
        role = "user" if m["role"] == "user" else "model"
        contents.append({"role": role, "parts": [{"text": m["content"]}]})
    result = {"contents": contents}
    if system:
        result["systemInstruction"] = {"parts": [{"text": system}]}
    return result

Tool/Function Calling Differences

Tool calling schemas diverge more sharply. OpenAI uses tools with function objects. Anthropic uses tools with input_schema. Gemini uses functionDeclarations inside tools. The response format for tool calls also differs — OpenAI returns tool_calls in the assistant message, Anthropic returns tool_use content blocks, and Gemini returns functionCall parts.

Feature	OpenAI	Anthropic	Google Gemini
System prompt	In messages array	Separate `system` field	`systemInstruction` field
Tool definition key	`tools[].function`	`tools[].input_schema`	`tools[].functionDeclarations`
Tool call response	`tool_calls` array	`tool_use` content block	`functionCall` part
Tool result	`role: "tool"` message	`role: "user"` with `tool_result`	`functionResponse` part
Streaming format	`data: {...}` SSE	`event: content_block_delta` SSE	JSON lines or SSE
Max output tokens	`max_tokens` (required in some models)	`max_tokens` (required)	`maxOutputTokens` in `generationConfig`

Structured Output

JSON mode and structured output also vary. OpenAI supports response_format: { type: "json_schema", json_schema: {...} }. Anthropic supports tool-based structured output (define a single tool with the desired schema and force the model to call it). Gemini supports responseMimeType: "application/json" with a responseSchema.

A gateway that claims to support structured output needs to translate these approaches bidirectionally — accepting one canonical format and emitting the right incantation for each provider.

Routing Strategies

Routing is where the gateway earns its keep. Five patterns cover most production needs:

1. Model-Based Routing

The simplest: the caller specifies a canonical model name like fast-cheap or best-quality, and the gateway maps it to a concrete provider model.

routes:
  fast-cheap:
    provider: openai
    model: gpt-4.1-nano
  balanced:
    provider: anthropic
    model: claude-sonnet-5
  best-quality:
    provider: anthropic
    model: claude-opus-5
  code:
    provider: google
    model: gemini-3.7-flash

This decouples application code from provider specifics. When a better model launches, one config change reroutes all traffic.

2. Cost-Based Routing

Route based on task complexity. A classifier (which can itself be a small, cheap model call or a heuristic based on prompt length and content) estimates whether the request needs a frontier model or a budget one.

Cost-based routing sends simple tasks to cheap models and reserves frontier models for complex ones.

A rough heuristic: if the prompt is under 200 tokens and the task is classification or extraction, route to the budget tier. If the prompt involves multi-step reasoning or code generation with context over 4,000 tokens, route to mid or frontier.

3. Latency-Based Routing

Some requests are latency-sensitive (user-facing chat) and others are latency-tolerant (batch processing, background agents). The gateway can maintain running latency percentiles per provider and route accordingly.

4. Compliance-Based Routing

Data residency requirements might dictate that EU user data only goes to providers with EU processing (Google’s EU endpoints, self-hosted models), while US traffic can go anywhere. The gateway inspects request metadata and enforces geographic constraints.

5. A/B and Canary Routing

Send 5% of traffic to a new model to compare quality before full rollout. The gateway assigns requests to variants based on a hash of the user ID or request ID and logs the variant for downstream evaluation.

Fallback and Retry Logic

LLM API failures come in distinct flavors, and each demands a different response:

Error Type	HTTP Status	Correct Response
Rate limit	429	Retry with backoff, or fail over to alternate key/provider
Server error	500, 502, 503	Retry with backoff (max 2-3 attempts)
Context length exceeded	400	Do not retry — reduce context or route to a model with larger context
Invalid request	400, 422	Do not retry — fix the request
Authentication	401, 403	Do not retry — rotate credential
Timeout	—	Retry once, then fail over

A well-configured fallback chain might look like:

fallback_chain:
  primary:
    provider: anthropic
    model: claude-sonnet-5
  fallbacks:
    - provider: openai
      model: gpt-5.5
      conditions: [rate_limit, server_error, timeout]
    - provider: google
      model: gemini-3.7-flash
      conditions: [rate_limit, server_error, timeout]
  max_retries_per_provider: 2
  backoff: exponential
  base_delay_ms: 500

Fallback chain with provider-level failover. Each provider gets retry attempts before moving to the next.

Circuit Breakers

A provider returning 500s on every request shouldn’t be hammered with retries. Circuit breaker logic tracks error rates per provider over a sliding window (e.g., 10 errors in 60 seconds) and temporarily removes that provider from the routing pool. After a cooldown period, the circuit half-opens — a single request is sent to test recovery, and the provider is restored if it succeeds.

class CircuitBreaker:
    def __init__(self, threshold: int = 5, window_s: int = 60, cooldown_s: int = 30):
        self.threshold = threshold
        self.window_s = window_s
        self.cooldown_s = cooldown_s
        self.failures: list[float] = []
        self.state = "closed"  # closed = healthy, open = tripped
        self.opened_at: float | None = None

    def record_failure(self):
        now = time.time()
        self.failures = [t for t in self.failures if now - t < self.window_s]
        self.failures.append(now)
        if len(self.failures) >= self.threshold:
            self.state = "open"
            self.opened_at = now

    def is_available(self) -> bool:
        if self.state == "closed":
            return True
        elapsed = time.time() - self.opened_at
        if elapsed > self.cooldown_s:
            self.state = "half-open"
            return True  # allow one probe request
        return False

    def record_success(self):
        if self.state == "half-open":
            self.state = "closed"
            self.failures.clear()

Timeout Calibration

LLM API calls are slow compared to typical HTTP backends. A chat completion generating 2,000 tokens at 50 tokens/second takes 40 seconds. Setting a 10-second timeout will trigger false failures. Gateway timeouts should be calibrated per model class:

Model Tier	Typical TTFT	Typical Full Response	Suggested Timeout
Budget (Nano, Haiku)	200-500ms	5-15s	30s
Mid (Sonnet, Flash)	300-800ms	10-30s	60s
Frontier (Opus, Sol)	500-2000ms	15-60s	120s
Reasoning/thinking	1-5s	30-180s	300s

Load Balancing Across Providers and Keys

LLM rate limits are typically per-API-key or per-organization, measured in both requests per minute (RPM) and tokens per minute (TPM). A gateway managing multiple keys for the same provider can distribute load across them.

Token-Aware Load Balancing

Request-count load balancing (round robin) doesn’t work well for LLMs because request sizes vary wildly. A 100-token classification request and a 50,000-token document analysis both count as one request but consume vastly different rate limit budget.

The gateway needs to estimate token count before routing. A fast tokenizer (tiktoken for OpenAI models, or a rough heuristic of 1 token ≈ 4 characters for English) provides sufficient accuracy for load balancing decisions.

import tiktoken

def estimate_tokens(text: str, model: str = "gpt-4") -> int:
    try:
        enc = tiktoken.encoding_for_model(model)
        return len(enc.encode(text))
    except KeyError:
        # Fallback: ~4 chars per token for English
        return len(text) // 4

class TokenAwareBalancer:
    def __init__(self, keys: list[dict]):
        # keys: [{"key": "sk-...", "tpm_limit": 800000, "used_tpm": 0}]
        self.keys = keys
    
    def select_key(self, estimated_tokens: int) -> dict | None:
        available = [
            k for k in self.keys
            if k["used_tpm"] + estimated_tokens < k["tpm_limit"]
        ]
        if not available:
            return None
        # Pick the key with the most remaining capacity
        return min(available, key=lambda k: k["used_tpm"])

Multi-Region Distribution

For providers offering regional endpoints (Google Cloud has regional Vertex AI endpoints, AWS Bedrock is region-specific), the gateway can distribute across regions to multiply effective rate limits and reduce latency for geographically distributed users.

Regional routing multiplies rate limit headroom and reduces cross-continent latency.

Cost Tracking and Token Accounting

The gateway sees every request and response, making it the natural place to track spend. Accurate cost tracking requires:

Token counting on both input and output (not just estimates — use the actual usage fields from provider responses)
Per-model pricing tables maintained in the gateway config
Attribution tags — team, feature, environment, user — attached to each request

Pricing Table

A gateway’s pricing config needs regular updates as providers change prices. A partial snapshot as of August 2026:

Provider	Model	Input $/M	Output $/M	Cached Input $/M
OpenAI	GPT-5.6 Sol	$5.00	$30.00	—
OpenAI	GPT-5.6 Terra	$2.00	$12.00	—
OpenAI	GPT-5.6 Luna	$0.20	$1.20	—
OpenAI	GPT-4.1 Nano	$0.10	$0.40	$0.025
Anthropic	Claude Opus 5	$5.00	$25.00	—
Anthropic	Claude Sonnet 5	$2.00*	$10.00*	—

Introductory pricing through Aug 31, 2026; standard rate $3.00/$15.00 from Sept 1, 2026. | Anthropic | Claude Haiku 4.5 | $1.00 | $5.00 | — | | Google | Gemini 3.7 Flash | $0.75 | $3.75* | — |

*Introductory pricing through Dec 31, 2026; standard rate $1.50/$7.50 from Jan 1, 2027. | Mistral | Mistral Large 3 | $0.50 | $1.50 | — | | Mistral | Mistral Small 4 | $0.15 | $0.60 | — |

Note: Sonnet 5 pricing shown is introductory through Aug 31, 2026 ($2/$10), rising to $3/$15 in September. Also, Sonnet 5’s new tokenizer emits ~30% more tokens for the same text, meaning actual per-task cost is roughly 40% higher than the per-token price suggests.

Cost Attribution

Tag requests at ingress with metadata that flows through to the accounting system:

{
  "model": "balanced",
  "messages": [...],
  "metadata": {
    "team": "search",
    "feature": "query-expansion",
    "environment": "production",
    "user_id": "u_abc123"
  }
}

The gateway strips metadata before forwarding to the provider and writes it alongside the cost record. This enables queries like “How much did the search team’s query-expansion feature spend on Anthropic last week?”

Budget Enforcement

Hard spending limits prevent runaway costs. The gateway tracks cumulative spend per attribution tag and rejects requests when a budget is exhausted:

budgets:
  - team: search
    monthly_limit_usd: 15000
    action_on_exceed: reject  # or "downgrade_model" or "alert_only"
  - team: experiments
    monthly_limit_usd: 2000
    action_on_exceed: downgrade_model
    downgrade_to: fast-cheap

Credential Management

A production gateway manages dozens of API keys — multiple keys per provider (for rate limit distribution), keys for different environments, keys with different permission scopes.

Key Rotation

API keys should rotate periodically. The gateway should support loading keys from a secrets manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) rather than environment variables, enabling rotation without restarts.

Keys are fetched from a secrets manager and cached with a TTL, enabling rotation without gateway restarts.

Per-Consumer Scoping

Different consumers of the gateway should get different capabilities. A customer-facing chatbot should only access budget models. An internal research tool might get access to frontier models. The gateway’s own authentication layer (API keys, JWTs, mTLS) maps each consumer to an allowed set of models and providers.

consumers:
  chatbot-prod:
    api_key_hash: "sha256:..."
    allowed_models: [fast-cheap, balanced]
    rate_limit_rpm: 1000
    budget_monthly_usd: 5000
  research-internal:
    api_key_hash: "sha256:..."
    allowed_models: [fast-cheap, balanced, best-quality, code]
    rate_limit_rpm: 200
    budget_monthly_usd: 20000

Credential Isolation

The gateway should never log API keys, even at debug level. Keys in error messages should be masked. Provider responses sometimes include the API key in error payloads — the gateway should strip these before returning to the consumer.

Caching at the Gateway Layer

Exact-match caching works better than expected for LLM workloads. Many applications send identical or near-identical requests repeatedly: the same system prompt with the same few-shot examples, classification of duplicate inputs, repeated tool descriptions.

Cache Key Construction

A cache key should include everything that affects the output:

import hashlib
import json

def cache_key(request: dict) -> str:
    # Include model, messages, temperature, tools, response_format
    # Exclude metadata, request_id, stream flag
    cacheable = {
        "model": request["model"],
        "messages": request["messages"],
        "temperature": request.get("temperature", 1.0),
        "tools": request.get("tools"),
        "response_format": request.get("response_format"),
    }
    serialized = json.dumps(cacheable, sort_keys=True)
    return hashlib.sha256(serialized.encode()).hexdigest()

Cache Invalidation

LLM responses don’t expire in the traditional sense — a cached response to “What is 2+2?” is valid indefinitely. But responses involving current events, live data, or user-specific context shouldn’t be cached. The gateway should support:

TTL-based expiration (e.g., 1 hour for general queries, 0 for queries tagged as time-sensitive)
Temperature-based cache skipping — if temperature > 0, the caller probably wants varied responses, so caching defeats the purpose. In practice, cache only when temperature == 0.
Explicit cache bypass via a request header

Provider-Level Caching

Several providers offer their own caching. OpenAI’s prompt caching automatically caches repeated prefixes and charges reduced rates (e.g., $0.025/M for cached input on GPT-4.1 Nano vs. $0.10/M standard). Anthropic offers similar prefix caching. The gateway’s cache sits in front of these — catching exact matches before the request reaches the provider — and the provider’s cache handles partial prefix matches that the gateway can’t detect.

Streaming Translation

Streaming is where provider differences are most painful. Each provider uses a different Server-Sent Events (SSE) format.

OpenAI sends data: {"choices":[{"delta":{"content":"token"}}]} events, terminated by data: [DONE].

Anthropic uses typed events: event: content_block_delta with data: {"type":"content_block_delta","delta":{"type":"text_delta","text":"token"}}. The stream starts with message_start, includes content_block_start events, and ends with message_stop.

Google Gemini sends JSON objects with candidates[0].content.parts[0].text in each chunk, either as SSE or newline-delimited JSON depending on the endpoint.

The gateway must:

Open a streaming connection to the upstream provider
Parse each provider-specific chunk
Translate it into the gateway’s canonical streaming format
Forward it to the consumer with minimal added latency

The gateway translates provider-specific streaming formats into a single canonical SSE format.

A critical subtlety: token counting for cost tracking requires accumulating the full response during streaming, because most providers only include usage data in the final chunk (Anthropic) or in a separate usage field after stream completion (OpenAI). The gateway needs to buffer token counts without buffering the actual content.

async def stream_translate(upstream_response, provider: str):
    total_output_tokens = 0
    
    async for chunk in upstream_response:
        if provider == "anthropic":
            if chunk.get("type") == "content_block_delta":
                text = chunk["delta"]["text"]
                yield {"type": "text", "content": text}
            elif chunk.get("type") == "message_delta":
                total_output_tokens = chunk.get("usage", {}).get("output_tokens", 0)
        elif provider == "openai":
            delta = chunk.get("choices", [{}])[0].get("delta", {})
            if "content" in delta:
                yield {"type": "text", "content": delta["content"]}
            if "usage" in chunk:
                total_output_tokens = chunk["usage"]["completion_tokens"]
    
    yield {"type": "done", "usage": {"output_tokens": total_output_tokens}}

Observability and Logging

The gateway is the single chokepoint for all LLM traffic, making it the ideal place to collect observability data.

Metrics to Emit

Metric	Type	Dimensions
`llm.request.duration_ms`	Histogram	provider, model, consumer, status
`llm.request.ttft_ms`	Histogram	provider, model (streaming only)
`llm.tokens.input`	Counter	provider, model, consumer
`llm.tokens.output`	Counter	provider, model, consumer
`llm.cost.usd`	Counter	provider, model, consumer, team
`llm.request.count`	Counter	provider, model, consumer, status
`llm.fallback.count`	Counter	from_provider, to_provider
`llm.cache.hit_rate`	Gauge	consumer, model
`llm.circuit_breaker.state`	Gauge	provider (0=closed, 1=open)

Request Logging

Log the full request and response for debugging and evaluation, but with care:

Redact PII before writing logs. The gateway can run a lightweight PII detector on inputs and outputs.
Separate hot and cold paths. Metrics go to a time-series database (Prometheus, Datadog) synchronously. Full request/response logs go to an async queue (Kafka, SQS) for batch processing into a data warehouse.
Sampling. Logging 100% of requests at high volume is expensive. Log all errors, all fallbacks, and a configurable percentage (e.g., 10%) of successful requests.

Integration with LLM Observability Tools

Gateways should emit data in formats compatible with dedicated LLM observability platforms — Langfuse, Helicone, Braintrust, and similar tools expect OpenTelemetry spans or custom webhook payloads. The gateway can attach trace IDs and span metadata so that a multi-step agent workflow’s individual LLM calls are correlated.

Observability data flows to three destinations: real-time metrics, async log storage, and LLM-specific observability platforms.

Open-Source and Commercial Gateways Compared

The gateway space has matured. Several open-source and commercial options exist, each with different strengths.

Open-Source Gateways

LiteLLM is the most widely adopted open-source LLM gateway. It supports 100+ providers behind an OpenAI-compatible API, handles streaming translation, and includes basic load balancing and fallback. Written in Python, deployable as a proxy server. Strengths: breadth of provider support, active community, quick to add new models. Weaknesses: Python performance ceiling under high concurrency, limited built-in cost tracking, basic routing logic.

Portkey AI Gateway is an open-source gateway (Node.js/TypeScript) with a focus on reliability features — fallbacks, retries, caching, and load balancing. It supports conditional routing and has a declarative config format. More opinionated than LiteLLM about how routing should work.

Kong AI Gateway extends the Kong API gateway with LLM-specific plugins — prompt decoration, rate limiting by tokens, multi-provider routing. Benefits from Kong’s mature ecosystem (authentication, rate limiting, logging plugins) but adds complexity if an organization isn’t already using Kong.

Commercial/Managed Gateways

AWS Bedrock acts as a managed gateway for multiple model providers (Anthropic, Meta, Mistral, and others) behind a unified AWS API. Handles credential management, provides CloudWatch integration, and supports cross-region inference. Pricing includes a per-token markup over direct provider pricing.

Google Vertex AI similarly unifies access to Gemini models and partner models. Offers Model Garden for model discovery and supports VPC Service Controls for network isolation.

Azure AI Foundry provides access to OpenAI models and others through Azure’s identity and networking stack. API Management (APIM) can sit in front for rate limiting and cost tracking.

These managed options trade flexibility for operational simplicity — no infrastructure to manage, but less control over routing logic and higher per-token costs.

Comparison Matrix

Feature	LiteLLM	Portkey	Kong AI	Bedrock	Vertex AI
Self-hosted	✓	✓	✓	✗	✗
Providers supported	100+	30+	15+	10+	10+
OpenAI-compatible API	✓	✓	✓	✗ (own SDK)	✗ (own SDK)
Streaming translation	✓	✓	✓	✓	✓
Fallback chains	✓	✓	✓	Limited	Limited
Token-aware rate limiting	Basic	✓	✓	✓	✓
Cost tracking	Basic	✓	Plugin	CloudWatch	Billing
Semantic caching	✗	✓	✗	✗	✗
Language	Python	TypeScript	Lua/Go	Managed	Managed

Building vs Buying

The build-vs-buy decision depends on three factors:

Traffic volume. Below 10,000 LLM requests/day, a managed gateway (Bedrock, Vertex) or a simple LiteLLM deployment is sufficient. The operational overhead of a custom gateway isn’t justified.

Routing complexity. If routing logic is simple (one provider, one model, maybe a fallback), LiteLLM or direct SDK calls with a retry wrapper work fine. If routing involves cost optimization, A/B testing, compliance constraints, and per-consumer policies, a custom gateway or heavily configured open-source option becomes necessary.

Latency sensitivity. A Python proxy adds 5-20ms of overhead per request. For streaming responses where time-to-first-token matters, a high-performance gateway (Go, Rust) or a managed service with colocated infrastructure reduces this.

When to Build Custom

Build a custom gateway when:

Routing logic is a competitive advantage (novel cost-optimization algorithms, proprietary model selection)
Security requirements demand full control over credential handling and network paths
Scale exceeds 1M+ requests/day and the Python overhead of LiteLLM becomes measurable
Integration with internal systems (identity, billing, compliance) requires deep customization

When to Use Off-the-Shelf

Use an existing gateway when:

The team should be spending engineering time on application logic, not infrastructure
Requirements are standard (failover, basic routing, cost tracking)
Provider coverage matters more than routing sophistication

A pragmatic middle ground: start with LiteLLM, add custom routing logic as middleware, and replace it with a custom solution only when LiteLLM becomes a bottleneck.

Reference Architecture

A production-grade LLM gateway architecture for a mid-size organization (50-200 developers, 100K-1M LLM requests/day):

High-level deployment: stateless gateway instances behind a load balancer, pulling config from a central store.

Component Details

Gateway instances are stateless — all state (rate limit counters, circuit breaker status, cache entries) lives in Redis or a similar shared store. This enables horizontal scaling and zero-downtime deployments.

Config store holds routing rules, pricing tables, consumer policies, and budget limits. Changes to config are hot-reloaded without restarting gateway instances. This can be a Git repository (with a webhook triggering reload), a database, or a dedicated config service like etcd.

Cache layer (Redis) stores exact-match cache entries and rate limit counters. Deployed as a Redis cluster for availability, with eviction policies sized to the working set.

Async pipeline (Kafka or equivalent) receives request/response logs from the gateway and feeds them into a data warehouse for cost reporting, evaluation, and audit.

Supporting infrastructure: Redis for hot state, Kafka for async log shipping to a data warehouse.

Deployment Patterns

Sidecar deployment: The gateway runs as a sidecar container alongside each application pod. Eliminates network hop to a central gateway but multiplies the number of gateway instances and complicates config management. Suitable for latency-critical applications in Kubernetes.

Central proxy: All LLM traffic routes through a dedicated gateway service. Simpler to manage, easier to enforce policies consistently, adds one network hop (typically 1-5ms within the same region).

Edge gateway + central gateway: An edge layer (CDN or API gateway like Kong/Envoy) handles authentication and rate limiting. A central LLM gateway handles provider translation and routing. This separates generic API management from LLM-specific logic.

The central proxy is the right default for most teams. Switch to sidecar only if the added latency is measurable and problematic.

Summary

An LLM gateway centralizes five concerns that otherwise sprawl across application code: provider translation, routing, failover, cost tracking, and credential management.

Provider abstraction requires translating message formats, tool calling schemas, streaming events, and structured output modes between incompatible APIs. The differences are deeper than field renames — Anthropic separates system prompts, Gemini uses a different tool result format, and streaming event schemas have nothing in common.

Routing ranges from static model mapping (canonical names to provider models) to dynamic cost-based, latency-based, and compliance-based selection. Cost-based routing — sending simple tasks to budget models — probably delivers the largest ROI of any gateway feature, given the 50x price difference between GPT-4.1 Nano and GPT-5.6 Sol.

Fallback and retry logic must distinguish between retryable errors (429, 5xx) and non-retryable ones (context length exceeded, auth failure). Circuit breakers prevent hammering a degraded provider. Timeout calibration matters — LLM calls are 10-100x slower than typical API calls.

Cost tracking requires accurate token counting from provider responses, maintained pricing tables, and attribution tags on every request. Budget enforcement at the gateway prevents runaway spend.

Start simple. LiteLLM behind a load balancer covers most needs. Add custom routing logic when the cost or quality gains justify it. Build a custom gateway only when routing logic is a competitive advantage or scale demands it.