How the inference stack works from API call to GPU kernel — routing, batching, KV cache management, quantization formats, speculative decoding, and where latency hides.

The Inference Stack Top to Bottom

Every time an application sends a chat completion request, that request passes through roughly eight distinct layers before a GPU executes a single matrix multiplication. Most developers interact only with the top layer — the HTTP API — and treat everything below it as a black box. Understanding what happens inside that box determines whether a production system hits its latency targets, stays within budget, and scales past the first thousand concurrent users.

This post traces a single inference request from the moment it leaves a client to the moment tokens stream back, covering every layer where performance is won or lost.

The Request Path Overview
Layer 1: API Gateway and Authentication
Layer 2: Request Routing and Model Selection
Layer 3: Prompt Processing and Tokenization
Layer 4: The Scheduling Layer
Layer 5: Continuous Batching
Layer 6: The KV Cache
Layer 7: Quantization
Layer 8: GPU Kernel Execution
Speculative Decoding
Putting It Together: Where Latency Hides
Summary
Further Reading

The Request Path Overview

End-to-end path of a single inference request, from client to GPU kernel execution.

The entire round trip for a short prompt on a fast model (say, GPT-6 Luna or Claude Haiku 5.5) takes 200–800ms to first token. For a longer prompt on a reasoning-heavy model like Claude Fable 5.1 or Gemini 4 Argon, time-to-first-token (TTFT) can stretch to several seconds. The breakdown of where that time goes is rarely what developers expect.

Layer 1: API Gateway and Authentication

The gateway is the first thing a request touches. For managed providers (OpenAI, Anthropic, Google), this layer handles:

TLS termination — every request is HTTPS. The TLS handshake adds 1–3 round trips on a fresh connection, which is why connection reuse via HTTP/2 or keep-alive matters.
API key validation — a database or cache lookup against the key, checking organization, tier, and permissions.
Rate limiting — token-aware throttling (covered in a previous deep dive), not just request-count limits. OpenAI, Anthropic, and Google all enforce both requests-per-minute and tokens-per-minute limits.
Request validation — schema checks, parameter bounds (temperature range, max_tokens ceiling), and model availability for the caller’s tier.

Gateway latency is typically 5–20ms for warm connections. The expensive case is a cold connection over high-latency links — a client in São Paulo hitting a US-East endpoint can add 150ms+ of network round-trip before the gateway even parses the JSON body.

# Connection reuse matters more than most optimizations
import httpx

# Bad: new connection per request
async def slow_inference(prompt: str):
    async with httpx.AsyncClient() as client:  # new TLS handshake each time
        return await client.post("https://api.openai.com/v1/chat/completions", ...)

# Good: reuse the client across requests
client = httpx.AsyncClient(http2=True, timeout=120.0)

async def fast_inference(prompt: str):
    return await client.post("https://api.openai.com/v1/chat/completions", ...)

For self-hosted stacks (vLLM, llama.cpp), the gateway is whatever sits in front of the inference server — often an nginx reverse proxy or a Go service. The same principles apply: TLS overhead, connection reuse, and request validation all happen here.

Layer 2: Request Routing and Model Selection

After authentication, the request must reach the correct model instance. This is straightforward for a single-model deployment but complex at provider scale.

Model-tier routing distributes requests across GPU pools optimized for different workload profiles.

Routing decisions at major providers probably include:

Model-to-cluster mapping — different models run on different hardware. A flagship like GPT-6 Astra likely runs on H100 or newer clusters, while a budget model like GPT-6 Luna might run on a mix of A100 and H100.
Geographic routing — requests route to the nearest datacenter with capacity, which is why latency varies by region.
Load-aware routing — if a cluster is saturated, requests queue or route to a secondary cluster. This is the primary cause of variable TTFT — a request hitting a busy cluster waits in queue.
Priority tiers — paid tiers and higher-rate-limit organizations probably get priority queue placement, which is why Tier 5 API users see more consistent latency than free-tier users.

For self-hosted inference, tools like LiteLLM and custom gateway layers handle routing across multiple model backends:

# LiteLLM routing across multiple backends
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "fast-model",
            "litellm_params": {
                "model": "openai/gpt-6-luna",
                "api_key": "sk-...",
            },
        },
        {
            "model_name": "fast-model",
            "litellm_params": {
                "model": "anthropic/claude-haiku-5-5",
                "api_key": "sk-ant-...",
            },
        },
    ],
    routing_strategy="latency-based-routing",
    fallbacks=[{"fast-model": ["fast-model"]}],
)

Layer 3: Prompt Processing and Tokenization

Before the model can process a prompt, the text must be converted into token IDs. Tokenization is deterministic and fast (microseconds for short prompts, low milliseconds for long ones), but the choice of tokenizer has downstream effects.

Tokenizer differences matter for cost. Claude 4.7 and later models use a tokenizer that emits roughly 30% more tokens for the same text compared to earlier Claude models. This means identical prompts cost more on newer Claude models purely due to tokenization — a consideration when comparing per-token pricing across providers.

Tokenization converts text to integer IDs, which the embedding layer maps to dense vectors on the GPU.

Most current models use Byte-Pair Encoding (BPE) variants:

Provider	Tokenizer	Vocab Size (approx.)	Notes
OpenAI (GPT-6 family)	cl200k or successor	~200K	Efficient for code and multilingual
Anthropic (Claude 5.x)	Custom BPE	Not publicly documented	30% more tokens than pre-4.7 tokenizer
Google (Gemini)	SentencePiece	~256K	Handles 100+ languages
Open-weight (Llama 4, Qwen3.8)	SentencePiece / tiktoken variants	128K–152K	Varies by model family

The token count directly determines:

Cost — billed per million input and output tokens.
Context consumption — a 1M-context model is 1M tokens, not 1M characters. English text averages roughly 4 characters per token, but code, non-Latin scripts, and structured data (JSON, XML) can be much less efficient.
Prefill compute — the time to process the prompt scales with token count (more on this in the GPU execution section).

Layer 4: The Scheduling Layer

The scheduler sits between the request queue and the inference engine. Its job is to decide when and in what grouping each request gets processed. This layer is where most of the latency variance originates.

The scheduler assigns requests to prefill and decode phases, managing GPU memory and SLO targets.

Two distinct compute phases exist for every request:

Prefill (prompt processing) — the entire input prompt is processed in parallel. This is compute-bound: proportional to prompt_length × model_parameters. For a 100K-token prompt on a large model, prefill can take 2–10 seconds.
Decode (token generation) — tokens are generated one at a time (or in small speculative batches). This is memory-bandwidth-bound: each decode step reads the full model weights and KV cache but does relatively little compute per byte read.

The scheduler must balance these two very different workloads. A common strategy is disaggregated prefill and decode: run prefill on one set of GPUs and decode on another. This prevents a long-prompt prefill from blocking decode for other requests that are actively streaming.

vLLM, the most widely used open-source serving engine, implements a priority-based scheduler that considers:

Available GPU memory (primarily KV cache space)
Request priority and arrival time
Whether a request can be preempted (swapped to CPU memory) if a higher-priority request arrives
Maximum batch size targets for throughput optimization

Layer 5: Continuous Batching

Static batching — collect N requests, process them together, wait for all to finish — wastes GPU cycles because short completions must wait for long ones. Continuous batching (also called iteration-level batching) solves this.

Continuous batching inserts new requests into the active batch as slots free up, rather than waiting for the entire batch to complete.

In continuous batching, the batch is dynamic:

When a request emits its end-of-sequence token, its slot frees immediately.
A new request from the queue fills that slot at the next iteration.
No GPU cycles are wasted waiting for the slowest request in the batch.

The throughput improvement is substantial. vLLM’s original paper reported 2–4x throughput gains over static batching on realistic workloads. Current production systems routinely batch 64–256+ concurrent decodes on a single GPU node.

The tradeoff: larger batch sizes increase per-token latency because each decode step processes more sequences. For latency-sensitive applications, operators cap the batch size and accept lower throughput. For batch processing (embeddings, evals, offline summarization), they maximize batch size for cost efficiency.

# vLLM configuration: balancing throughput vs latency
from vllm import LLM, SamplingParams

# High-throughput configuration
llm = LLM(
    model="Qwen/Qwen3.8-27B",
    max_num_batched_tokens=32768,   # large prefill batch
    max_num_seqs=256,               # many concurrent decodes
    gpu_memory_utilization=0.92,    # aggressive memory use
)

# Low-latency configuration
llm = LLM(
    model="Qwen/Qwen3.8-27B",
    max_num_batched_tokens=8192,
    max_num_seqs=32,                # smaller batch = lower per-token latency
    gpu_memory_utilization=0.85,
)

Layer 6: The KV Cache

The KV (key-value) cache is the single most important data structure in inference. It stores the attention keys and values computed during prefill so that each decode step doesn’t need to recompute attention over the entire prompt.

Without a KV cache, generating 1,000 tokens from a 10,000-token prompt would require reprocessing all previous tokens at every step — 10,000 + 10,001 + 10,002 + … attention operations. With the cache, each decode step only computes attention for the new token against the cached keys and values.

Memory consumption is the limiting factor. The KV cache for a single request scales as:

KV cache size = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element

For a 70B-parameter model with 80 layers, 8 KV heads, 128-dimensional heads, and FP16 (2 bytes):

2 × 80 × 8 × 128 × seq_len × 2 bytes = 327,680 × seq_len bytes

At 100K tokens: 327,680 × 100,000 = ~30.5 GB for a single request’s KV cache. This is why long-context inference is expensive — the KV cache, not the model weights, becomes the dominant memory consumer.

Each decode step reads from the KV cache and appends a new entry, growing the cache by one position per generated token.

PagedAttention

The key innovation in vLLM (from the 2023 Kwon et al. paper) is PagedAttention — managing KV cache memory like an operating system manages virtual memory. Instead of allocating a contiguous block of GPU memory for each request’s maximum possible sequence length, PagedAttention allocates fixed-size blocks (pages) on demand.

This eliminates two problems:

Internal fragmentation — a request allocated for 4,096 tokens but only using 500 wastes 87% of its allocation under naive schemes. PagedAttention allocates only the pages needed.
External fragmentation — non-contiguous free memory can still be used because pages don’t need to be physically adjacent.

The memory savings are typically 50–70%, which translates directly into higher batch sizes and throughput.

Prefix Caching

When multiple requests share the same prompt prefix (common with system prompts, few-shot examples, or RAG templates), the KV cache for that prefix can be computed once and reused. This is what providers call “cached input” pricing:

Provider	Standard Input	Cached Input	Savings
GPT-6 Astra	$10/M	$1.00/M	90%
GPT-6.1 Sol	$2/M	$0.10/M	95%
Claude Opus 5.5	$4/M	$0.20/M	95%
Claude Sonnet 5.5	$2/M	$0.10/M	95%
Claude Fable 5.1	$10/M	$0.25/M	97.5%
Gemini 4 Argon	$2/M (introductory)	95% off	95%

Prefix caching reduces both cost and latency. The prefill phase for cached tokens is nearly free — the system loads precomputed KV entries from a cache store rather than running them through the model.

In self-hosted vLLM deployments, automatic prefix caching is enabled with a single flag:

llm = LLM(
    model="meta-llama/Llama-4-Scout",
    enable_prefix_caching=True,  # enable automatic prefix cache
)

KV Cache Compression

For very long contexts (1M tokens on Gemini 4 Argon or Claude Fable 5.1), raw FP16 KV caches are impractical. Several compression strategies exist:

KV cache quantization — storing keys and values in INT8 or FP8 instead of FP16, halving memory with minimal quality loss. vLLM supports FP8 KV cache quantization.
Sliding window attention — models like Mistral use a fixed attention window (e.g., 4,096 tokens) for most layers, only attending to the full context in select layers. This caps KV cache growth.
GQA (Grouped Query Attention) — using fewer KV heads than query heads (e.g., 8 KV heads for 64 query heads). Most current models use GQA; it reduces KV cache size by 4–8x compared to multi-head attention.
Token eviction / compression — dropping or merging old KV entries that the model is unlikely to attend to. Research-stage for most systems.

Layer 7: Quantization

Quantization reduces model weight precision from FP16 (2 bytes) or BF16 to lower-precision formats, shrinking memory footprint and increasing throughput. The tradeoffs are nuanced.

Quantization levels trade precision for memory savings, with quality impact varying by model and method.

Quantization Formats in Production

Format	Bits	Typical Quality Loss	Where Used
BF16	16	Baseline	Training, high-quality inference
FP8 (E4M3/E5M2)	8	<1% on most benchmarks	H100/B100 native, vLLM default for large models
INT8 (W8A8)	8	<1%	TensorRT-LLM, vLLM
GPTQ (INT4)	4	1–3% depending on model	Widely available on Hugging Face
AWQ (INT4)	4	1–2%	Generally better than GPTQ at 4-bit
GGUF Q4_K_M	~4.5	1–3%	llama.cpp, Ollama
GGUF Q5_K_M	~5.5	<1%	llama.cpp, best quality/size tradeoff
GGUF Q8_0	8	<0.5%	llama.cpp, near-lossless

FP8 is the default for production serving on H100s. The H100’s FP8 Tensor Cores deliver nearly 2x the throughput of FP16, and quality differences are negligible on current-generation models. vLLM and TensorRT-LLM both default to FP8 when running on H100 hardware.

INT4 quantization is where consumer hardware gets interesting. A 70B-parameter model in FP16 requires ~140GB of memory (far exceeding any single consumer GPU). In INT4 (AWQ or GPTQ), it fits in ~35GB — runnable on two RTX 4090s (24GB each) or a single GPU with sufficient VRAM. The Qwen3.8-27B model in Q4_K_M GGUF fits comfortably in ~16GB and runs on a single consumer GPU.

# Running a quantized model with llama.cpp
./llama-server \
  --model Qwen3.8-27B-Q4_K_M.gguf \
  --ctx-size 32768 \
  --n-gpu-layers 99 \
  --flash-attn \
  --port 8080

Weight-Only vs. Weight-and-Activation Quantization

Weight-only quantization (GPTQ, AWQ, GGUF) — only model weights are stored in low precision. Activations are computed in FP16/BF16. Simpler to implement, works with any hardware.
Weight-and-activation quantization (W8A8, FP8) — both weights and activations use reduced precision. Requires hardware support (Tensor Cores on H100 for FP8). Higher throughput because the actual matrix multiplications happen in lower precision.

The distinction matters for throughput. Weight-only INT4 reduces memory and speeds up weight loading, but the actual compute still runs at FP16. W8A8 quantization on H100 hardware reduces both memory and compute cost.

Layer 8: GPU Kernel Execution

At the bottom of the stack, everything reduces to GPU kernel launches. The two dominant operations in transformer inference are:

Matrix multiplications (GEMMs) — weight × activation products in linear layers, consuming 90%+ of compute FLOPs.
Attention — computing softmax(QK^T/√d)V over the KV cache, which is memory-bandwidth-bound during decode.

Core GPU operations per transformer layer: GEMMs for linear projections, FlashAttention for the attention mechanism, and element-wise operations for normalization and activation functions.

Prefill vs. Decode: Different Bottlenecks

Prefill is compute-bound. Processing a 10K-token prompt means running large GEMMs where the batch dimension equals the prompt length. The GPU’s Tensor Cores are fully utilized. On an H100 (989 TFLOPS FP16, ~1,979 TFLOPS FP8), prefill for a 70B model at 10K tokens takes roughly 1–2 seconds.

Decode is memory-bandwidth-bound. Each decode step generates one token per sequence. The GEMM batch dimension equals the number of concurrent sequences (the batch size), which is much smaller than prompt length. The bottleneck becomes reading the model weights from HBM (High Bandwidth Memory). An H100 has 3.35 TB/s HBM bandwidth. Reading a 70B-parameter model in FP16 (140GB) takes ~42ms. In FP8 (70GB), ~21ms.

This is why decode throughput scales roughly linearly with batch size until the compute units saturate: doubling the batch size barely increases the time per decode step (same weight reads, slightly more activation compute), effectively doubling throughput.

FlashAttention

FlashAttention (Dao et al., 2022; now at version 3 for H100) restructures the attention computation to minimize HBM reads/writes. Standard attention materializes the full N×N attention matrix in HBM; FlashAttention tiles the computation and keeps intermediates in on-chip SRAM (the GPU’s fast L1 cache / shared memory).

The result: 2–4x faster attention and dramatically reduced memory usage (no N×N matrix in HBM). FlashAttention is the default in every major inference engine — vLLM, TensorRT-LLM, and llama.cpp all use it.

For decode-phase attention against a long KV cache, FlashDecoding (a variant optimized for the case where query length is 1 but KV cache length is large) splits the KV cache across thread blocks and reduces across them, keeping GPU utilization high.

Tensor Parallelism

Models too large for a single GPU’s memory are split across multiple GPUs using tensor parallelism (TP). Each linear layer’s weight matrix is sharded column-wise or row-wise across GPUs. After each layer, an all-reduce operation synchronizes results.

Tensor parallelism shards each layer’s weights across GPUs, with an all-reduce synchronization after each layer.

NVLink bandwidth determines how much overhead tensor parallelism adds. On H100 SXM (900 GB/s bidirectional NVLink per GPU in an 8-GPU node), the overhead is small — typically 5–15% for TP-8. On PCIe-connected GPUs, the overhead is much larger (PCIe Gen5 maxes out at 128 GB/s per link), making TP across PCIe GPUs practical only for TP-2.

Configuration	Typical TP Overhead	Use Case
TP-2 (NVLink)	2–5%	70B models on 2× H100
TP-4 (NVLink)	5–10%	200B+ models
TP-8 (NVLink)	10–15%	400B+ models, large MoE
TP-2 (PCIe)	15–25%	Consumer/workstation GPUs
TP-4+ (PCIe)	30%+	Generally not worth it

Speculative Decoding

Speculative decoding is the most impactful inference optimization that doesn’t require model changes. The insight: decode is memory-bandwidth-bound, and the GPU has spare compute capacity during each step.

Speculative decoding uses a small draft model to propose K candidate tokens, then the full target model verifies them in a single forward pass, accepting the longest valid prefix.

The mechanism:

A small draft model (often 1–2B parameters, or a separate prediction head on the main model) generates K candidate tokens autoregressively. This is fast because the draft model is small.
The target model runs a single forward pass with all K candidates as input. Because prefill processes tokens in parallel, this single pass is barely more expensive than generating one token.
The system accepts the longest prefix of draft tokens that matches what the target model would have generated (using a probabilistic acceptance criterion that preserves the target model’s output distribution exactly).
If all K tokens are accepted, the system gained K tokens for the cost of ~1 decode step on the target model. If only J < K are accepted, it still gained J tokens plus the correctly sampled (J+1)th token.

Throughput gain depends on the acceptance rate. If the draft model agrees with the target model 80% of the time and K=5, expected tokens per step ≈ 4, a ~4x speedup in decode throughput with zero quality degradation.

Speculative decoding is implemented in vLLM, TensorRT-LLM, and llama.cpp. Current frontier providers almost certainly use variants of it in production, though they don’t publicly confirm implementation details.

# vLLM speculative decoding configuration
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-4-Maverick",
    speculative_model="meta-llama/Llama-4-Scout",  # smaller model as draft
    num_speculative_tokens=5,  # propose 5 tokens per step
    speculative_draft_tensor_parallel_size=1,  # draft on 1 GPU
)

Self-Speculative Decoding and Medusa

Two variants avoid needing a separate draft model:

Self-speculative decoding — the target model skips some of its own layers to produce a draft, then runs the full model for verification. Avoids loading a separate model into memory.
Medusa heads — additional prediction heads attached to the target model that predict multiple future tokens simultaneously. Each head is a small MLP trained to predict the token at position t+1, t+2, etc. These heads add minimal parameters and run in parallel with the main model’s final layer.

Putting It Together: Where Latency Hides

The latency profile of an inference request breaks down into components that are often surprising:

Latency breakdown for a typical inference request. Queue wait time is the most variable component and the hardest to control as a client.

Time-to-First-Token (TTFT) Breakdown

Component	Short Prompt (500 tokens)	Long Prompt (100K tokens)
Network (warm connection)	10–30ms	10–30ms
Gateway processing	5–15ms	5–15ms
Queue wait (varies with load)	0–2000ms	0–5000ms
Prefill	50–200ms	2–10 seconds
Total TTFT	65–2245ms	2.015–15.045s

The queue wait is the variable that makes TTFT unpredictable. Under light load, it’s near zero. During peak traffic, it can dominate total latency. This is why provider SLAs typically guarantee 99th-percentile latency rather than median — the tail is long.

Decode Throughput

Output token rates vary by model size, hardware, quantization, and batch size:

Setup	Approximate Output Speed
GPT-6 Luna (API, normal mode)	100–200 tok/s per request (estimated)
Claude Haiku 5.5 (API)	80–150 tok/s per request (estimated)
Qwen3.8-27B Q4_K_M on RTX 4090	35–50 tok/s
Llama 4 Scout FP8 on H100	50–80 tok/s per request
Same model, batch of 64	15–25 tok/s per request, 1000–1600 tok/s aggregate

API providers don’t publish exact throughput numbers, and real-world rates depend on current load. The estimates above are based on typical observed streaming speeds.

Optimization Priority

For latency-sensitive applications (chatbots, code completion), the optimization priority is:

Reduce queue wait — use higher-tier API access, deploy in the same region as the inference cluster, or self-host to control queuing.
Minimize TTFT via prefix caching — reuse system prompts and few-shot examples. The 95%+ cost savings on cached input tokens come with proportional latency savings.
Enable speculative decoding — for self-hosted deployments, this is the single biggest decode throughput win.
Right-size the model — GPT-6 Luna or Claude Haiku 5.5 at 200ms TTFT beats GPT-6 Astra at 2s TTFT for most interactive use cases. Quality differences may not justify the latency penalty.
Connection reuse — HTTP/2, persistent connections, and regional endpoints.

For throughput-sensitive applications (batch processing, evals, data extraction):

Maximize batch size — use the batch API (OpenAI’s Batch API gives 50% cost reduction) or self-host with aggressive batching.
Use continuous batching — vLLM or TensorRT-LLM with high max_num_seqs.
Quantize aggressively — FP8 or INT4 for self-hosted. The quality difference at INT4 is rarely measurable for extraction tasks.
Parallelize requests — concurrent API calls up to rate limits.

The Full Stack for a Self-Hosted Deployment

A production self-hosted inference deployment touches every layer:

Self-hosted inference stack: each layer maps to the conceptual layers described in this post.

A concrete example with vLLM serving Qwen3.8-27B:

# Launch vLLM with production settings
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3.8-27B \
  --dtype float16 \
  --quantization fp8 \
  --max-model-len 32768 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 2 \
  --port 8000

This configuration:

Serves the model in FP8 quantization across 2 GPUs (tensor parallel)
Enables prefix caching for system prompt reuse
Allows up to 128 concurrent sequences in the decode batch
Uses 90% of available GPU memory (mostly for KV cache)
Exposes an OpenAI-compatible API on port 8000

Summary

The inference stack has eight layers, each with distinct optimization levers:

Gateway — connection reuse and regional routing dominate client-side latency. HTTP/2 and persistent connections save 100ms+ on cold starts.
Routing — model selection is the highest-leverage decision. A fast model at low latency often beats a frontier model at high latency for interactive use cases.
Tokenization — deterministic and fast, but tokenizer choice affects cost. Claude’s post-4.7 tokenizer produces ~30% more tokens for the same text.
Scheduling — disaggregated prefill/decode prevents long prompts from blocking interactive requests. Queue wait time is the primary source of TTFT variance.
Continuous batching — 2–4x throughput over static batching. Every production serving engine uses it.
KV cache — the dominant memory consumer for long contexts. PagedAttention, GQA, and prefix caching are the key optimizations. Prefix caching saves 95%+ on repeated prompt prefixes.
Quantization — FP8 is the production default on H100s with negligible quality loss. INT4 (AWQ/GPTQ) enables large models on consumer hardware with 1–3% quality degradation.
GPU kernels — FlashAttention and Tensor Core utilization determine raw throughput. Prefill is compute-bound; decode is memory-bandwidth-bound.
Speculative decoding — 2–4x decode speedup with zero quality loss, limited by draft-target agreement rate.

The most common mistake is optimizing the wrong layer. Shaving 5ms off kernel execution while requests sit in a queue for 2 seconds is wasted effort. Profile the full stack, identify the bottleneck, and optimize there.