Every major AI provider handles GPU memory, batching, and model serving differently. This is how the inference stack works from HTTP request to matrix multiply.

The GPU Landscape for AI Inference: H100 vs A100 vs L40S vs Consumer GPUs, Cloud Pricing, and the Self-Host vs API Decision

The Hardware Hierarchy
Memory Architecture and Why It Dominates Everything
The NVIDIA Datacenter Lineup
The B-Series Generation: B100 and B200
Consumer and Prosumer GPUs for Inference
AMD and Intel: The Alternatives
Apple Silicon for Inference
Cloud GPU Pricing Comparison
Spot and Preemptible Instances
Model-to-GPU Mapping: What Fits Where
Quantization and Its Effect on Hardware Requirements
Self-Host vs API: The Decision Framework
The Cost Crossover Math
Multi-GPU and Tensor Parallelism
Inference Engines and Their Hardware Preferences
Summary
Further Reading

GPU selection for AI inference is a memory bandwidth problem disguised as a compute problem. The limiting factor for autoregressive LLM inference — where tokens are generated one at a time — is almost never FLOPS. It’s how fast weights can be moved from HBM to the compute cores. This reality shapes every hardware decision, from choosing between an H100 and an L40S to deciding whether self-hosting makes financial sense at all.

The Hardware Hierarchy

The GPU landscape for inference splits into four tiers, each with distinct cost-performance profiles and use cases.

The GPU tier hierarchy from datacenter premium to consumer hardware, with approximate per-unit cost ratios.

The key insight: moving down tiers trades throughput and multi-GPU scaling for dramatically lower cost per card. A single RTX 4090 can run a 70B parameter model (quantized) at acceptable latency for a single user. An H100 can serve that same model to hundreds of concurrent users. The choice depends entirely on scale.

Memory Architecture and Why It Dominates Everything

LLM inference has two distinct phases with different hardware bottlenecks:

Prefill (prompt processing): All input tokens are processed in parallel. This phase is compute-bound — more FLOPS means faster time-to-first-token (TTFT). Matrix multiplications are large and efficient.

Decode (token generation): Tokens are generated one at a time. Each token requires reading the entire model’s weights from memory for what amounts to a matrix-vector multiply. This phase is memory-bandwidth-bound.

For most production workloads, decode dominates wall-clock time. A model with 70B parameters in FP16 occupies 140 GB of memory. Generating each token requires reading those 140 GB (plus KV cache) from HBM. At 3.35 TB/s (H100 SXM), that’s a theoretical minimum of ~42ms per token, or ~24 tokens/second — regardless of the H100’s 989 TFLOPS of FP16 compute, most of which sits idle during decode.

Prefill is compute-bound; decode is memory-bandwidth-bound. Most production latency comes from decode.

This is why memory bandwidth per dollar is the single most important metric for inference GPU selection. Not FLOPS, not memory capacity (though capacity sets the floor for what models can load at all), but bandwidth.

The NVIDIA Datacenter Lineup

H100 SXM vs H100 PCIe

The H100 exists in two form factors with meaningfully different specs:

Spec	H100 SXM	H100 PCIe
HBM3 Capacity	80 GB	80 GB
Memory Bandwidth	3.35 TB/s	2.0 TB/s
FP16 TFLOPS	989	756
FP8 TFLOPS	1,979	1,513
TDP	700W	350W
NVLink	900 GB/s	None
Interconnect	NVSwitch	PCIe Gen5 x16
Typical cloud cost/hr	$2.50–3.50	$2.00–2.80

The SXM variant’s 67% higher memory bandwidth translates almost directly to 67% higher decode throughput for bandwidth-bound models. NVLink enables efficient tensor parallelism across multiple GPUs — critical for models too large to fit on a single card. The PCIe variant lacks NVLink, making multi-GPU setups reliant on PCIe, which is 14x slower for inter-GPU communication.

For single-GPU inference of models that fit in 80 GB, the PCIe variant is often the better value. For multi-GPU serving of large models, SXM is necessary.

H200

The H200 is an H100 with upgraded memory:

Spec	H100 SXM	H200 SXM
Memory	80 GB HBM3	141 GB HBM3e
Bandwidth	3.35 TB/s	4.8 TB/s
Compute	989 FP16 TFLOPS	989 FP16 TFLOPS

Same compute, 43% more bandwidth, 76% more capacity. The H200 can run a 70B FP16 model on a single GPU (140 GB fits in 141 GB) where the H100 requires two cards with tensor parallelism overhead. The bandwidth increase pushes single-GPU decode throughput for bandwidth-bound workloads up by roughly 40%.

Cloud availability for H200s increased substantially in early 2026, and on-demand pricing from major providers runs $3.50–4.50/hr per GPU.

A100

The A100 remains widely available and meaningfully cheaper than the H100:

Spec	A100 80GB SXM	A100 40GB PCIe
Memory	80 GB HBM2e	40 GB HBM2e
Bandwidth	2.0 TB/s	1.6 TB/s
FP16 TFLOPS	312	312
FP8	Not supported	Not supported
NVLink	600 GB/s	None
Cloud cost/hr	$1.50–2.20	$1.00–1.60

The A100 lacks FP8 support, which means quantized models that benefit from FP8 kernels on H100 have to fall back to FP16 or INT8 on A100. Memory bandwidth is 40% lower than H100 SXM. But at 40–60% lower cloud cost, the A100 delivers competitive inference cost-efficiency for models that fit in 80 GB, especially at lower concurrency.

L40S

The L40S occupies an interesting niche — Ada Lovelace architecture (same generation as RTX 4090) in a datacenter form factor:

Spec	L40S
Memory	48 GB GDDR6X
Bandwidth	864 GB/s
FP16 TFLOPS	362
FP8 TFLOPS	733
TDP	350W
Cloud cost/hr	$1.20–1.80

The L40S uses GDDR6X instead of HBM. This is the key tradeoff: 864 GB/s bandwidth vs 3.35 TB/s on H100 SXM — nearly 4x lower. For decode-heavy workloads, the L40S produces roughly 1/4 the tokens per second of an H100 at roughly 1/2 the cost. The math works out to about 2x worse cost-efficiency for pure decode.

Where L40S shines: prefill-heavy workloads (long prompts, short responses), batch inference where throughput matters more than latency, and any workload where 48 GB is sufficient and the lower per-hour cost matters more than per-token efficiency.

The B-Series Generation: B100 and B200

NVIDIA’s Blackwell generation (B100, B200, GB200) represents the largest generational leap in inference performance:

Spec	B200	B100	H100 SXM
Memory	192 GB HBM3e	192 GB HBM3e	80 GB HBM3
Bandwidth	8.0 TB/s	8.0 TB/s	3.35 TB/s
FP8 TFLOPS	~4,500	~3,500	1,979
FP4 TFLOPS	~9,000	~7,000	N/A
NVLink	1.8 TB/s	1.8 TB/s	900 GB/s

The 192 GB capacity means a 70B FP16 model fits on a single GPU with 52 GB to spare for KV cache. Memory bandwidth of 8 TB/s is 2.4x the H100, translating roughly to 2.4x decode throughput for bandwidth-bound models. FP4 support enables running models at 4-bit precision with hardware-accelerated kernels rather than software dequantization.

Cloud availability for B200s began in Q2 2026. Pricing is approximately $5.00–7.00/hr per GPU from major providers, though spot pricing can be substantially lower. The GB200 (two B200 dies + a Grace CPU on a single board with 384 GB unified HBM) targets the largest models and is primarily available in reserved/dedicated configurations.

The inference cost-efficiency improvement over H100 is roughly 2–3x per token when running FP8, and potentially 4x+ at FP4 precision (if model quality holds at FP4, which varies by architecture).

Consumer and Prosumer GPUs for Inference

RTX 4090

The RTX 4090 is the most popular GPU for self-hosted inference by individual developers and small teams:

Spec	RTX 4090
Memory	24 GB GDDR6X
Bandwidth	1,008 GB/s
FP16 TFLOPS	330
Purchase price	~$1,600 (used), ~$2,000 (new)

24 GB fits a 7B FP16 model (~14 GB) with room for KV cache, or a 13B model at Q4 quantization (~7–8 GB), or a 70B model at Q4 quantization across 3 cards (~24 GB per card using tensor parallelism over PCIe, which works but incurs latency overhead from PCIe interconnect).

The bandwidth of 1,008 GB/s is actually higher than the L40S (864 GB/s), making the 4090 faster for decode on a per-card basis. The limitation is 24 GB capacity and consumer-grade drivers that lack some enterprise features.

RTX 5090

The RTX 5090 (Blackwell consumer, launched early 2025) offers:

Spec	RTX 5090
Memory	32 GB GDDR7
Bandwidth	~1,792 GB/s
FP16 TFLOPS	~419
Purchase price	~$2,000–2,200

32 GB is a meaningful upgrade from 24 GB — enough to run a 13B FP16 model or a 30B Q4 model on a single card. The bandwidth increase to ~1,792 GB/s makes it competitive with the A100 80GB SXM (2.0 TB/s) for per-card decode throughput while costing a fraction per hour amortized.

RTX A6000 / RTX 6000 Ada

Spec	RTX A6000 (Ampere)	RTX 6000 Ada
Memory	48 GB GDDR6	48 GB GDDR6
Bandwidth	768 GB/s	960 GB/s
Purchase price	~$3,500 (used)	~$6,500

48 GB cards fit into the gap between consumer GPUs and datacenter A100s. The RTX 6000 Ada is essentially a workstation-class L40S with ECC memory. These cards are popular for on-premise inference where HBM-class bandwidth isn’t needed but 24 GB isn’t enough.

AMD and Intel: The Alternatives

AMD Instinct MI300X

The MI300X is AMD’s datacenter inference GPU:

Spec	MI300X	H100 SXM
Memory	192 GB HBM3	80 GB HBM3
Bandwidth	5.3 TB/s	3.35 TB/s
FP16 TFLOPS	1,307	989
Cloud cost/hr	$2.50–3.50	$2.50–3.50

On paper, the MI300X is superior in every category. In practice, the software ecosystem creates friction. ROCm (AMD’s CUDA equivalent) has improved substantially through 2025–2026, and vLLM, llama.cpp, and most major inference engines now support MI300X. But some quantization formats, custom kernels, and newer features (like certain FlashAttention variants) ship on CUDA first, sometimes months earlier.

For straightforward inference of popular model architectures (Llama, Mistral, Qwen), the MI300X offers excellent price-performance and that 192 GB capacity means many models that need 2x H100 fit on 1x MI300X. The MI325X (256 GB HBM3e) is shipping in limited quantities.

AMD Consumer: RX 7900 XTX

The RX 7900 XTX (24 GB GDDR6, ~960 GB/s, ~$800) is a viable consumer inference GPU through ROCm and llama.cpp. Software support is less polished than NVIDIA — expect to spend more time on setup and debugging. Performance is roughly comparable to an RTX 4090 for inference, at about half the price.

Intel Gaudi 3

Intel’s Gaudi 3 accelerator targets inference with 128 GB HBM2e and ~3.7 TB/s bandwidth. Cloud availability is limited (primarily AWS with dl2q instances). Software support goes through Habana’s SynapseAI stack and an Intel-maintained fork of vLLM. Pricing can be competitive where available, but the ecosystem is thin enough that most teams default to NVIDIA or AMD.

Apple Silicon for Inference

Apple’s M-series chips have unified memory architectures where CPU and GPU share the same memory pool:

Chip	Unified Memory	Bandwidth	GPU TFLOPS (FP16)
M4 Max	Up to 128 GB	546 GB/s	~53
M4 Ultra	Up to 512 GB	819 GB/s	~106
M3 Ultra	Up to 192 GB	800 GB/s	~89

The M4 Ultra with 512 GB unified memory can hold a 200B+ parameter model in memory — something that requires multiple datacenter GPUs otherwise. The bottleneck is bandwidth: 819 GB/s is less than an L40S and roughly 1/4 of an H100 SXM.

This translates to viable single-user inference speeds. A 70B Q4 model (~35 GB) on an M4 Ultra produces roughly 15–25 tokens/second via llama.cpp with Metal acceleration. Not fast enough for production multi-user serving, but perfectly adequate for local development, prototyping, and personal use.

The economics: an M4 Ultra Mac Studio (512 GB) costs approximately $8,000–10,000. A comparable cloud instance serving the same model runs $2–4/hr. The Mac Studio pays for itself in 2,500–5,000 hours of inference — about 4–8 months of continuous 24/7 operation, or 1.5–3 years at 8 hours/day.

Apple Silicon trades bandwidth for large unified memory capacity — a different optimization point than discrete GPUs.

Cloud GPU Pricing Comparison

Pricing varies substantially by provider, commitment level, and availability. These are representative on-demand rates as of mid-2026:

GPU	AWS	GCP	Azure	Lambda	CoreWeave
A100 80GB SXM	$3.97 (p4d.24xl / 8-GPU)	$2.21 (a2-highgpu-1g)	$3.40	$1.29	$1.28
H100 SXM	$4.76 (p5.48xl / 8-GPU)	$3.35 (a3-highgpu-1g)	$3.69	$2.49	$2.49
H200 SXM	—	$4.30	—	$3.29	$3.49
L40S	$1.72 (g6e)	$1.70	$1.84	$0.99	$1.14
B200	Limited	Preview	Preview	$4.99	$5.49

Prices are per-GPU-hour, on-demand, rounded. Multi-GPU instances (e.g., AWS p5.48xl has 8x H100) are divided by GPU count. Rates change frequently.

Several patterns emerge:

Hyperscalers (AWS, GCP, Azure) charge 30–60% more than GPU-focused providers (Lambda, CoreWeave, Vast.ai) for equivalent hardware. The premium buys ecosystem integration, reliability, and compliance certifications.
1-year reserved instances typically save 30–40% vs. on-demand. 3-year reservations save 50–60%. These commitments make sense for steady-state production workloads.
The real cost is often not the GPU. CPU, memory, networking, and storage add 20–40% on top of GPU-hour pricing on hyperscalers. GPU-focused providers tend to include more reasonable compute/memory ratios.

Spot and Preemptible Instances

Spot instances offer 60–90% discounts with the risk of interruption:

Provider	GPU	Spot Price/hr	Discount vs On-Demand	Typical Availability
AWS	H100 SXM	~$1.50–2.00	60–70%	Moderate (varies by region)
GCP	A100 80GB	~$0.80–1.10	50–65%	Good
Lambda	H100 SXM	~$1.50	40%	Good
RunPod	H100 SXM	~$1.50–2.00	40–50%	Variable
Vast.ai	Various	Market-based	50–80%	Good for consumer GPUs

Spot instances work well for:

Batch inference (processing a queue of requests where interruption just means retry)
Development and testing
Workloads with checkpointing and automatic restart logic

They work poorly for:

Production serving with latency SLAs
Long-running inference sessions without state checkpointing

A common pattern: use on-demand or reserved instances for baseline production load, with spot instances autoscaling for burst capacity. Most inference engines (vLLM, TGI) can restart and reload models within 30–60 seconds, making spot viable for non-latency-critical production if the orchestration handles failover.

Model-to-GPU Mapping: What Fits Where

The first-order question: does the model fit in GPU memory? Model memory footprint is straightforward to calculate:

Memory ≈ (Parameters × Bytes per Parameter) + KV Cache + Overhead

For weights alone:

Model Size	FP16 (2 bytes)	FP8 (1 byte)	Q4 (~0.5 bytes)
7–8B	14–16 GB	7–8 GB	4–5 GB
13–14B	26–28 GB	13–14 GB	7–8 GB
27–35B	54–70 GB	27–35 GB	14–18 GB
70–72B	140–144 GB	70–72 GB	35–38 GB
200B+	400+ GB	200+ GB	100+ GB

KV cache adds memory proportional to batch size × sequence length × number of layers × head dimension. For a 70B model serving 32 concurrent requests at 4K context, KV cache adds roughly 8–16 GB (varies by architecture; models with GQA like Llama use less).

Practical GPU assignments:

Model size determines GPU tier. Quantization shifts everything down by one tier.

Concrete examples:

Model	Precision	GPU Config	Approx. Throughput
Mistral Small 4 (24B MoE, ~4B active)	FP16	1x RTX 4090	40–60 tok/s
Qwen3.6-27B	Q4	1x RTX 4090	20–30 tok/s
Llama 4 Maverick (400B MoE, ~17B active)	FP8	8x H100	25–40 tok/s
Mistral Large 3 (675B MoE, ~41B active)	FP8	8x H100	15–25 tok/s
DeepSeek V4-Flash (284B MoE, ~13B active)	FP8	4x H100	20–35 tok/s

Throughput numbers are per-user for a single concurrent request. Production throughput with batching is much higher but latency increases.

Quantization and Its Effect on Hardware Requirements

Quantization reduces model precision, shrinking memory footprint and increasing decode speed (fewer bytes to read from memory per token). The quality-efficiency tradeoff depends on the model and the quantization method.

Method	Bits	Memory Reduction	Typical Quality Impact	Hardware Support
FP16	16	Baseline	None	All GPUs
FP8	8	2x	Minimal (<1% on most benchmarks)	H100, B200, RTX 5090
INT8 (W8A8)	8	2x	Minimal	A100+, consumer GPUs
GPTQ/AWQ Q4	4	4x	Small (1-3% on benchmarks)	All GPUs (software decode)
GGUF Q4_K_M	4	4x	Small	CPU + GPU (llama.cpp)
FP4	4	4x	Small-moderate	B100/B200 only
Q2/Q3	2-3	5-8x	Moderate-large	All (software)

FP8 on H100/B200 is essentially free quality — hardware FP8 tensor cores run at 2x the FLOPS of FP16 with negligible quality loss for most models. This is why the H100’s FP8 support is so important for inference economics.

GPTQ and AWQ (weight-only quantization to 4-bit) provide excellent quality-to-compression ratios for most models above 7B parameters. Smaller models (< 7B) are more sensitive to 4-bit quantization. The GGUF format used by llama.cpp supports mixed quantization levels within a model (e.g., attention layers at higher precision, FFN layers at lower) which can improve quality over uniform quantization.

Quantization ladder: each step halves memory with incrementally more quality risk.

Self-Host vs API: The Decision Framework

The self-host vs API decision has a clean quantitative framework, but qualitative factors often dominate.

When APIs win:

Low to moderate volume (< $2,000–5,000/month in API costs)
Need for frontier model quality (GPT-5.6 Sol, Claude Opus 5, Gemini 3.7 Flash are not available for self-hosting)
Variable, bursty traffic (APIs handle autoscaling; self-hosted GPUs sit idle during low periods)
Compliance requirements met by the provider (SOC 2, HIPAA BAAs from OpenAI/Anthropic/Google)
Rapid iteration on model selection (switching between providers/models is an API call change, not a re-deployment)

When self-hosting wins:

High, steady volume (> $5,000–10,000/month on API costs, with consistent utilization)
Data sovereignty requirements that prohibit sending data to third parties
Latency requirements that API providers can’t meet (self-hosted can achieve < 50ms TTFT; API providers typically 200–800ms)
Customization needs (custom tokenizers, fine-tuned models, specific decoding strategies)
Cost optimization at scale — the economics favor self-hosting above a certain volume threshold

Factors that don’t cleanly favor either:

Reliability: API providers have outages. Self-hosted infrastructure has hardware failures. Both require redundancy planning. API providers have experienced outages; both self-hosted and cloud-based systems require redundancy planning.
Model quality: Open models like Kimi K3, DeepSeek V4, Mistral Large 3, and Qwen3.6 match or exceed older frontier models. But the newest frontier models (GPT-5.6 Sol, Claude Opus 5, Claude Fable 5) remain ahead on hard reasoning tasks.

The Cost Crossover Math

The crossover calculation: at what monthly API spend does self-hosting become cheaper?

API cost model:

Monthly cost = (input_tokens × input_price) + (output_tokens × output_price)

Self-hosted cost model:

Monthly cost = (gpu_hours × hourly_rate) + ops_overhead

The ops_overhead term is real and often underestimated. Budget 20–40% on top of raw GPU costs for:

Engineering time managing inference infrastructure
Monitoring, alerting, and on-call
Model updates and deployment pipeline maintenance
Redundancy (at minimum 2x GPUs for failover)

Example calculation for a Llama 4 Maverick deployment:

# API route: using a provider that serves Llama 4 Maverick
# Hypothetical pricing: $0.50/M input, $1.50/M output
monthly_input_tokens = 500_000_000   # 500M input tokens
monthly_output_tokens = 100_000_000  # 100M output tokens
api_cost = (500 * 0.50) + (100 * 1.50)  # $400/month

# Self-hosted route: 8x H100 node on Lambda
# Maverick needs ~400 GB for weights at FP8 -> 8x H100 80GB (640 GB)
gpu_hourly = 2.49 * 8                # $19.92/hr for the node
gpu_monthly = gpu_hourly * 24 * 30   # $14,342/month
ops_overhead = gpu_monthly * 0.30    # $4,303/month
self_hosted_cost = gpu_monthly + ops_overhead  # $18,645/month

At 500M input + 100M output tokens/month, the API is dramatically cheaper ($400 vs $18,645). The crossover happens somewhere around 3–5B output tokens/month for this model, assuming ~80% GPU utilization on the self-hosted side.

# Crossover calculation
# API: (input_M * $0.50) + (output_M * $1.50) = self_hosted_monthly
# Assume 5:1 input:output ratio
# (5x * $0.50) + (x * $1.50) = $18,645
# $4.00 * x = $18,645
# x ≈ 4,661M output tokens/month → ~23.3B input tokens/month

# With a more expensive API model (e.g., $3/M input, $15/M output):
# (5x * $3.00) + (x * $15.00) = $18,645
# $30 * x = $18,645
# x ≈ 622M output tokens → much lower crossover

The crossover is highly sensitive to which API model is being compared. Self-hosting a 70B open model vs. calling Claude Sonnet 5 at $2/$10 per M has a much lower crossover point than self-hosting that same model vs. calling a cheap API endpoint.

Volume-based heuristic for the API vs self-host decision. The crossover point shifts based on API pricing and GPU utilization.

Multi-GPU and Tensor Parallelism

When a model doesn’t fit on a single GPU, tensor parallelism (TP) splits the model across multiple GPUs. Each GPU holds a slice of every layer, and GPUs communicate intermediate activations during inference.

The critical constraint is inter-GPU bandwidth. Every forward pass requires all-reduce operations proportional to the number of TP ranks.

Interconnect	Bandwidth	Typical Latency per All-Reduce
NVLink (H100 SXM)	900 GB/s	10–20 μs
NVLink (B200)	1.8 TB/s	5–10 μs
PCIe Gen5 x16	64 GB/s	50–100 μs
PCIe Gen4 x16	32 GB/s	100–200 μs
Ethernet (RoCE)	100–400 Gbps	1–10 ms

TP across NVLink is nearly lossless — the overhead is small relative to the compute time per layer. TP across PCIe adds meaningful latency per token (an extra 2–10ms per token for a 70B model split across 2 GPUs over PCIe Gen4). TP across network links is prohibitively slow for interactive inference; pipeline parallelism (PP) is used instead, where different layers live on different GPUs and requests are pipelined through them.

Practical GPU counts for common models:

Model	Precision	Min GPUs (H100 80GB)	Recommended GPUs
Llama 4 Scout (109B total, 17B active)	FP8	2	2
Llama 4 Maverick (17B active, 400B total)	FP8	4–8	8
Mistral Large 3 (675B MoE)	FP8	8	8
DeepSeek V4-Pro or DeepSeek V4-Flash (MoE, ~200B+ active in Pro variant)	FP8	4–8	8
Kimi K3 (2.8T)	FP8	16+	32

MoE (Mixture of Experts) models complicate this because total parameter count determines memory needed to load the model, but active parameter count determines compute per token. Maverick has 400B total params (~400 GB FP8) but only ~17B active per token — it needs 4–8 H100s for capacity but the per-token compute is modest.

Inference Engines and Their Hardware Preferences

The inference engine mediates between the model and the hardware. Engine choice affects throughput by 2–5x for the same hardware.

Engine	Best Hardware	Key Features	Primary Use
vLLM	NVIDIA datacenter (H100, A100), AMD MI300X	PagedAttention, continuous batching, speculative decoding, FP8	Production serving
TensorRT-LLM	NVIDIA only	Compiled kernels, maximum NVIDIA optimization	Highest throughput on NVIDIA
llama.cpp (via llama-server)	Any (CPU, NVIDIA, AMD, Apple Silicon)	GGUF format, CPU+GPU hybrid, broad hardware support	Local/edge inference
SGLang	NVIDIA datacenter	RadixAttention (prefix caching), constrained decoding	High-throughput with structured output
Ollama	Any (wraps llama.cpp)	Easiest setup, model management	Developer experience
ExLlamaV2	NVIDIA consumer GPUs	Optimized GPTQ/EXL2 kernels	Consumer GPU inference

Inference engines optimized for different hardware targets and use cases.

vLLM is the default choice for most production deployments on NVIDIA and AMD GPUs. PagedAttention eliminates memory fragmentation in KV cache management, enabling higher batch sizes. Continuous batching ensures the GPU stays utilized even with variable-length requests. vLLM supports FP8 quantization on H100+, AWQ/GPTQ weight quantization, and speculative decoding.

TensorRT-LLM (NVIDIA’s own engine) compiles models into optimized execution plans with fused kernels. It typically achieves 10–30% higher throughput than vLLM on the same NVIDIA hardware, at the cost of longer compilation times and less flexibility. Worth the effort for high-volume production on NVIDIA GPUs.

llama.cpp excels on heterogeneous hardware. It can split inference between CPU and GPU (offloading some layers to GPU, keeping others in system RAM), supports Apple Metal, Vulkan (AMD/Intel GPUs), and runs on ARM CPUs. The GGUF format with mixed-precision quantization (e.g., Q4_K_M) provides the best quality per bit of any quantization format. For Apple Silicon specifically, llama.cpp with Metal is the highest-performing option.

SGLang introduces RadixAttention, which caches KV states for shared prompt prefixes across requests. For workloads with common system prompts or RAG contexts, this can reduce compute by 50–90% for the shared prefix portion.

A benchmark comparison on the same H100 SXM for a 70B-class dense open model at FP8:

Engine	Throughput (tok/s, batch=32)	Median TTFT (ms)	Median TPOT (ms)
vLLM	~2,400	~180	~28
TensorRT-LLM	~2,900	~150	~23
SGLang	~2,500	~120 (with prefix caching)	~27

Approximate benchmarks — actual results depend on model architecture, sequence length, and batch composition. TPOT = time per output token.

Practical Configuration Examples

Small Team / Startup (< $1,000/month budget)

Option A: API-only Use GPT-5.6 Luna ($0.20/$1.20 per M) or Gemini 3.7 Flash ($0.75/$3.75 per M) for production. Use DeepSeek V4-Flash or Mistral Small 4 for high-volume, lower-stakes tasks.

Option B: Single consumer GPU + API fallback

1x RTX 4090 or RTX 5090 ($1,800–2,100 one-time)
Run Mistral Small 4 or Qwen3.6-27B locally via Ollama or vLLM for development and low-latency needs
Fall back to API for complex reasoning tasks

# Model routing pseudocode
def route_request(request):
    if request.requires_reasoning or request.complexity == "high":
        return call_api("claude-sonnet-5")  # API for hard tasks
    else:
        return call_local("qwen3.6-27b-q4")  # Local for routine tasks

Mid-Scale Production (10M–100M tokens/month)

Option A: Reserved cloud GPU

1x H100 SXM on Lambda or CoreWeave (~$2.49/hr → ~$1,793/month)
Run Qwen3.6-27B at FP8, or Llama 4 Scout at Q4 (~55 GB), via vLLM
This handles ~50–100M output tokens/month at reasonable latency

Option B: Hybrid API

Use Gemini 3.7 Flash for most requests ($0.75/$3.75 per M — at 50M output tokens that’s ~$188/month)
Route complex tasks to Claude Opus 5 or GPT-5.6 Sol
Total cost likely $500–2,000/month depending on mix

At this scale, the API route is usually still cheaper unless data sovereignty or latency requirements force self-hosting.

High-Scale Production (1B+ tokens/month)

8x H100 SXM cluster (reserved 1-year: ~$1.50–2.00/hr per GPU → ~$8,600–11,500/month total)
Run largest open models (Kimi K3, DeepSeek V4) with tensor parallelism
Use vLLM or TensorRT-LLM with continuous batching
At 1B output tokens/month, equivalent API costs would be $1,500–15,000+ depending on model choice
Self-hosting likely breaks even or saves money at this scale, especially with reserved pricing

Hardware Trends and What to Buy Now

A few trends that affect purchasing decisions in August 2026:

B200 availability is improving but not abundant. Spot B200 instances are appearing on Lambda, CoreWeave, and GCP. For new deployments at scale, B200 offers roughly 2.5x the inference performance per dollar of H100 — making H100 reserved commitments a questionable investment for workloads that can wait 1–3 months for B200 availability.

H100 pricing is declining. As B200 supply increases, H100 spot and on-demand prices are falling. Expect H100 SXM spot to stabilize around $1.00–1.50/hr, making it increasingly competitive for inference workloads that don’t need B200-class performance.

Consumer GPUs keep getting more capable. The RTX 5090’s 32 GB and ~1.8 TB/s bandwidth make it a genuine inference workhorse. Future consumer GPU generations with larger memory pools would further blur the line between consumer and professional cards for inference.

MoE models favor memory capacity over bandwidth. As more frontier-class open models adopt MoE architectures (Llama 4 Maverick, Mistral Large 3, DeepSeek V4), the ability to hold all expert weights in memory becomes more important than raw bandwidth. This favors GPUs with larger memory (MI300X at 192 GB, H200 at 141 GB, B200 at 192 GB) even if bandwidth per dollar isn’t optimal.

API pricing continues to fall. Gemini 3.7 Flash at $0.75/$3.75 per M with introductory pricing, GPT-5.6 Luna at $0.20/$1.20, and DeepSeek V4-Flash at competitive rates mean the API cost crossover keeps moving to higher volumes. For many workloads, the breakeven point for self-hosting is now billions of tokens per month rather than hundreds of millions.

Summary

GPU selection for LLM inference reduces to four variables: memory capacity (does the model fit?), memory bandwidth (how fast can tokens be generated?), cost per GPU-hour, and GPU utilization (how many hours per day will it actually run?).

Memory bandwidth — not FLOPS — is the binding constraint for autoregressive decode. The H100 SXM’s 3.35 TB/s makes it the current workhorse. The B200’s 8 TB/s is 2.4x faster per card. Consumer GPUs (RTX 4090 at 1 TB/s, RTX 5090 at ~1.8 TB/s) offer surprisingly competitive per-card decode speed at a fraction of the cost, limited by memory capacity.

The API vs self-host decision is primarily about volume and utilization. Below ~$2,000/month in API spend, APIs almost always win. Above ~$5,000/month with consistent utilization, self-hosting starts to make economic sense, especially for open models served on reserved GPU instances.

Quantization shifts every model down one GPU tier. FP8 on H100+ is free performance. Q4 quantization (GPTQ, AWQ, GGUF) enables 70B models on single consumer GPUs with 1–3% quality loss.

The inference engine matters as much as the hardware. vLLM is the default for production. TensorRT-LLM extracts maximum NVIDIA performance. llama.cpp runs everywhere. Pick the engine that matches the target hardware and operational requirements.