Office Hours — What's the best open-source LLM you can run on a 24GB VRAM GPU for production workloads?

What’s the best open-source LLM you can run on a 24GB VRAM GPU for production workloads?

You’ve got a solid constraint—24GB VRAM is the sweet spot where you can run meaningful models without enterprise hardware. The answer depends on what “production workload” means for you, but the honest take is that no single model dominates this tier universally.

The Contenders

Llama 4 Scout (10M context) fits comfortably at 24GB in 4-bit quantization and handles general tasks well. It’s mature enough that you’ll find extensive deployment guides, but it’s not specialized for anything specific—think of it as the practical baseline that doesn’t surprise you.

Mistral Large 3 (675B MoE) is the heavyweight, but here’s the catch: sparse mixture-of-experts means only 38B parameters activate per token, so it fits. The problem is you need careful quantization and good inference infrastructure. It’s capable, but the operational complexity often isn’t worth it unless you’re extracting serious performance gains you can’t get elsewhere.

Devstral 2 (72.2% SWE-bench) makes sense if you’re primarily doing code generation or retrieval-augmented generation over software. It’s explicitly optimized for that task class and outperforms much larger base models at it. Fits at 24GB in 4-bit.

Qwen3 from Alibaba handles long context (1M tokens with careful management) and is genuinely solid at multimodal tasks if your workload involves images or mixed modality. At 24GB you’ll run the smaller variants, but they’re surprisingly capable.

Why This Matters for Production

The real constraint isn’t fitting the model into VRAM—it’s fitting the model plus batch throughput plus caching plus safety margin. A 24GB GPU running at 100% utilization on one inference request gives you terrible latency and no ability to absorb variance. In practice, plan for 60–70% actual utilization to avoid OOM on spikes.

Here’s a concrete example: if you’re running Llama 4 Scout at 24GB with 4-bit quantization, you get roughly 400–500 tokens/second throughput on a single request. But if you need to handle concurrent requests, context window management becomes the real bottleneck. Each request keeps its context in memory, so two simultaneous 8K-token requests will exhaust your GPU fast.

A Working Config

Use vLLM with continuous batching and 4-bit AWQ quantization on Llama 4 Scout:

python -m vllm.entrypoints.openai_compatible_server \
  --model meta-llama/Llama-4-Scout-8B \
  --quantization awq \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.7 \
  --max-model-len 8192 \
  --dtype half

This pins GPU utilization at 70%, leaves breathing room for context, and keeps inference latency under 100ms for typical prompts. The --max-model-len 8192 is intentional—overcommitting context window kills production reliability worse than it helps throughput.

The Tradeoff You Need to Accept

Open-source models at 24GB are production-grade but not frontier-grade. Llama 4 Scout is a competent retrieval-augmented generation backbone and handles most classification tasks fine. It struggles with nuanced reasoning tasks where Claude Opus 4.8 or GPT-5.5 would succeed. If your workload is “extract structured data from documents” or “classify customer support tickets,” you’re good. If it’s “reason through ambiguous architectural decisions,” you’ll hit walls.

If your primary constraint is sovereignty or latency (on-device, no API calls), this is absolutely the right tier. If cost is the constraint but you have API access, running locally might actually cost more in electricity and engineering overhead than using a frontier API with cheaper batch processing.

Bottom line:

Start with Llama 4 Scout in 4-bit quantization on vLLM at 70% GPU utilization. It’s battle-tested, fits 24GB cleanly, and handles 80% of production retrieval and classification workloads without surprise failures. Switch to Devstral 2 if you’re doing serious code generation, or run a hybrid pattern where local Scout handles retrieval and a frontier API handles the reasoning pass.

Question via Hacker News