Office Hours — What minimum hardware specs do you actually need to run capable open-source LLM models locally for agent development?

What minimum hardware specs do you actually need to run capable open-source LLM models locally for agent development?

The answer depends entirely on what “capable” means for your workload. If you mean “runs without hanging,” most laptops work. If you mean “produces reliable agent outputs at acceptable latency,” the hardware requirements jump significantly. Let me ground this in reality.

The Sweet Spot: 24GB VRAM

For agent development on commodity hardware, aim for 24GB of VRAM. This is the threshold where you can run models like Qwen3.8-27B, Llama 4 Maverick, or Mistral Large 3 at reasonable speeds without aggressive quantization degrading output quality. An RTX 4090 or RTX 6000 Ada hits this. So does an M4 Max with unified memory, though Apple’s silicon has different throughput characteristics.

At 24GB, you’re not stuck with toy models. The Qwen3.8-27B released in November 2026 runs locally and handles genuine multi-step agent tasks: tool calling, reasoning chains, debugging, documentation. It’s fast enough that you don’t start second-guessing whether to submit a task to the model or just do it manually.

Below 24GB: Brutal Tradeoffs

With 16GB (like a consumer RTX 4060 Ti or base M3 Max), you hit a ceiling fast. You can load Mistral Small 4 or smaller dense models, but quantization to 4-bit becomes mandatory. At 4-bit, models lose precision on nuanced reasoning and tool calling gets flaky. Agent reliability drops noticeably. You’re not blocked, but you’ll spend time debugging whether failures are your agent harness or the model starving for precision.

8GB maxes out around a 7B model running 4-bit. Fine for Q&A or classification. Barely functional for agent work. Agents need enough capacity to reason through multi-step plans, track tool responses, and decide what to do next. At 8GB you’re constantly hitting context limits and eviction decisions become painful.

The CPU/Memory Bandwidth Problem Nobody Mentions

VRAM size isn’t the only bottleneck. If your model fits in VRAM but your CPU-to-GPU bandwidth is throttled, you’ll wait. A lot. This matters more than people realize.

Local inference on a laptop with a GPU bottlenecked to 8x PCIe (common on older hardware) will be slower than a cloud API call for anything but the smallest models. You’re paying the latency tax without the scale economics. If you’re developing an agent locally, this makes iteration painful.

Compare:

Desktop RTX 4090 (GPU-to-PCIe bottleneck, but CPU cores abundant): inference at 50–120 tokens/sec for a 27B model at 4-bit
Macbook M4 Max (unified memory, no PCIe transfer penalty): same model at 40–80 tokens/sec
Older laptop with RTX 4060 Mobile (severe bandwidth limit): same model at 10–25 tokens/sec

The third one makes agent development tedious. Plan accordingly.

Disk and Setup

You also need fast storage. A 27B model at 4-bit is roughly 16–20GB on disk. Loading it from a spinning disk into VRAM will block for 30–60 seconds on first load. Use NVMe. M.2 makes iteration tolerable. USB external SSDs are slower than you think for this use case.

What “Capable” Actually Means for Agents

Capable for agent work means the model can:

Call tools correctly (parsing function arguments from unstructured output)
Retry or self-correct when a tool fails
Maintain context across 5–10 sequential actions before hallucinating
Generate code that’s more likely to work than not

At 24GB you get most of this. At 16GB with aggressive quantization you lose error correction and retry logic gets noisy. Below that, agent tasks fail often enough that you’ll spend more time debugging than building.

Practical Config Example

Here’s a realistic local setup for agent dev:

# Using ollama + llama.cpp
# Hardware: RTX 4090 (24GB), 64GB system RAM, NVMe

ollama pull qwen:32b-q4_k_m  # Qwen 3.8-27B, 4-bit
# File size: ~18GB, loads in ~15 seconds

# Agent harness in Python
from ollama import Client
import json

client = Client()

def agent_step(task, context):
    response = client.generate(
        model="qwen:32b-q4_k_m",
        prompt=f"Task: {task}\nContext: {context}",
        stream=False,
    )
    return response["response"]

# First call: ~3-5 seconds (model already in VRAM)
# Subsequent calls: ~100-200ms overhead (tokenization + queuing)

This setup lets you iterate on agent logic in real time without waiting for API responses or paying per token. You’re bottlenecked by model inference speed, not external APIs.

When Local Doesn’t Make Sense

If you need:

Models larger than 70B (Llama 4 Scout needs 200GB+ VRAM)
Multi-GPU parallelism (expensive to set up, requires infrastructure)
Fast iteration on changing model weights (use cloud services with preemption)

Then you’re back to cloud APIs or rented cloud GPUs. The economics flip: $5 for an A100-backed inference service beats $2,500 in hardware if you’re not using it constantly.

Real-World Estimate

For solo development or a small team:

One-time hardware cost: $2,000–$4,000 for a 24GB discrete GPU or modern Mac
Operating cost: electricity (negligible for occasional use)
Iteration speed: 50–100ms overhead per agent step once the model is loaded
Break-even point: Pay for itself in 2–3 months of heavy local use versus API calls

Bottom line: If you’re building agents locally and can afford it, get 24GB VRAM. Anything less creates enough friction that you’ll either switch to cloud APIs (costing more long-term) or ship unreliable agents because you didn’t iterate enough to catch failures.

Question via Hacker News