Office Hours — What's your recommended programming setup and workflow for building LLM applications without getting caught up in hype?

What’s your recommended programming setup and workflow for building LLM applications without getting caught up in hype?

Start with a boring stack

Use what works and is well-understood. For most LLM applications, that means a scripting language (Python), a lightweight orchestration library (LangGraph or similar), and a database (Postgres). Don’t use specialized “AI frameworks” unless you’ve already hit a real wall with the basics. The flashiest frameworks often disappear in 18 months, but a well-structured Python script with explicit function calls and clear data flow will still run in five years.

Pick one model provider and one model tier, then lock that choice for at least a quarter. Don’t chase every benchmark release. GPT-5.6 Sol and Claude Opus 5 are both solid—pick one, understand its failure modes, and build around it. The difference between them matters less than you think. What matters is consistency. When you switch models mid-project to save 20% on tokens, you’re going to spend three weeks retuning prompts and dealing with subtle behavioral differences.

Build evals first, not prompts

Before you touch a prompt, write a test harness. Take 20 examples of the task you’re solving—real ones from your domain, not synthetic—and score them manually. Load these into a simple evaluation script that can run your system against them and show you pass/fail rates and false positive/false negative breakdowns. This takes an afternoon and saves weeks of vibe-checking.

Your evals script should output a CSV, not a dashboard. You want to see failures line by line so you can notice patterns. When Claude suddenly drops in accuracy after you update a prompt, your evals catch it immediately instead of in production.

Use structured outputs, but verify the structure

Frontier models like GPT-5.6 Sol and Claude Opus 5 now support structured outputs (JSON schemas, TypeScript interfaces). Use them. They’re faster and more reliable than parsing free-text responses. But don’t trust the structure blindly.

# Example: use structured output, then validate
from typing import TypedDict
import json

class ExtractedData(TypedDict):
    invoice_id: str
    amount: float
    vendor_name: str

# Get structured response from Claude
response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    messages=[...],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "ExtractedData",
            "schema": ExtractedData.__annotations__
        }
    }
)

# Parse and validate
try:
    data = json.loads(response.content[0].text)
    amount = float(data["amount"])  # Can still fail
    if amount < 0 or amount > 1_000_000:
        # Flag for review, don't assume it's right
        log_anomaly(data)
except (json.JSONDecodeError, ValueError, KeyError) as e:
    log_parse_error(e, response.content[0].text)

Structured outputs reduce parsing errors by ~90%, but they don’t eliminate them. The model can still hallucinate a vendor name or misparse a currency. Your validation layer is not negotiable.

Cache aggressively, but measure the tradeoffs

Both Claude Opus 5 and GPT-5.6 Sol support prompt caching. If you have a stable context (system prompt, company docs, codebase reference) that you reuse across many requests, cache it. The cost per cached token is roughly 10% of the base cost, so the math is straightforward: if your cached context is more than 1,000 tokens and you’ll reuse it more than 10 times, caching pays for itself.

But measure your end-to-end latency. Cached requests sometimes have higher latency on first call (the cache setup overhead), and that matters if you’re building something latency-sensitive. For batch jobs, cache everything. For real-time user-facing tasks, test it first.

Keep humans in the loop where it actually matters

AI agents are real and autonomous in controlled environments like code generation with test suites. Outside of that, keep a human approval gate for anything that touches money, user data, or production systems.

The pattern that works: AI agent executes the task, produces a summary or diff, human reviews in <30 seconds (because the summary is clear), human clicks “approve.” This is not a bottleneck if your summaries are good and your approval UI is fast. Most teams fail here by generating incomprehensible diffs or burying the approval step in email.

Measure token costs upfront

Don’t guess. Benchmark your actual usage before you ship. Run your workflow 100 times with your chosen model, measure the input and output tokens, multiply by the per-token cost, and see if you can defend that number to your finance team.

Multi-agent systems are particularly easy to mess up: if you’re orchestrating three sub-agents, each one calling a model, and each one passing context to the next, you’re duplicating tokens. Token economics often break multi-agent systems before capability issues do.

# Log and sum token usage per request
total_input_tokens = 0
total_output_tokens = 0

for call in model_calls:
    total_input_tokens += call.usage.input_tokens
    total_output_tokens += call.usage.output_tokens
    
cost = (total_input_tokens * input_price + 
        total_output_tokens * output_price)
print(f"Request cost: ${cost:.4f} ({total_input_tokens + total_output_tokens} tokens)")

Do this for every change that might affect token usage. Prompt compression, context windowing, and multi-step workflows all have hidden costs.

Pick local-first or cloud, then commit

If you’re building something that touches customer data, has compliance requirements, or needs to work offline, run an open-weight model locally. Alibaba’s Qwen3.8-Flash-Next or Mistral Large 3 are solid choices. Accept that you’ll spend time on quantization, vLLM setup, and GPU provisioning. But you’ll own the latency and data flow completely.

If you’re building consumer-facing features or need reliability at scale, use API providers. Don’t try to hybrid it unless you have a very specific reason (failover, model switching). Hybrid setups add complexity that usually isn’t worth the flexibility gains.

Don’t fine-tune unless you’ve exhausted everything else

Fine-tuning is seductive because it feels like “owning” the model. In practice, for most LLM applications in 2026, prompt engineering, RAG (retrieval-augmented generation), and agentic patterns solve the problem faster and cheaper than fine-tuning. Fine-tuning makes sense if you have a high-volume, repetitive task where you can measure a clear ROI on the training investment. Otherwise, you’re adding infrastructure you don’t need.

Monitor what actually breaks

Most LLM applications fail silently. The model produces plausible-looking garbage that passes basic syntax checks. Set up monitoring for:

1. Parsing errors (structured output can’t be decoded)
2. Validation failures (the output fails your domain rules)
3. Semantic drift (the output is syntactically correct but semantically wrong; this is harder but worth sampling manually every week)

Track these rates per model, per prompt version. When a new model drops and you consider upgrading, run it side-by-side for a week and compare these metrics, not just benchmark scores.

Bottom line: Build with a single model on a single provider, write explicit evals before tweaking prompts, measure token costs on real data, and keep expensive operations (money, compliance, production systems) under human review. You don’t need exotic infrastructure—you need clear failure detection and the discipline to measure before you optimize.

Question via Hacker News