Office Hours — What are the practical differences in building real-world AI agent systems versus the simplified examples in documentation?
A daily developer question about AI/LLMs, answered with a direct, opinionated take.
What are the practical differences in building real-world AI agent systems versus the simplified examples in documentation?
Documentation shows you a loop: think, act, observe. Real production systems spend 80% of effort on the 20% of cases that break that loop, and you won’t see those cases until customers hit them.
The Gap Between Tutorial and Production
Tutorial agents operate in clean environments with clear success signals. A coding agent runs a test, the test passes or fails, next step is obvious. Real systems don’t work that way. You’re routing between models based on cost, dealing with APIs that timeout inconsistently, managing context windows that fill up mid-task, and deciding when to escalate to a human because the agent’s confidence is high but its reasoning is circular.
The biggest shock for most teams is that reliability doesn’t scale linearly with model quality. A better model helps, but the infrastructure around it matters more. You can take GPT-5.4 and Claude Opus 4.8 and get wildly different reliability depending on how you structure retries, caching, error handling, and feedback loops. This is what the recent Daily Signal piece on “tail control” captured: you’re not optimizing for average latency or cost, you’re optimizing for the 99th percentile behavior because customers only remember when something breaks.
Cost and Economics Get Brutal Fast
A well-intentioned team at Coinbase thought they could optimize by routing complex queries to GPT-5.5 and simple queries to a cheaper model. Three months in, their bill had grown 4x because the routing heuristics were wrong and cost optimization itself became a bottleneck. They ended up switching to Chinese models (GLM, Kimi) with stricter routing and caching. That’s not in the docs.
The economics math changes when you move from “let’s add an AI feature” to “let’s run an autonomous agent for 8 hours a day.” A single agent task that chains 15 model calls with retries can cost $0.50-$2 per execution. If 30% of executions fail and need to retry, you’re not paying for 15 calls, you’re paying for 19-20. If you’re processing 10,000 tasks a day, that’s $5,000-$20,000 daily before you’ve optimized anything.
Real teams solve this with multi-model routing: use GPT-4.1 Nano or Gemini 3.1 Flash-Lite for classification and filtering, reserve frontier models for tasks that actually need them, and layer in caching aggressively. But this requires building routing logic that understands task difficulty, which means labeling your own data or running continuous A/B tests to figure out which model class handles which task type. That’s weeks of work that never appears in examples.
Context Management and State Become Architectural Problems
An agent that runs for 30 minutes and makes 50 decisions can’t keep all 50 decisions in a single prompt. Context windows fill up. You have to choose between summarizing (which loses information), truncating (which forgets context), or switching to a different model with a larger window (which changes behavior and costs more).
Real systems build context compression layers that aren’t in documentation. You might use Claude Opus 4.8 to summarize the first 20 steps, then feed that summary to the next 30 steps, but now you’ve added a dependency and latency. Or you use a vector database to store decision history and retrieve only relevant context, but now you’ve added search latency and the risk of retrieving wrong decisions. Or you implement a state machine that limits what the agent can even consider at each step, which reduces autonomy but increases reliability.
The Daily Signal piece on “context window management for long-running agents” nailed this: there’s no one right answer. Each strategy (summarization, chunking, state machine, retrieval-augmented memory) has different latency, cost, and capability profiles. You have to pick based on your actual workload, and then monitor whether your choice is working.
Verification and Feedback Loops Are Friction Points
Documentation shows agents making decisions and moving forward. Production agents get stuck. You ask the model to write code and it generates plausible-looking but broken code. You ask it to extract data from a PDF and it hallucinates fields. The agent doesn’t know it’s wrong because there’s no signal to correct it.
The teams that work build explicit verification layers. After each step, they run a check: did the code compile? Does the extracted data match the schema? Is the decision consistent with prior decisions? If verification fails, they have a fallback: retry with more context, escalate to a human, or roll back the action.
This creates a new engineering problem: what counts as verification? If you require human review on everything, you’ve built a chatbot, not an agent. If you require zero human review, you’ll ship broken decisions. Most teams end up with tiered escalation: auto-retry on clear failures, alert humans on ambiguous cases, log everything.
Autonomous Action Has Guardrails, Not Autonomy
The most successful agents in production (coding agents like Claude Code, Cursor, Devin) don’t actually run autonomously. They execute in bounded environments where failure is recoverable. Claude Code runs in a sandbox. Cursor has access to your repo but not your production database. Devin can open PRs but can’t merge them. They can’t break things that matter.
Real teams add similar constraints. You might let an agent interact with your test environment but not production. You might let it commit to a branch but not merge to main. You might let it read your documentation but not modify it. These constraints are friction, but they’re necessary friction because the alternative is trusting the agent completely, which nobody does.
The Daily Signal piece on “what breaks when you run AI agents unsupervised” crystallizes this: agents at scale need isolation, monitoring, and kill switches. You’re not removing human judgment, you’re automating the parts where judgment is already clear and creating guardrails around the rest.
Real Example: A Coding Agent Task
Here’s what actually happens when you build a coding agent for production:
# Documentation version:
agent = CodeAgent(model="gpt-5.4")
result = agent.run("Add authentication to the login endpoint")
print(result)
# Production version:
from tenacity import retry, stop_after_attempt, wait_exponential
import json
class ProductionCodeAgent:
def __init__(self, model_primary, model_fallback, sandbox_env):
self.primary = model_primary
self.fallback = model_fallback
self.sandbox = sandbox_env
self.max_retries = 3
self.cost_limit = 5.00 # dollars per task
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def generate_code(self, task, context):
# Estimate cost before calling
estimated_tokens = estimate_tokens(context + task)
if estimated_tokens * model_price > self.cost_limit:
# Fall back to cheaper model or reject
model = self.fallback
else:
model = self.primary
response = model.call(prompt=build_prompt(task, context))
return response
def verify_code(self, generated_code):
# Run in sandbox, capture output
try:
result = self.sandbox.execute(generated_code, timeout=30)
# Check if tests pass
if "FAIL" in result.stderr or result.returncode != 0:
return False, result.stderr
return True, result.stdout
except TimeoutError:
return False, "Execution timeout"
except Exception as e:
return False, str(e)
def run(self, task):
context = load_codebase_context(limit_tokens=8000)
logs = []
for attempt in range(self.max_retries):
generated = self.generate_code(task, context)
logs.append({"attempt": attempt, "cost": estimate_cost(generated)})
verified, feedback = self.verify_code(generated.code)
if verified:
# Code passed, but check for security issues
security_check = security_model.check(generated.code)
if security_check.passed:
return {"success": True, "code": generated.code, "logs": logs}
else:
logs.append({"security_failed": security_check.issues})
else:
logs.append({"verification_failed": feedback})
# Pass verification failure back to agent
*Question via [Hacker News](https://news.ycombinator.com/item?id=47466679)*