Office Hours — Which AI model subscriptions (Claude, GPT-4, etc.) actually provide the best value for production applications right now?
A daily developer question about AI/LLMs, answered with a direct, opinionated take.
Which AI model subscriptions (Claude, GPT-4, etc.) actually provide the best value for production applications right now?
The honest answer: it depends on whether you’re optimizing for cost per task, latency, or capability ceiling, and those three things almost never align. But there’s a clearer path through the noise than there was six months ago.
The Cost Picture Has Inverted
GPT-5.6 Sol just cut prices up to 80% through recursive self-improvement and distillation. If you’re doing heavy inference, that’s table-stakes now. Claude Opus 5 trades some of that price advantage for better security (0% prompt injection success rate on browser agent tasks) and slightly lower token burn per task. For most production work, you’re choosing between Sol’s raw speed and cost versus Opus 5’s reliability and containment guarantees.
The wild card: Sonnet 5 launched with introductory pricing through end of August, but its new tokenizer emits roughly 30% more tokens for the same text. The list price didn’t change, but real per-task costs are about 40% higher than the numbers suggest. That matters at scale.
Open-Weight Models Actually Work Now
Six months ago, “self-hosted” was code for “accept worse quality or massive latency.” Not anymore. Nvidia’s Nemotron 3.5 Lightning achieves parity with much larger models at 670 tokens per second on consumer hardware. Meta’s Muse Glimmer fits on a single RTX 3090. These aren’t research demos—practitioners are running them in production.
The catch: you’re trading API convenience for infrastructure overhead. A Databricks engineer tested open-source GLM-5.2 on their own million-line codebase and found it matched Claude Opus 4.8 performance while cutting costs from $1.94 to $1.28 per task. But that required building the evaluation framework, understanding where the model actually fails, and maintaining a deployment pipeline. Not every team has the bandwidth.
Where Frontier Models Actually Pull Their Weight
Coding agents are the clearest case. Claude Code with Auto Mode is now the default, with a safety classifier catching 89% of dangerous commands versus 13.6% for humans. Cursor’s agent architecture separates planning (frontier model) from execution (cheaper model), successfully rebuilt SQLite in Rust with full test coverage. That works because planning is the bottleneck—cheaper models handle most of the mechanical work once someone smart has figured out the shape of the problem.
For anything involving novel problem-solving or adversarial reasoning, frontier models still dominate. Claude Opus 5 scored 30.2% on ARC-AGI-3 benchmarks, nearly 4x the previous record. But if your task is “extract structured data from PDFs” or “summarize documents,” you’re probably overpaying for capability you don’t use.
A Concrete Comparison
Say you’re building a customer support agent that handles 10,000 requests per month, averaging 2,000 tokens input and 500 tokens output.
GPT-5.6 Sol: Input costs roughly $0.08 per M tokens (after the 80% cut); output around $0.24 per M. That’s about $1.60 per request ($160/month).
Claude Opus 5: Introductory pricing was $3/$15 per M (standard $6/$18). Same request: $0.06 input + $0.07 output = $0.13 per request ($130/month). But you get better jailbreak resistance.
Claude Sonnet 5 with 30% token inflation: List price doesn’t move, but effective cost is closer to $0.20 per request ($200/month). Not worth it for routine tasks.
Open-source Nemotron-3.5 Lightning on H100: Roughly $0.10 per request if you amortize infrastructure costs, assuming you’re running it hot enough to justify the GPU time. You own the uptime risk.
For a support chatbot, Sol makes sense if reliability and jailbreak resistance don’t matter much. Opus 5 if you’re worried about prompt injection or need safety guarantees. Open-weight if you have the ops chops and enough volume to justify the fixed infrastructure cost.
The Hidden Token Bleed
Most teams don’t realize that agentic loops compound costs. Retry mechanisms, internal reasoning loops, re-planning—these aren’t one API call, they’re five or ten. The Daily Signal surfaced this recently: hidden token costs in agentic loops are draining budgets faster than anyone expects, and most builders don’t have visibility into where the budget is actually going.
GPT-5.6 Sol’s tiered pricing (Luna tier for cheaper tasks, Sol for frontier reasoning) lets you route based on complexity. Claude’s flat pricing doesn’t. If you’re running a mixed workload, that routing flexibility adds up.
What Actually Works in Production
Databricks’ real-world result matters here: they benchmarked on their own codebase, not vendor benchmarks. Vendor benchmarks are routinely gamed. Build your own evals on your actual workload. SWE-Bench Pro itself has issues—OpenAI found ~30% of its tasks are broken and withdrew its endorsement. Don’t trust published numbers.
The Remote Labor Index shows the top AI agent completing roughly 16% of real freelance jobs at professional quality. That’s up from 2.5% eight months earlier, but it illustrates the gap: agents work when you can verify success automatically (tests pass, linter runs, CI succeeds). They drift when success is ambiguous.
Practical Recommendations by Use Case
High-volume routine tasks (summaries, extractions, classification): Start with GPT-5.6 Luna or Claude Sonnet 5 (accounting for token inflation). Cheaper models can handle most of it; use Sol/Opus 5 only for edge cases that fail classification.
Coding agents or reasoning-heavy work: Claude Opus 5 for safety guarantees; GPT-5.6 Sol if you’re willing to trade some containment for speed. Cursor’s split-planning architecture suggests the pattern: use a frontier model for planning, cheaper for execution.
Self-hosted or compliance-sensitive: Evaluate Nemotron-3.5 Lightning or Muse Glimmer on your actual workload first. Don’t assume open-weight is slower until you measure it.
Cost-sensitive at massive scale: Build an eval framework on your own data, test open-source models, and only move to API models if they outperform on your metrics. The cost difference compounds.
Bottom line:
Use Claude Opus 5 for production systems where containment and security matter (agents with tool access, browser automation); GPT-5.6 Sol for cost-sensitive workloads where you can tolerate some prompt injection risk; evaluate open-weight models on your own data at scale before dismissing them as “research only.” The real leverage isn’t the model choice—it’s routing complexity appropriately and measuring on your actual workload, not vendor benchmarks.
Question via Hacker News