Where randomness meets reason

Office Hours — What's the current state of using LLMs for code generation while keeping infrastructure and API costs reasonable? A daily developer question about AI/LLMs, answered with a direct, opinionated take. 2026-07-09T12:00:00.000Z Office Hours Office Hours office-hoursq-and-apractical-ai

Office Hours — What's the current state of using LLMs for code generation while keeping infrastructure and API costs reasonable?

A daily developer question about AI/LLMs, answered with a direct, opinionated take.

office-hoursq-and-apractical-ai
Daily One question from the trenches, one opinionated answer.

What’s the current state of using LLMs for code generation while keeping infrastructure and API costs reasonable?

The Real Cost Problem Isn’t Models, It’s Orchestration

Your LLM bill isn’t high because GPT-5.5 or Claude Opus 4.8 are inherently expensive. It’s high because most teams aren’t systematically managing context, caching, or routing. A single developer burning through token budgets on repeated context loads—same codebase files, same API specs, same project README loaded on every request—will blow through thousands of dollars a month. The same work routed through a competent orchestration layer costs a fraction of that.

The bad news: no one ships production code generation without understanding this. The good news: the fixes are concrete and don’t require you to accept lower code quality.

Concrete Cost Controls That Actually Work

Prompt caching is your first lever. If your system loads the same 50KB of codebase context (function definitions, type stubs, architectural docs) across multiple requests, cache it. Claude Opus 4.8 and GPT-5.5 both support this natively. A typical enterprise codebase context that costs $10 in input tokens on the first request costs $0.30 on the second request (cached tokens are 90% cheaper). For a team generating code regularly, this alone drops costs 40-60%.

Here’s what a cached context setup looks like with Claude:

import anthropic
import json

client = anthropic.Anthropic()

# Load your codebase context once
codebase_context = """
# Project Architecture
- Backend: FastAPI in src/api/
- Database: PostgreSQL with SQLAlchemy ORM
- Key models: User, Post, Comment in src/models/
- Middleware: auth.py enforces JWT validation

# Type Definitions (excerpt)
class User(Base):
    id: int = Column(Integer, primary_key=True)
    email: str = Column(String, unique=True)
    created_at: datetime = Column(DateTime, default=utcnow)

# Style Guide
- Use type hints everywhere
- Docstrings for all public functions
- 4-space indent, max 88 chars
"""

def generate_code_with_caching(user_request: str, task_type: str):
    """
    Generate code with cached codebase context.
    Only the user request is sent on every call.
    """
    response = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=2048,
        system=[
            {
                "type": "text",
                "text": "You are an expert code generator. Follow the style guide and architecture patterns.",
            },
            {
                "type": "text",
                "text": f"Here is the codebase context:\n{codebase_context}",
                "cache_control": {"type": "ephemeral"},
            },
        ],
        messages=[
            {"role": "user", "content": user_request}
        ],
    )
    return response.content[0].text

# First call: pays full price for codebase context
result1 = generate_code_with_caching(
    "Write a POST endpoint to create a new user",
    "endpoint"
)
print(f"First call - Input tokens: {response.usage.input_tokens}")

# Second call: cache hit, 90% cheaper on context
result2 = generate_code_with_caching(
    "Write a GET endpoint to fetch user by ID",
    "endpoint"
)
print(f"Second call - Input tokens: {response.usage.input_tokens}")
# Cache tokens are NOT counted against quota on the second call

That’s step one. You’re now paying once for context, reusing it across requests within a session window.

Model Routing: Match Cost to Task Complexity

Not every code generation task needs your flagship model. A simple docstring generation or variable rename works fine on Claude Sonnet 5 (released June 30, 2026, with introductory pricing at $3/$15 per M input/output tokens through August 31, 2026, then $3/$15 standard). Complex multi-file refactors or architectural decisions need Opus 4.8 or GPT-5.5.

Simple routing heuristic:

def choose_model_for_task(request: str, task_type: str) -> str:
    """Route to appropriate model based on complexity."""
    
    expensive_tasks = {
        "refactor_cross_file",
        "architectural_decision",
        "security_review",
        "complex_algorithm",
    }
    
    if task_type in expensive_tasks:
        return "claude-opus-4-8"  # Best reasoning, highest cost
    
    if "docstring" in request.lower() or "comment" in request.lower():
        return "claude-sonnet-5"  # Good enough, 70% cheaper
    
    if len(request) < 200 and task_type == "snippet":
        return "claude-sonnet-5"
    
    return "claude-opus-4-8"  # Default to best for safety

Typical result: 30% of requests go to Sonnet, 70% to Opus. Your effective cost per request drops by about 20-30% without sacrificing quality on the tasks that matter.

Context Compression Buys You Token Efficiency

Long codebase contexts are expensive. Before loading an entire 200KB repository into a prompt, let the LLM tell you what it actually needs. Use a two-step pattern:

  1. Send a brief query (“I need to add a new admin endpoint”) plus a small index (file list, function signatures).
  2. Let the model ask for specific files.
  3. Load only what it requested.

This cuts context size 50-70% on average.

def smart_context_loading(request: str, repo_structure: dict):
    """First pass: determine what context the model actually needs."""
    
    index_prompt = f"""
    I need to: {request}
    
    Available files (summary):
    {json.dumps(repo_structure, indent=2)}
    
    Which files do you need to see? List them.
    """
    
    response = client.messages.create(
        model="claude-sonnet-5",  # Cheap model for routing
        max_tokens=200,
        messages=[{"role": "user", "content": index_prompt}],
    )
    
    needed_files = parse_file_list(response.content[0].text)
    
    # Now load only what was requested
    full_context = load_files(needed_files)
    
    return full_context

Real Numbers: What This Looks Like in Production

Baseline scenario: A team of 10 engineers using GPT-5.5 for code generation without orchestration. Each engineer makes 20 requests per day, each request loads 50KB of context (codebase files, docs, examples). That’s 10 engineers × 20 requests × 50KB = 10MB of input context daily. At typical API rates, that’s roughly $150-200 per day, or $3,600-4,800 per month.

Same team, same usage, with:

  • Prompt caching (90% cost reduction on repeated context): saves ~$3,000/month
  • Model routing (30% of requests to cheaper tier): saves ~$400/month
  • Context compression (50% smaller contexts): saves ~$300/month

Total: $4,000 down to roughly $350-400 per month for the same throughput and quality.

That math assumes you’re disciplined about caching cache keys correctly and actually routing decisions, not aspirational.

The Catch: Autonomous Agents Still Have Bad Economics

The Daily Signal reported in early July that even Microsoft and Uber found autonomous agents uneconomical for spreadsheet-level tasks at scale. The issue isn’t the model—it’s retry loops, verification steps, and fallback queries that add up. A single failing agent task might cost $5 in retries to fix what a human could do for $2 in labor.

Autonomous coding agents (Claude Code, Cursor Agent, GitHub Copilot with Opus 4.8) do work in production for bounded tasks with clear success signals (tests pass/fail, linter runs, CI succeeds). But the cost-per-task is still 3-5x what you’d pay

Question via Hacker News