Office Hours — How do you optimize LLM usage costs when building AI-powered coding features?

How do you optimize LLM usage costs when building AI-powered coding features?

Cost optimization for LLM-powered coding features isn’t a single lever—it’s a stack of decisions about model selection, routing, caching, and when to stop using LLMs altogether. Get any one of these wrong and you’ll hemorrhage money fast. Get them right and you’ll ship faster than competitors still token-counting in spreadsheets.

Model Selection Is Your Biggest Lever

The gap between using GPT-5.5 and Gemini 3.5 Flash for the same task can be 10x on cost, sometimes with better latency. Don’t pick a model based on leaderboards. Pick it based on your actual task.

For code completion and inline suggestions, Gemini 3.5 Flash dominates on latency and cost. If you’re building a Copilot-style feature that needs to feel snappy, Flash is the default starting point. For more complex code generation or refactoring, Claude Sonnet 5 often produces cleaner output but costs more and uses more tokens (the tokenizer change from July 2026 added a hidden ~30% token overhead that caught a lot of teams off guard).

GPT-5.5 is overkill for most code generation unless you’re doing novel architecture design or security-critical refactoring. GPT-4.1 Nano exists specifically for high-throughput, cost-sensitive work. If you’re running thousands of small linting fixes or auto-formatting tasks, Nano should be your default, not your fallback.

Open-weight models matter here too. Devstral 2 (72.2% SWE-bench) can run locally or via providers like Together AI. For a feature you control end-to-end—like an internal code review bot—self-hosting might cost less than API calls after month two.

Caching and Deduplication

Most teams don’t implement request deduplication until they’ve already wasted thousands. Before you call an LLM, check if you’ve already answered this question in the last N hours.

If a user asks “refactor this function to use async/await” and you’ve already paid for that exact refactoring for a similar function, cache the reasoning. Don’t cache the raw output (code changes), but cache the reasoning chain and reapply it with minimal re-inference. This is where persistent latent memory (from July’s ILCP research) starts mattering—instead of re-tokenizing context on every call, transfer compressed hidden states between similar requests.

For Copilot-style in-IDE features, cache the file context. Claude’s prompt caching costs 90% less for cached tokens. If you’re analyzing a 5,000-line file repeatedly (as edits come in), pay once for context and reuse it. Same principle with system prompts—write them once, cache them, hit the cache repeatedly.

Routing by Task Complexity

Not every code generation task is equally hard. Route simple tasks to cheaper models and reserve expensive models for tasks that actually need them.

A concrete example: if a user asks for a docstring, use Gemini 3.5 Flash (~$0.075/M input tokens). If they ask for a full module rewrite with architectural changes, route to Claude Sonnet 5 or GPT-5.5. You can detect intent from the prompt itself or from past success rates on similar tasks.

Build a small classifier (doesn’t need to be ML—pattern matching works) that tags incoming requests. 70% of code requests are probably simple refactorings or completions. Route those to Flash. 20% are moderate complexity. Route those to Sonnet 5. 10% are genuinely hard. Route those to GPT-5.5 or skip the LLM entirely and ask the user to clarify.

When to Stop Using LLMs

This is where real money gets saved. Not every coding problem needs an LLM.

Linting, formatting, and structural transforms don’t need LLMs—use AST manipulation, tree-sitter, or deterministic tools. If you’re normalizing imports or fixing code style, a regex pass + a linter is 1000x cheaper and faster. Retrieving API docs from your codebase doesn’t need an LLM—use search and embeddings if necessary, but often a good keyword search into a trie beats semantic search on cost and latency.

Test generation is a place teams waste huge amounts on LLMs. You can generate test cases deterministically from code coverage, mutation testing, or property-based testing frameworks. LLMs are useful for writing assertions after you know what to test, not for discovering what to test.

Token Budgeting and Rate Limiting

Set hard limits per user, per feature, per day. If a user hits their budget, either queue the request, degrade to a cheaper model, or fail gracefully. Don’t let one user’s runaway loop burn your month’s budget in an hour.

Implement exponential backoff on failures. If an LLM call fails, don’t retry immediately at full price. Wait, then retry with a cheaper model, then ask the user to break the problem into smaller pieces.

Example: A Refactoring Feature

Let’s say you’re building a code refactoring assistant in your IDE.

User asks: "Convert this Promise-based code to async/await"

1. Check cache: Is this exact request cached? (No → proceed)
2. Route decision: Is this simple? (Yes → use Gemini 3.5 Flash)
3. Make request: $0.00075 for input tokens (cached file context), $0.003 for output
4. Cache result: Store reasoning for similar "Promise → async" requests
5. User edits code: Re-use cached context if file is mostly unchanged (Claude's cache saves 90% here)
6. Next request: "Also convert the error handler" 
   → Re-use same context from cache, small delta, pay $0.0001

Total cost: ~$0.005 for a multi-turn interaction that might have cost $0.10 if you regenerated context every time.

Compare that to not caching and not routing: same interaction with GPT-5.5 and no deduplication could easily run $0.40 per user per day. At 1,000 active users, that’s $400/day or $12,000/month.

Budget Tracking and Monitoring

Use structured logging to track which features, which models, and which user behaviors cost the most. Tag every LLM call with: feature name, model used, input/output token counts, latency, success/failure.

Run a weekly report. You’ll often find 80% of your costs come from 20% of features or user behaviors. Fix the top 3 offenders and you’ve probably cut costs in half.

Bottom line: Pick the right model for each task (Gemini Flash for simple, Claude Sonnet for moderate, GPT-5.5 for hard), implement prompt caching aggressively, and route away from LLMs entirely whenever a deterministic tool works. Most teams can cut costs 60-75% without touching code quality by just being systematic about model selection and caching.

Question via Hacker News