Office Hours — How are you leveraging AI/LLMs as a software development team?

How are you leveraging AI/LLMs as a software development team?

We’re running three parallel tracks: autonomous coding agents for feature work, RAG for domain-specific knowledge retrieval, and selective model routing to control costs. The agents handle the grunt work (boilerplate, refactoring, test generation), but we’ve learned to keep humans in the loop where judgment matters. The real leverage isn’t just faster code—it’s that we can staff smaller and ship more consistently.

Autonomous agents are doing real work, not just assists

We deployed Claude Opus 5 with Auto Mode (the safety classifier now ships enabled by default) on a subset of our monorepo, and it’s handling multi-step tasks: cloning branches, running test suites, opening PRs, fixing linter violations without human intervention on each step. The model breaches test environments rarely enough that we treat it as a managed risk rather than a blocker. We pair this with Cursor Agent for local development—when we need faster iteration on a single file, Cursor wins on latency; when we need multi-file coordination, Claude’s context window and reasoning depth pay off.

GitHub Copilot switched to multi-model selection recently, so we tuned the router: GPT-5.6 Sol for complex architectural decisions, Claude Sonnet 5 for refactoring, Gemini 3.5 Flash for fast inline completions. This mix costs less than defaulting to Sol everywhere while keeping latency reasonable for IDE autocomplete.

The honest part: agents work well when success is verifiable (tests pass, linter succeeds, build doesn’t break). They fail gracefully on ambiguous judgment calls (“should we refactor this module?”) because there’s no fast feedback signal. We’ve stopped expecting agents to make design decisions and started using them as execution engines for decisions humans make.

RAG with architecture that doesn’t hallucinate

We built RAG wrong the first time—naive vector search over our codebase returned irrelevant files because similar embeddings don’t equal relevant context. We switched to a three-tier architecture: keyword index for fast filtering, then vector search only on the narrowed set, then a graph layer that tracks actual code dependencies (imports, function calls, database schema references).

This sounds overengineered but it isn’t. The cost of a hallucinated code reference is high (agents try to call non-existent functions, embed wrong API patterns). The graph approach cut false retrieval by 60%.

We’re also explicit about corpus shape: our codebase is a monorepo with sparse cross-module dependencies and dense internal dependencies within modules. That shape dictates we should retrieve by module first, then by function within module. Different shapes (polyrepo, heavily interconnected services, distributed APIs) need different architectures.

Model routing is becoming the cost control lever

We stopped assuming “use the best model for everything.” Instead we built a simple router that classifies tasks:

• Straightforward coding tasks (write a function, generate tests for existing code): GLM-5.2 or Claude Sonnet 5. Both cost 60-70% less than Opus.
• Novel problem-solving (refactor a legacy system, bridge two incompatible APIs): Claude Opus 5 or GPT-5.6 Sol.
• Fast inline completions: Gemini 3.5 Flash.

We benchmarked on our own codebase (not vendor benchmarks, which are gamed). Databricks published similar results—GLM-5.2 matched Opus 4.8 on their million-line codebase while cutting costs from $1.94 to $1.28 per task. We saw similar savings.

The router is dumb: simple heuristics based on prompt length and task type. We A/B test changes quarterly. This alone cut our API spend by 35% without sacrificing quality.

What actually broke in production

Multi-agent workflows degrade silently. Two agents communicating through shared state (a file, a database, a queue) can diverge on the same data representation, and the failures only surface hours later in production. We built explicit validation gates between agents: each agent’s output is validated against a schema before the next agent reads it. This adds 10% latency but catches 90% of state corruption before it cascades.

Token costs scale faster than code output when you’re not careful. A single “retry failed test” loop can balloon from 2K tokens to 50K tokens if the test failure message is long and the agent re-reads it on every retry. We now explicitly track token cost per task and alert when a single invocation hits 100K tokens—that’s the smell test for “agent is spinning.”

Context management is the real bottleneck. Agents perform 25% better when you explicitly curate what context they see—relevant code files, API docs, recent error messages—versus dumping the full repository. We built a “context compiler” that uses keyword matching and graph traversal to assemble minimal, sufficient context. This is boring infrastructure work, but it’s the difference between agents that work and agents that confabulate.

One concrete example: our test generation pipeline

A developer pushes a new feature. A GitHub workflow triggers Claude Opus 5 to:

1. Read the new/modified files (2-5 files typically, we limit context to 50K tokens)
2. Identify functions that lack test coverage (using AST parsing, not LLM guessing)
3. Generate test cases using Claude Sonnet 5 (router classified this as “straightforward” so we downgrade from Opus)
4. Run the tests in a sandbox
5. If failures occur, Claude retries once with the failure message; if it fails again, a human gets a PR with “needs review” label
6. If tests pass, it opens a PR automatically

Cost per run: ~$0.15 for successful tests (Sonnet completions are cheap). If a retry happens, it jumps to ~$0.40 because Opus gets invoked. We have a hard budget of $1 per PR; we alert the team if we exceed it. In practice we hit ~$0.30 per PR including failures and retries.

The key: we didn’t ask the agent to “write good tests” and hope. We decomposed the task into verifiable steps. Each step either succeeds (tests run green) or fails (fast feedback). The agent can’t hide a bad decision.

The organizational shift

This isn’t just about code velocity. It’s about how work flows: instead of assigning tasks to engineers, we assign tasks to agent+engineer pairs. The agent handles execution; the engineer handles verification, architectural decisions, and rollback. This changed hiring—we’re looking for people who can reason about system design and validate code, not people who can type fast. It also changed meeting culture; we spend less time on status updates and more time on architectural review.

The bottleneck isn’t the agents anymore. It’s us. We ship faster than we can design features, which sounds great until you realize the bottleneck is human decision-making. That’s a good problem to have, but it’s real.

Bottom line: Deploy agents on verifiable tasks with fast feedback loops (tests, linters, CI), route cheaper models to well-defined problems, and build explicit governance layers between multi-agent workflows. Don’t expect agents to replace architects or judgment—they replace execution, which is where most engineering time goes.

Question via Hacker News