Office Hours — How has using AI coding assistants actually changed the types and frequency of bugs you're seeing in production?

How has using AI coding assistants actually changed the types and frequency of bugs you’re seeing in production?

The bug landscape has shifted, not disappeared. We’re seeing fewer logic errors in straightforward code paths and more subtle failures in edge cases, error handling, and integration points. The assistants are competent at happy paths—they generate correct implementations of standard algorithms and common patterns. Where things break is exactly where they’ve always been hardest to debug: the boundaries between systems, the error states nobody talks about, and the architectural decisions that live outside any single function.

The New Bug Signature

Most production incidents I’ve tracked fall into a few categories that weren’t as common before.

Assistants tend to generate plausible-looking error handling that doesn’t actually handle the error. A try/except block that logs the exception and continues as if nothing happened. A retry loop that doesn’t backoff. A database transaction that doesn’t fail correctly when a constraint is violated. The code runs and doesn’t throw—it just silently does the wrong thing. You catch these in staging if you stress-test error paths, but they slip through if your tests only check the success case.

Second, integrations between AI-generated modules fail in ways that suggest each function was written without the full context of how it fits together. A function that accepts user input doesn’t validate. The caller assumes it did. A function returns a string when the rest of the system expects a list. The types check but the semantics don’t. You find these when you’re assembling pieces that were generated separately, even if they’re in the same pull request.

Third, we’re seeing a spike in off-by-one errors and subtle algorithmic mistakes in search, sorting, and iteration logic. The assistant generates code that looks correct at a glance—it iterates from 0 to len(array)—but misses a boundary condition or uses the wrong comparison operator. These are easier to spot than they used to be because they’re so consistent. If you see a .length - 1 where it should be .length, you immediately know it’s probably AI-generated.

What’s Actually Different

Before coding assistants, a developer writing code had to maintain context for the entire function. That constraint forced a kind of discipline—you didn’t write something you couldn’t trace through mentally. Now, developers accept longer functions from the assistant and review them at a higher level. That trades local correctness for distributed fragility.

The assistant is also generating code that compiles and passes type checking but violates invariants that only live in your documentation or architectural decisions. A function that modifies shared state without locking. A caching layer that doesn’t invalidate correctly. A permission check that passes the user object but forgets to check the scope. These aren’t syntax errors—they’re logic errors that emerge from incomplete context.

Here’s a concrete example: an assistant generated a database migration that looked correct in isolation.

def upgrade():
    op.add_column('users', sa.Column('status', sa.String(50), nullable=False, default='active'))

def downgrade():
    op.drop_column('users', 'status')

The migration ran fine in dev. In production, it locked the table for 45 seconds on a table with millions of rows. Nobody had told the assistant that this particular table can’t tolerate full locks. The code was right—the context was missing. That’s the pattern now. Fewer bugs per line of code written. More bugs where context and code diverge.

The Frequency Question

We’re not writing more code that fails overall, but we are writing more code per developer. The failure rate per thousand lines is probably lower. The failure rate per developer per day is probably higher because of sheer volume. You ship more code faster, and more of it makes it to production before the edge cases surface.

The critical shift: assistants are genuinely good at scaffolding and glue code. They’re bad at anything requiring domain knowledge, historical context, or understanding why something matters. If you’re building something where the interesting part is not the code but the decision about what code to write, the assistant will generate code that’s syntactically perfect and strategically wrong.

What Changed in Practice

Our code review process now focuses harder on “does this function actually do what we need” rather than “is this syntactically correct.” We’ve added more integration tests because the seams between components are where failures hide. We’ve gotten more paranoid about error paths—if it’s not explicitly tested, assume the assistant got it wrong.

The surprising win: assistants are good at generating defensive code boilerplate. Null checks, bounds checks, type validation. When we explicitly ask for “add logging at every decision point,” they do it consistently. When we say “generate the test cases too,” the test coverage is often more thorough than what a tired developer would write at 4pm.

Bottom line: AI coding assistants have shifted bugs from “straightforward logic errors” to “missing context and edge cases.” Your production reliability depends less on code quality per line and more on how well you’ve captured requirements and error conditions before generating code. Treat the assistant output as a draft, not a finished product, and add explicit evals for the domain-specific constraints that no prompt can fully encode.

Question via Hacker News