Office Hours — How do I handle AI auditors that catch almost nothing in production but flag everything as wrong?

How do I handle AI auditors that catch almost nothing in production but flag everything as wrong?

The Real Problem: Auditor Precision vs. Practitioner Signal

Anthropic published a study last week showing that AI-based auditors achieve 98% detection rates on synthetic test cases but generate so many false positives that reviewers stop trusting the system entirely. You end up with two terrible outcomes at once: missing actual agent failures while drowning in noise that nobody acts on.

The root issue is that auditing agent behavior isn’t a classification problem. It’s a signal-to-noise problem. A model trained to be sensitive enough to catch real failures will flag everything that looks vaguely suspicious, including behavior that’s actually fine. You get what’s called the precision-recall tradeoff turned inside out: high recall (you catch the bad stuff) means low precision (you flag tons of false positives), and the false positives vastly outnumber real failures in production, so your reviewers learn to ignore alerts.

This is exactly why alert fatigue kills production observability across infrastructure. The moment humans stop trusting your auditor, it stops being useful.

Why Synthetic Training Data Makes This Worse

Most auditor implementations train on curated examples of “bad” behavior: agents taking unauthorized actions, hallucinating API calls, violating constraints. These datasets are clean, balanced, and artificial. They don’t look like production, where the vast majority of agent steps are mundane and correct, with failures scattered sparsely throughout thousands of requests.

When your auditor sees real production data, it’s calibrated completely wrong. It sees patterns in normal behavior that resembled the edge cases it trained on, and it flags them. Your agent successfully queries a database, but the auditor flags it because “database queries sometimes go wrong in the training set.” Your agent makes a legitimate API call with slightly unusual parameters, and the auditor flags it because “unusual parameters occurred in failed examples.”

The fix isn’t to retrain on more production data. The fix is to stop asking the auditor to be the decision-maker and start asking it to be a proposal generator.

A Practical Pattern: Auditor as Proposer, Not Gatekeeper

Instead of asking “Is this agent step acceptable or not?”, ask “What aspects of this step should a human look at?” This reframes the problem. You’re not trying to classify behavior as good or bad. You’re trying to surface relevant information.

Here’s a concrete approach:

Extract observable facts, not judgments. Don’t ask the auditor to judge whether an agent action is safe. Ask it to extract: “Agent called DELETE on table X, affecting Y rows. User permission level is Z. Most recent similar call was N minutes ago.”
Use deterministic rules for high-confidence gates. Any check that’s certain (agent is operating outside its declared scope, calling an API it shouldn’t have access to, modifying production data it explicitly shouldn’t touch) should be deterministic code, not an LLM. LLMs are bad at saying “no” reliably. They hedge.
Route edge cases to human queues with full context. If an agent action is unusual but plausible, put it in a queue with: the full action transcript, the context that triggered it, similar historical actions, and what the auditor extracted as relevant facts. Don’t make the auditor decide. Give the human enough information to decide in 30 seconds.
Measure auditor precision on false positives explicitly. Track: “Of all the alerts this auditor generated, what fraction did a human actually need to act on?” If it’s below 20%, your auditor is worse than useless. If it’s above 50%, you’re not being sensitive enough to actual failures.

A production example: An agent needs to modify a database schema. You don’t ask an LLM auditor “Is this schema change safe?” You extract: table name, change type, affected row count, whether it’s reversible, when the last schema change happened, and whether this agent has made schema changes before. Route to a senior DBA with that context. The DBA decides in seconds. Your auditor was just a fact extractor.

The Deeper Issue: Agents Need Verifiable Outcomes

The real solution is architectural. AI agents work best when success is objectively verifiable: tests pass or fail, queries return results or don’t, linter reports zero errors or not. Auditing agent behavior is hard because you’re trying to judge intent and correctness from side effects. That’s inherently ambiguous.

If you can restructure your agent workflows so that every action produces a signal you can check automatically, do that first. “Did the test pass after this code change?” is much easier to verify than “Was this code change good?” Test coverage, linter passes, type checking—these all bypass the need for LLM auditing entirely.

For actions that don’t have automatic verification (customer interactions, judgment calls, creative work), accept that you need humans in the loop and design for that explicitly. Don’t pretend an auditor can replace human judgment at scale.

Bottom line: Stop treating auditors as decision-makers. Use deterministic rules for high-confidence gates, extract observable facts with LLMs, and route edge cases to humans with full context. If you’re generating more alerts than humans can act on, your auditor is creating liability, not reducing it.