Office Hours — We're using an LLM to file insurance claims at scale in production. How do I handle the case where the agent confidently does the wrong thing and no one catches it until weeks later?
A daily developer question about AI/LLMs, answered with a direct, opinionated take.
We’re using an LLM to file insurance claims at scale in production. How do I handle the case where the agent confidently does the wrong thing and no one catches it until weeks later?
The Real Problem Isn’t Detection, It’s Silent Drift
You’ve already identified the core issue: an LLM agent can submit a malformed claim form with absolute confidence, bypass all your immediate checks, and sit in some insurer’s queue for two weeks before anyone realizes it was wrong. By then you’ve probably filed 50,000 more claims through the same buggy pipeline.
The good news: this is now the dominant failure mode in production agent systems. The Daily Signal reported on 100+ real deployments showing that quality and token cost drift account for 38% of agent failures, while actual errors cause only ~1%. Your problem isn’t rare—it’s the problem everyone’s hitting.
The bad news: traditional monitoring won’t catch this. You can’t exception-log your way out of it because the agent isn’t throwing errors. It’s executing cleanly, making plausible decisions, and building confidence in its outputs. The problem is systematic degradation over time or under conditions you didn’t test.
Why Your Current Safeguards Are Probably Insufficient
If you’re checking “did the form submit successfully?” you’re only catching the happy path. You need to separate form submission success from claim correctness. A form can submit perfectly and still contain the wrong medication dosage, wrong patient ID, or inverted claim amount.
Anthropic’s research on using AI to audit agent misbehavior found that models achieve high detection rates on synthetic test cases but generate so many false positives on real data that the signal becomes useless—they miss 96% of actual failures while flooding reviewers with false alarms. This means your strategy can’t be “have Claude audit Claude.”
Three Concrete Patterns That Actually Work
Pattern 1: Deterministic Validation Gates Before Submission
Before the agent submits anything, run a lightweight deterministic check that doesn’t use another LLM. Extract key fields (patient ID, claim amount, procedure code) and validate against known constraints:
def validate_claim(form_data: dict) -> tuple[bool, list[str]]:
errors = []
# Hard constraints that should never vary
if not form_data.get("patient_id"):
errors.append("missing_patient_id")
elif not form_data["patient_id"].isdigit():
errors.append("patient_id_not_numeric")
# Range checks (amount should not exceed claim ceiling)
claim_amount = form_data.get("claim_amount", 0)
if claim_amount > 100000: # Your actual ceiling
errors.append(f"claim_amount_exceeds_max: {claim_amount}")
# Procedure code format (CPT codes are 5 chars)
procedure_code = form_data.get("procedure_code", "")
if procedure_code and len(procedure_code) != 5:
errors.append(f"invalid_procedure_code_length: {procedure_code}")
# Cross-field consistency: procedure code should map to known specialties
if procedure_code and not is_valid_procedure_code(procedure_code):
errors.append(f"unknown_procedure_code: {procedure_code}")
return len(errors) == 0, errors
This catches systematic errors (agent consistently mis-formatting procedure codes) before they propagate. It’s boring, deterministic, and 100% auditable.
Pattern 2: Stratified Sampling With Human Review Loop
Don’t review every claim. Instead, use stratified sampling based on risk factors you can compute without another LLM:
- 100% review for claims over $50K
- 10% random sample of claims under $5K
- 50% of claims flagged by your deterministic validator (even after correction)
- 100% of claims for first-time patient IDs or new procedure codes
The key insight: you’re not trying to catch every error. You’re trying to catch systematic errors early. If 2% of sampled claims have the wrong specialty code, you know something is drifting. If 0.1% do, you have a rare edge case.
Schedule these reviews asynchronously (batch them into a Monday morning queue) so you’re not blocking claim submission but you’re catching problems within days, not weeks.
Pattern 3: Output Fingerprinting and Anomaly Detection
Track statistical properties of agent outputs over time without using another LLM to judge them:
def fingerprint_claim(form_data: dict) -> dict:
return {
"avg_claim_amount": form_data.get("claim_amount", 0),
"procedure_code_format": len(form_data.get("procedure_code", "")),
"patient_id_format": len(form_data.get("patient_id", "")),
"timestamp": datetime.now(),
"model_version": get_model_version(),
}
# Track rolling statistics
def detect_drift(last_1000_fingerprints: list) -> dict:
amounts = [f["avg_claim_amount"] for f in last_1000_fingerprints]
return {
"amount_std_dev": np.std(amounts),
"amount_mean": np.mean(amounts),
"amount_outliers": sum(1 for x in amounts if x > np.mean(amounts) + 3*np.std(amounts))
}
If the agent starts submitting claims that are consistently $5K higher than its historical average, or procedure codes that suddenly shift from mostly orthopedic to mostly cardiac, you get an alert without needing a human reviewer.
The Timing Question: When to Catch Problems
You have three windows:
-
Before submission (deterministic gates + field validation): Catches format errors, range violations, internal inconsistencies. Blocks submission. Latency: milliseconds.
-
Within 24 hours (stratified human review): Catches systematic errors in judgment (agent choosing the wrong diagnosis code for the documented symptoms). Blocks submission or requires correction. Latency: hours.
-
In batch after 1-2 weeks (downstream audit when insurers process claims): Catches errors your agent made that passed your checks but fail insurer validation. Too late to prevent, but early enough to identify systemic problems and retrain. Latency: days to weeks.
You need all three. Focusing only on #1 is why you catch format errors but miss semantic errors. Focusing only on #3 is your current situation.
Model-Specific Considerations
GPT-6 Luna (the model CVS and Cigna are reportedly using at $0.09/task) is cheap enough that you can afford to be paranoid. Run your deterministic validator on every single claim. Run your stratified sampler on 10-20% of them. The additional compute cost is negligible compared to the cost of a denied claim or regulatory fine.
If you were using GPT-6 Astra (10x more expensive per task), you’d need to optimize sampling more aggressively. At Luna pricing, just validate everything deterministically.
The One Thing to Avoid
Don’t try to use a cheaper LLM (like Gemini 3.5 Flash or Claude Haiku) to audit the output of your main agent. You’ll spend twice the tokens and catch half the problems. Stick to deterministic rules and human sampling.
Bottom line: Set up deterministic validation gates before submission (catches format errors immediately), stratified human review within 24 hours (catches semantic errors early), and fingerprint your outputs to detect drift (catches systemic degradation). This three-layer approach will catch problems weeks earlier than waiting for downstream insurer feedback, and it’s the architecture the highest-volume agent deployments are now using.