Office Hours — We're using LLMs to file insurance claims at scale in production. The model confidently makes mistakes and we have no visibility into failures until customers complain weeks later. How do I catch this before it becomes a liability nightmare?
A daily developer question about AI/LLMs, answered with a direct, opinionated take.
We’re using LLMs to file insurance claims at scale in production. The model confidently makes mistakes and we have no visibility into failures until customers complain weeks later. How do I catch this before it becomes a liability nightmare?
This is the exact scenario playing out right now at CVS Health and Cigna. Insurance claims are high-stakes: a confident hallucination buried in a form doesn’t surface as an error until someone actually processes the downstream paperwork. By then you’ve potentially filed hundreds or thousands of wrong claims, and the liability compounds fast.
The core problem isn’t the model. It’s that you’ve built a system where the LLM is the primary decision-maker and the feedback loop is weeks long. That’s a recipe for liability.
The Visibility Gap
You can’t monitor what you can’t see. LLMs don’t fail by raising exceptions. They fail by producing plausible-looking nonsense. A missing field, an inverted claim amount, a spouse marked as the patient—these all look “complete” to a model that’s just trying to minimize cross-entropy loss.
The question isn’t “did the model finish?” It’s “did the model finish correctly?” And you won’t know that by looking at logs.
What Actually Works: Verification Before Filing
Stop filing claims immediately. Insert a verification layer that runs before any submission.
For each claim, run a second pass through the form using a different model (or the same model with a verification prompt) that checks for logical consistency:
verification_prompt = f"""
You are a claim auditor. Review this completed insurance claim form for errors.
Check for:
1. Missing required fields (marked with *)
2. Logical contradictions (e.g., surgery date before diagnosis date)
3. Out-of-range values (e.g., age > 120, claim amount > policy limit)
4. Suspicious patterns (e.g., claimant and beneficiary are the same person when they shouldn't be)
Form:
{completed_form}
If you find ANY issues, list them explicitly with the field name and what's wrong.
If the form is clean, respond with: "VERIFIED: Safe to file"
"""
This doesn’t cost much. Verification is a lightweight, deterministic check. It runs on the completed form, not on parsing messy PDFs. Claude Haiku 5.5 or GPT-6 Luna can do this for pennies per claim.
The critical part: if verification finds issues, the claim does NOT file. It goes to a human queue instead. You’ve now inverted the feedback loop from “complain weeks later” to “escalate within hours.”
Audit Trail and Sampling
Log every claim with its full LLM reasoning, the verification result, and what path it took (auto-filed or escalated). This gives you:
-
Deterministic auditability: If a regulator asks why claim XYZ was filed a certain way, you have the exact prompt, model response, verification logic, and decision point. That’s defensible. “The LLM did it” is not.
-
Sampling strategy: Pull a random 1-5% sample of auto-filed claims daily and manually spot-check them. If you find errors above a threshold (say, >0.5%), pause auto-filing and escalate all claims until root cause is clear. This catches systemic problems before they become thousands of wrong claims.
# After filing N claims, sample and audit
sample_size = max(100, int(total_filed * 0.05))
sample_claims = random.sample(filed_claims, min(sample_size, len(filed_claims)))
audit_results = []
for claim in sample_claims:
manual_review = human_auditor.review(claim)
if manual_review.has_errors:
audit_results.append({
'claim_id': claim.id,
'error_type': manual_review.error_type,
'severity': manual_review.severity
})
error_rate = len([r for r in audit_results if r.severity == 'critical']) / len(sample_claims)
if error_rate > 0.005:
alert("Critical error rate detected. Pausing auto-filing.")
Confidence Thresholds and Escalation Rules
Insurance claims are structured. Some fields are easier to extract correctly than others. A birth date in the wrong format is different from a diagnosis code that doesn’t exist in the insurance company’s database.
Define escalation rules upfront:
- High confidence, low risk (e.g., claimant name, date of birth from clear source document): Auto-file.
- Medium confidence or medium risk (e.g., diagnosis code inferred from narrative): Human review required.
- Low confidence or high risk (e.g., claim amount exceeds policy limits, or the model is unsure): Always escalate.
This isn’t fuzzy. It’s a decision tree. Use the model’s own uncertainty signals (token probabilities, consistency across multiple passes) plus the field’s business risk to determine routing.
Why This Beats Post-Hoc Detection
You could theoretically catch errors downstream when the insurance company rejects a malformed claim. That doesn’t work because:
- Insurance companies may silently accept and process wrong claims, especially small ones.
- By the time rejection happens, you’ve already filed it and created a paper trail.
- The reputational and regulatory damage from “we filed hundreds of wrong claims and didn’t notice” is severe.
The only defensible position is: “We caught issues before filing through automated verification and human review.” That’s what prevents liability nightmares.
Bottom line:
Insert a verification layer before filing any claim, sample audit filed claims daily, and escalate anything above your confidence threshold to humans. The LLM files claims; verification and auditing catch its mistakes before they become legal liability. This is how CVS and Cigna should be running this, and it’s the only approach that holds up under regulatory scrutiny.