Office Hours — We're using LLMs to file insurance claims at scale in production. The model is confident about its decisions, but we have no visibility into failure modes until weeks later when customers complain. How do I catch this before it becomes a liability nightmare?
A daily developer question about AI/LLMs, answered with a direct, opinionated take.
We’re using LLMs to file insurance claims at scale in production. The model is confident about its decisions, but we have no visibility into failure modes until weeks later when customers complain. How do I catch this before it becomes a liability nightmare?
The Real Problem Isn’t Confidence, It’s Silent Drift
You’re describing the exact failure mode that production data has just exposed: quality and token-cost drift account for 38% of agent failures across 100+ real deployments, while actual errors cause only ~1%. Your problem isn’t that the model is wrong. It’s that you’re monitoring the wrong signals and catching failures way too late.
Insurance claim filing is mission-critical. When an agent confidently files a claim with the wrong diagnosis code, miscalculates benefits, or omits a required disclosure, you don’t find out when the model fails—you find out when regulators investigate or customers sue. That’s the cost structure you’re in.
The gap between “this output looks right” and “this output will cause legal exposure” is precisely where insurance agents fail. You need deterministic gates, not just output monitoring.
Build Three Layers of Determinism Before the LLM Even Runs
First layer: schema validation at intake. Before the LLM sees a claim, extract and validate required fields using deterministic parsing. If the input is malformed or missing critical data, reject it and require human review. This catches garbage-in-garbage-out before the model wastes tokens.
from pydantic import BaseModel, validator
from typing import Optional
class InsuranceClaimInput(BaseModel):
policy_number: str
claim_type: str # "medical", "dental", "vision"
service_date: str
provider_id: str
diagnosis_codes: list[str]
amount_claimed: float
@validator("policy_number")
def validate_policy(cls, v):
# Check format, verify it exists in your system
if not v.startswith("POL"):
raise ValueError("Invalid policy format")
return v
@validator("amount_claimed")
def validate_amount(cls, v):
# Reject obviously wrong amounts
if v <= 0 or v > 1_000_000:
raise ValueError("Amount out of reasonable bounds")
return v
Second layer: post-LLM rule enforcement. After the model generates a decision, run it through a deterministic rule engine that checks for coverage rules, benefit limits, and regulatory constraints. The LLM proposes; a dumb rule engine decides.
def validate_claim_decision(decision: dict, policy: dict) -> tuple[bool, str]:
"""
Deterministic validation of LLM-generated claim decisions.
Returns (is_valid, reason_if_invalid)
"""
# Rule 1: Benefit category must match policy coverage
if decision["benefit_category"] not in policy["covered_benefits"]:
return False, f"Benefit {decision['benefit_category']} not covered"
# Rule 2: Approved amount cannot exceed remaining annual limit
remaining = policy["annual_limit"] - policy["ytd_paid"]
if decision["approved_amount"] > remaining:
return False, f"Exceeds remaining limit: ${remaining}"
# Rule 3: Diagnosis codes must be valid SNOMED CT or ICD-10
for code in decision["diagnosis_codes"]:
if not is_valid_medical_code(code):
return False, f"Invalid diagnosis code: {code}"
# Rule 4: Service date must be within policy effective period
if not (policy["effective_date"] <= decision["service_date"] <= policy["termination_date"]):
return False, "Service date outside policy period"
return True, ""
Third layer: confidence thresholding and human escalation. Don’t let the model decide at all for borderline cases. If the model’s confidence in its decision is below 0.85, or if it’s handling an edge case (unusual diagnosis, high claim amount, prior denials on the same issue), route to a human instead.
def should_escalate_to_human(decision: dict, claim: dict, policy: dict) -> bool:
"""
Determine if a claim should bypass LLM and go straight to a human reviewer.
"""
# Always escalate low-confidence decisions
if decision.get("confidence", 0) < 0.85:
return True
# Always escalate high-dollar claims
if claim["amount_claimed"] > 10_000:
return True
# Escalate if claim type is rare in training data
claim_type_frequency = get_frequency(claim["claim_type"], policy["product_line"])
if claim_type_frequency < 0.01: # Less than 1% of claims
return True
# Escalate if customer has prior denials on similar claims
prior_denials = count_prior_denials(policy["policy_number"], claim["claim_type"])
if prior_denials > 2:
return True
return False
Monitor Quality Signals That Actually Matter, Not Error Logs
You said you don’t see failures until weeks later. That’s because you’re probably monitoring exception rates and API latency, which tell you almost nothing about whether the claims are actually correct.
Instead, monitor these:
Approval rate drift. Track the percentage of claims your LLM approves, broken down by claim type and diagnosis code. If the approval rate suddenly jumps from 68% to 74% without any policy change, something is wrong. Set automated alerts on significant shifts.
Decision consistency on duplicate claims. Resubmit 2% of previously-filed claims with minor variations and compare decisions. If the model approves a claim one day and denies it the next for the same underlying diagnosis and benefit, your model is drifting.
Downstream complaint rate. Don’t wait for complaints. Measure the rate of customer calls to your support line disputing claim decisions, grouped by LLM decision type. High dispute rates on a specific diagnosis code or claim type are an early warning signal.
Manual review override rate. When humans review LLM decisions (even a sample), what percentage do they disagree with? Track this by decision type. If override rates spike above 5%, the model is degrading.
import logging
from datetime import datetime, timedelta
class ClaimQualityMonitor:
def __init__(self, alert_threshold=0.05):
self.alert_threshold = alert_threshold
def track_approval_rate(self, decision_type: str, is_approved: bool):
"""Log approval decisions for drift detection."""
logging.info(f"claim_decision decision_type={decision_type} approved={is_approved} timestamp={datetime.now().isoformat()}")
def alert_if_drifting(self, decision_type: str, window_days: int = 7):
"""Check if approval rate has drifted significantly."""
current_window = self._get_approval_rate(decision_type, days=window_days)
previous_window = self._get_approval_rate(decision_type, days=window_days * 2, start_offset=window_days)
drift = abs(current_window - previous_window)
if drift > self.alert_threshold:
logging.warning(f"ALERT: {decision_type} approval rate drifted {drift:.1%} in past {window_days} days. Current: {current_window:.1%}, Previous: {previous_window:.1%}")
return True
return False
def track_manual_override(self, llm_decision: str, human_decision: str, decision_type: str):
"""Log disagreements between LLM and human reviewers."""
if llm_decision != human_decision:
logging.warning(f"override decision_type={decision_type} llm_said={llm_decision} human_said={human_decision}")
Set Up Staged Rollout, Not Binary Production/Test
Don’t deploy an insurance agent to 100% of your claims and then wait to see what breaks. Instead, shadow a sample.
For the first month, run the LLM on 2% of incoming claims in parallel with your existing decision system. Compare decisions. Don’t use the LLM’s output yet, just observe. When you’re confident on