Office Hours — We're using LLMs to file insurance claims at scale in production. The model is confident about its decisions, but we have no visibility into failure modes until weeks later when customers complain. How do I catch this before it becomes a liability nightmare? A daily developer question about AI/LLMs, answered with a direct, opinionated take. 2026-11-19T12:00:00.000Z Office Hours Office Hours office-hoursq-and-apractical-ai

Office Hours — We're using LLMs to file insurance claims at scale in production. The model is confident about its decisions, but we have no visibility into failure modes until weeks later when customers complain. How do I catch this before it becomes a liability nightmare?

A daily developer question about AI/LLMs, answered with a direct, opinionated take.

Daily One question from the trenches, one opinionated answer.

We’re using LLMs to file insurance claims at scale in production. The model is confident about its decisions, but we have no visibility into failure modes until weeks later when customers complain. How do I catch this before it becomes a liability nightmare?

The Real Problem Isn’t Confidence, It’s Silent Drift

You’re describing the exact failure mode that production data has just exposed: quality and token-cost drift account for 38% of agent failures across 100+ real deployments, while actual errors cause only ~1%. Your problem isn’t that the model is wrong. It’s that you’re monitoring the wrong signals and catching failures way too late.

Insurance claim filing is mission-critical. When an agent confidently files a claim with the wrong diagnosis code, miscalculates benefits, or omits a required disclosure, you don’t find out when the model fails—you find out when regulators investigate or customers sue. That’s the cost structure you’re in.

The gap between “this output looks right” and “this output will cause legal exposure” is precisely where insurance agents fail. You need deterministic gates, not just output monitoring.

Build Three Layers of Determinism Before the LLM Even Runs

First layer: schema validation at intake. Before the LLM sees a claim, extract and validate required fields using deterministic parsing. If the input is malformed or missing critical data, reject it and require human review. This catches garbage-in-garbage-out before the model wastes tokens.

from pydantic import BaseModel, validator
from typing import Optional

class InsuranceClaimInput(BaseModel):
    policy_number: str
    claim_type: str  # "medical", "dental", "vision"
    service_date: str
    provider_id: str
    diagnosis_codes: list[str]
    amount_claimed: float
    
    @validator("policy_number")
    def validate_policy(cls, v):
        # Check format, verify it exists in your system
        if not v.startswith("POL"):
            raise ValueError("Invalid policy format")
        return v
    
    @validator("amount_claimed")
    def validate_amount(cls, v):
        # Reject obviously wrong amounts
        if v <= 0 or v > 1_000_000:
            raise ValueError("Amount out of reasonable bounds")
        return v

Second layer: post-LLM rule enforcement. After the model generates a decision, run it through a deterministic rule engine that checks for coverage rules, benefit limits, and regulatory constraints. The LLM proposes; a dumb rule engine decides.

def validate_claim_decision(decision: dict, policy: dict) -> tuple[bool, str]:
    """
    Deterministic validation of LLM-generated claim decisions.
    Returns (is_valid, reason_if_invalid)
    """
    # Rule 1: Benefit category must match policy coverage
    if decision["benefit_category"] not in policy["covered_benefits"]:
        return False, f"Benefit {decision['benefit_category']} not covered"
    
    # Rule 2: Approved amount cannot exceed remaining annual limit
    remaining = policy["annual_limit"] - policy["ytd_paid"]
    if decision["approved_amount"] > remaining:
        return False, f"Exceeds remaining limit: ${remaining}"
    
    # Rule 3: Diagnosis codes must be valid SNOMED CT or ICD-10
    for code in decision["diagnosis_codes"]:
        if not is_valid_medical_code(code):
            return False, f"Invalid diagnosis code: {code}"
    
    # Rule 4: Service date must be within policy effective period
    if not (policy["effective_date"] <= decision["service_date"] <= policy["termination_date"]):
        return False, "Service date outside policy period"
    
    return True, ""

Third layer: confidence thresholding and human escalation. Don’t let the model decide at all for borderline cases. If the model’s confidence in its decision is below 0.85, or if it’s handling an edge case (unusual diagnosis, high claim amount, prior denials on the same issue), route to a human instead.

def should_escalate_to_human(decision: dict, claim: dict, policy: dict) -> bool:
    """
    Determine if a claim should bypass LLM and go straight to a human reviewer.
    """
    # Always escalate low-confidence decisions
    if decision.get("confidence", 0) < 0.85:
        return True
    
    # Always escalate high-dollar claims
    if claim["amount_claimed"] > 10_000:
        return True
    
    # Escalate if claim type is rare in training data
    claim_type_frequency = get_frequency(claim["claim_type"], policy["product_line"])
    if claim_type_frequency < 0.01:  # Less than 1% of claims
        return True
    
    # Escalate if customer has prior denials on similar claims
    prior_denials = count_prior_denials(policy["policy_number"], claim["claim_type"])
    if prior_denials > 2:
        return True
    
    return False

Monitor Quality Signals That Actually Matter, Not Error Logs

You said you don’t see failures until weeks later. That’s because you’re probably monitoring exception rates and API latency, which tell you almost nothing about whether the claims are actually correct.

Instead, monitor these:

Approval rate drift. Track the percentage of claims your LLM approves, broken down by claim type and diagnosis code. If the approval rate suddenly jumps from 68% to 74% without any policy change, something is wrong. Set automated alerts on significant shifts.

Decision consistency on duplicate claims. Resubmit 2% of previously-filed claims with minor variations and compare decisions. If the model approves a claim one day and denies it the next for the same underlying diagnosis and benefit, your model is drifting.

Downstream complaint rate. Don’t wait for complaints. Measure the rate of customer calls to your support line disputing claim decisions, grouped by LLM decision type. High dispute rates on a specific diagnosis code or claim type are an early warning signal.

Manual review override rate. When humans review LLM decisions (even a sample), what percentage do they disagree with? Track this by decision type. If override rates spike above 5%, the model is degrading.

import logging
from datetime import datetime, timedelta

class ClaimQualityMonitor:
    def __init__(self, alert_threshold=0.05):
        self.alert_threshold = alert_threshold
    
    def track_approval_rate(self, decision_type: str, is_approved: bool):
        """Log approval decisions for drift detection."""
        logging.info(f"claim_decision decision_type={decision_type} approved={is_approved} timestamp={datetime.now().isoformat()}")
    
    def alert_if_drifting(self, decision_type: str, window_days: int = 7):
        """Check if approval rate has drifted significantly."""
        current_window = self._get_approval_rate(decision_type, days=window_days)
        previous_window = self._get_approval_rate(decision_type, days=window_days * 2, start_offset=window_days)
        
        drift = abs(current_window - previous_window)
        if drift > self.alert_threshold:
            logging.warning(f"ALERT: {decision_type} approval rate drifted {drift:.1%} in past {window_days} days. Current: {current_window:.1%}, Previous: {previous_window:.1%}")
            return True
        return False
    
    def track_manual_override(self, llm_decision: str, human_decision: str, decision_type: str):
        """Log disagreements between LLM and human reviewers."""
        if llm_decision != human_decision:
            logging.warning(f"override decision_type={decision_type} llm_said={llm_decision} human_said={human_decision}")

Set Up Staged Rollout, Not Binary Production/Test

Don’t deploy an insurance agent to 100% of your claims and then wait to see what breaks. Instead, shadow a sample.

For the first month, run the LLM on 2% of incoming claims in parallel with your existing decision system. Compare decisions. Don’t use the LLM’s output yet, just observe. When you’re confident on