Where randomness meets reason

Office Hours — How do you build a hybrid local and cloud LLM architecture for regulated financial document processing? A daily developer question about AI/LLMs, answered with a direct, opinionated take. 2026-07-04T12:00:00.000Z Office Hours Office Hours office-hoursq-and-apractical-ai

Office Hours — How do you build a hybrid local and cloud LLM architecture for regulated financial document processing?

A daily developer question about AI/LLMs, answered with a direct, opinionated take.

office-hoursq-and-apractical-ai
Daily One question from the trenches, one opinionated answer.

How do you build a hybrid local and cloud LLM architecture for regulated financial document processing?

You’re facing two hard constraints that pull in opposite directions: financial regulations (SOC 2, data residency, audit trails) demand you keep sensitive documents away from third-party APIs, but frontier models live in the cloud and your local hardware probably can’t run them reliably. The answer isn’t choosing one—it’s routing tasks intelligently between them based on what each does well.

The Real Architecture Pattern

Split your pipeline into classification, extraction, and reasoning tiers. Use a local model for sensitive document classification and initial data sanitization. Route non-sensitive analysis and complex reasoning to Claude Opus 4.8 or GPT-5.5. Keep the document itself local. Send only structured queries or anonymized context to the cloud.

This isn’t about running everything locally or everything remotely. It’s about understanding what each layer needs.

Local Tier: What It Should Handle

Run Llama 4 (Scout variant, 10M context) or Gemma 3.1 Flash-Lite locally for document classification, field extraction validation, and PII detection. These models are good enough for deterministic tasks where you can verify correctness against rules or regex patterns. More importantly, your compliance team won’t need to audit cloud API logs for every page of a mortgage application.

A typical workflow: Document arrives → local model classifies document type (mortgage, tax return, bank statement) → local model masks or extracts regulated fields → structured output goes to the cloud model if needed. The actual document never leaves your infrastructure.

import ollama
import json
from pydantic import BaseModel

class DocumentClassification(BaseModel):
    doc_type: str
    extracted_fields: dict
    contains_pii: bool
    safe_for_cloud: bool

def classify_locally(pdf_text: str) -> DocumentClassification:
    # Run local model for initial triage
    response = ollama.generate(
        model="llama4-scout",
        prompt=f"""Classify this financial document and extract key fields.
Only extract fields that are safe to send to cloud APIs.
Mark if document contains PII that must stay local.

Document:
{pdf_text[:2000]}  # Truncate to avoid token bloat

Return JSON with: doc_type, extracted_fields, contains_pii, safe_for_cloud"""
    )
    
    result = json.loads(response['response'])
    return DocumentClassification(**result)

The local model acts as a gatekeeper. It’s not trying to be perfect—it’s trying to be safe and consistent.

Routing Logic: The Critical Layer

This is where your architecture lives or dies. You need explicit rules about what goes where.

  • Classification only (mortgage vs. auto loan vs. tax return): Local, always.
  • Field extraction with no PII: Local first, validate result, then optionally send to cloud if confidence is low.
  • Complex reasoning over sanitized data (risk assessment, comparison across documents): Cloud, but only with anonymized values and masked account numbers.
  • Anything involving actual account numbers, SSNs, or raw bank statements: Local only, or don’t send it at all.

The key is making the routing decision before you send anything, not after.

class HybridRouter:
    def route_task(self, doc_type: str, task: str, contains_pii: bool) -> str:
        # Deterministic routing rules
        if contains_pii and task == "complex_analysis":
            return "local_only"  # Extract, anonymize, or reject
        
        if doc_type in ["mortgage", "tax_return"] and task == "classification":
            return "local"  # Fast, deterministic
        
        if task == "risk_scoring" and not contains_pii:
            return "cloud"  # Cloud models better at reasoning
        
        # Default: safer to process locally
        return "local"

Cloud Tier: What It Gains You

Don’t try to run GPT-5.5 locally. Use the cloud for tasks where frontier model reasoning actually matters: risk assessment across multiple documents, narrative analysis of loan officer notes, or compliance rule interpretation that requires real judgment.

The move to the cloud happens after sanitization. You’re sending structured summaries, not raw documents. This satisfies both compliance (audit trail shows what left the system) and capability (frontier models get the reasoning they’re good at).

import anthropic

def risk_assessment_cloud(sanitized_data: dict) -> dict:
    # Risk data is anonymized: amounts, dates, income ratios
    # No PII leaves your infrastructure
    
    client = anthropic.Anthropic()
    
    prompt = f"""Assess lending risk based on this anonymized borrower profile:
    
    {json.dumps(sanitized_data, indent=2)}
    
    Return risk score (0-100), key factors, and recommendation."""
    
    response = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=500,
        messages=[{"role": "user", "content": prompt}]
    )
    
    return {"risk_score": response.content[0].text}

Cost and Latency Reality

Local inference on a single GPU (RTX 4090 or H100) costs nothing per request once hardware is amortized. Latency is 500ms–2s for small models. Cloud APIs cost $0.003–$0.015 per 1K tokens. For a 10,000-document batch, cloud-only would cost $30–$150. Hybrid costs hardware + cloud for reasoning tasks only, often cutting total cost by 70%.

Latency matters less for batch processing (mortgages, tax documents) but matters a lot if you’re doing real-time decisioning at loan origination. Hybrid splits the load: local classification is instant, cloud reasoning runs async in the background.

Compliance and Audit

Document what leaves your infrastructure. Create immutable logs showing: document type, classification decision (local), which fields were extracted, which were sent to cloud (sanitized), and when. Every request to the cloud API should include a trace ID that ties back to the local processing step.

Regulators don’t care if you use cloud models—they care if you can prove you didn’t send data you shouldn’t have. Logging at the routing layer is how you prove that.

The Hardest Part

It’s not technical. It’s defining what “sensitive” actually means. Tax documents have income, mortgages have property addresses, bank statements have transaction history. None of those are strict PII under GDPR or CCPA, but financial institutions treat them conservatively. Work with compliance early. Get written guidance on what can leave the building. Use that guidance in your router.

Bottom line: Build a routing layer that classifies risk before sending anything to the cloud, run deterministic tasks (classification, field extraction, validation) locally on Llama 4 or Gemma, and only send sanitized, structured data to frontier models like Claude Opus 4.8 for complex reasoning. This cuts cloud costs by 70%, satisfies audit requirements, and keeps your compliance team happy.

Question via Hacker News