Multi-Modal AI in Practice: Vision, Audio, and Document Understanding Across Every Major Provider

Multi-modal AI has moved from demo to infrastructure. Every major provider now accepts images, audio, video, and documents alongside text — but the implementations, capabilities, pricing, and failure modes differ in ways that matter for production systems. This post covers the actual mechanics of multi-modal processing across providers, the architectural patterns that work, and the integration details that documentation buries.

What Multi-Modal Actually Means at the API Level
Vision: How Image Understanding Works
Audio: Speech-to-Text, Speech-to-Speech, and Audio Understanding
Document Understanding: PDFs, Scans, and Structured Extraction
Video Understanding
Provider Comparison Matrix
Architecture Patterns for Multi-Modal Applications
Token Costs and Optimization
Failure Modes and Edge Cases
Building a Multi-Modal Pipeline
Summary
Further Reading

“Multi-modal” covers three distinct capabilities that providers implement differently:

Multi-modal input — the model accepts non-text content (images, audio, video, files) in the prompt and reasons over it alongside text
Multi-modal output — the model generates non-text content (images, audio, structured data) in its response
Cross-modal reasoning — the model connects information across modalities (describing what’s happening in a video while transcribing the audio, or referencing a chart in a document while answering a question about it)

Most current models handle (1) well, are expanding (2), and vary enormously on (3). The API surface looks similar across providers — you pass content blocks with type annotations — but the underlying tokenization, resolution handling, and context window consumption differ in ways that directly affect cost and quality.

How multi-modal models process heterogeneous inputs through modality-specific encoders into a unified token sequence.

Vision: How Image Understanding Works

The Vision Encoder Pipeline

Modern vision-language models use a two-stage architecture: a vision encoder (typically a ViT variant) processes images into patch embeddings, which are then projected into the language model’s embedding space. The language model sees these as a sequence of “visual tokens” interleaved with text tokens.

The critical implementation detail: image resolution directly determines token count, and token count determines both cost and quality.

Resolution and Token Consumption by Provider

Provider	Model	Max Resolution	Token Calculation	Tiles/Detail Modes
OpenAI	GPT-5.6 Sol	4096×4096 (auto-scaled)	85 tokens (low) or 170 tokens per 512×512 tile + 85 base (high)	`detail: low` / `detail: high` / `detail: auto`
Anthropic	Claude Opus 4.8 / Sonnet 5	7680×4320 (recommended ≤1568px on long side)	~1,600 tokens per 1MP image (varies by aspect ratio)	Automatic scaling; no manual detail toggle
Google	Gemini 3.5 Flash	7680×4320	258 tokens per tile (each tile ~768×768)	Automatic tiling; resolution-aware

OpenAI’s tiling system is the most explicit. An image in detail: high mode gets split into 512×512 tiles, each costing 170 tokens. A 2048×2048 image produces 16 tiles = 2,720 + 85 base = 2,805 tokens. In detail: low mode, the same image costs 85 tokens regardless of resolution — but the model loses the ability to read small text or identify fine details.

Anthropic’s approach scales more continuously. A 1568×1568 image costs roughly 1,600 tokens. Images larger than the recommended maximum get downscaled automatically, which means the API silently degrades quality without warning.

Passing Images to APIs

All three major providers accept images as base64-encoded data or URLs:

# OpenAI — GPT-5.6 Sol vision
from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-5.6-sol",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract all text from this receipt."},
            {
                "type": "image_url",
                "image_url": {
                    "url": "data:image/jpeg;base64,{base64_string}",
                    "detail": "high"
                }
            }
        ]
    }],
    max_tokens=1024
)

# Anthropic — Claude Sonnet 5 vision
import anthropic
client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5-20260630",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/jpeg",
                    "data": base64_string
                }
            },
            {"type": "text", "text": "Extract all text from this receipt."}
        ]
    }]
)

# Google — Gemini 3.5 Flash vision
import google.generativeai as genai
model = genai.GenerativeModel("gemini-3.5-flash")

response = model.generate_content([
    "Extract all text from this receipt.",
    {"mime_type": "image/jpeg", "data": image_bytes}
])

The Anthropic API requires image content blocks to appear before text content blocks in the message. OpenAI and Google are order-agnostic. This matters in multi-image scenarios where interleaving text instructions between images improves accuracy.

Multi-Image Reasoning

All major providers support multiple images per request. The practical limits:

Provider	Max Images per Request	Context Consumed
OpenAI GPT-5.6 Sol	No hard limit (context window bound)	Sum of all tile tokens
Claude Opus 4.8	100 images	Sum of all image tokens; ~100 images at 1MP ≈ 160K tokens
Gemini 3.5 Flash	3,600 images (video frame extraction)	258 tokens per tile per image

Multi-image reasoning quality drops measurably after ~20 images for most models. The model can reference earlier images, but cross-image spatial reasoning (e.g., “which of these 30 floor plans has the largest kitchen?”) degrades with count.

Multi-image processing pipeline: each image is independently encoded, then all tokens participate in shared attention.

Vision Accuracy: Where Models Fail

Systematic failure modes across all providers:

Small text in large images: Text below ~10px in the source image is unreliable even in high-detail mode. OCR-specific models (or preprocessing with traditional OCR) outperform vision-language models for dense small text.
Spatial reasoning: “Is object A to the left of object B?” is correct ~80% of the time. “How far apart are A and B?” is unreliable.
Counting: Counting objects above ~15 is consistently inaccurate. Models tend to estimate rather than enumerate.
Charts and graphs: Bar charts and line graphs are generally read well. Scatter plots with overlapping points, stacked area charts, and small sparklines are error-prone.
Handwriting: Varies enormously by language and legibility. English cursive is roughly 85–90% character-accurate on the best models; non-Latin scripts drop to 60–75%.

Audio: Speech-to-Text, Speech-to-Speech, and Audio Understanding

Audio Input Architectures

Audio processing in multi-modal models follows two architectures:

Pipeline (transcribe-then-reason): Audio → ASR model → text → LLM. Used by most providers for “audio understanding” features. Loses prosody, tone, speaker identity.
Native audio tokens: Audio is encoded directly into tokens that the language model processes alongside text. Preserves paralinguistic features. Used by Gemini’s native audio mode and OpenAI’s advanced voice mode.

Two architectures for audio input: pipeline (transcription first) vs. native audio tokens.

Provider Audio Capabilities

Capability	OpenAI	Anthropic	Google
Audio input (native)	GPT-5.6 Sol (advanced voice)	Not available natively; requires preprocessing	Gemini 3.5 Flash (native audio tokens)
Audio input (transcription)	Whisper v3 API	No built-in; use external ASR	Gemini ASR built-in
Audio output	TTS API (6 voices); advanced voice mode	Not available	Gemini 3.1 Flash TTS; Gemini 3.5 Live
Max audio length	25 MB per file (Whisper); ~15 min voice mode	N/A	Up to 9.5 hours (Gemini)
Supported formats	MP3, MP4, WAV, FLAC, OGG, WebM	N/A	MP3, WAV, FLAC, OGG, AAC

Google’s Gemini models have the strongest native audio understanding. They accept audio files directly in the content array and process them as first-class tokens:

# Gemini 3.5 Flash — native audio understanding
import google.generativeai as genai
model = genai.GenerativeModel("gemini-3.5-flash")

audio_file = genai.upload_file("meeting_recording.mp3")
response = model.generate_content([
    "Summarize this meeting. List action items with owners.",
    audio_file
])

Gemini’s audio token cost is roughly 32 tokens per second of audio. A 60-minute meeting recording consumes ~115,200 audio tokens — well within the context window but meaningful for cost.

Real-Time Audio and Voice

OpenAI’s Realtime API and Gemini 3.5 Live both support bidirectional streaming audio — the user speaks, the model responds with audio, with interruption handling. The architectures differ:

OpenAI Realtime API: WebSocket-based. Client sends audio chunks, receives audio chunks. Supports function calling mid-conversation. Latency target: ~300ms first-byte audio response.
Gemini Live: Server-Sent Events or WebSocket. Native multimodal — can process video frames alongside audio simultaneously. Supports Gemini 3.5 Live Translate for real-time translation.

Both require persistent connections and have different billing models (per-minute vs. per-token).

Document Understanding: PDFs, Scans, and Structured Extraction

Document understanding is where multi-modal capabilities deliver the most immediate production value. The three approaches:

1. PDF as Images (Page Rendering)

Convert each PDF page to an image, send images to the vision model. This works for any PDF regardless of structure and handles scanned documents.

# PDF-to-images pipeline using pdf2image
from pdf2image import convert_from_path
import base64, io

def pdf_to_base64_images(pdf_path, dpi=200):
    pages = convert_from_path(pdf_path, dpi=dpi)
    images = []
    for page in pages:
        buffer = io.BytesIO()
        page.save(buffer, format="PNG")
        images.append(base64.standard_b64encode(buffer.getvalue()).decode())
    return images

DPI matters. At 150 DPI, a standard letter page (8.5×11”) produces a 1275×1650 image — adequate for body text but marginal for footnotes. At 200 DPI, the same page is 1700×2200, which captures most text reliably. Going above 300 DPI rarely improves accuracy but increases token cost proportionally.

2. Native PDF Support

Anthropic and Google now accept PDF files directly:

# Claude — native PDF input
import anthropic, base64
client = anthropic.Anthropic()

with open("contract.pdf", "rb") as f:
    pdf_data = base64.standard_b64encode(f.read()).decode()

response = client.messages.create(
    model="claude-opus-4.8-20260501",
    max_tokens=4096,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "document",
                "source": {
                    "type": "base64",
                    "media_type": "application/pdf",
                    "data": pdf_data
                }
            },
            {"type": "text", "text": "Extract all parties, dates, and obligations from this contract."}
        ]
    }]
)

Native PDF processing in Claude extracts both the text layer (if present) and renders pages as images for visual elements. This dual extraction means the model can read text-selectable PDFs more cheaply (text tokens are much cheaper than image tokens) while still understanding charts, tables, and layout.

3. Hybrid Extraction

For production document pipelines, the highest-accuracy approach combines traditional extraction with multi-modal understanding:

Hybrid document processing: classify the document type, extract text and visual elements separately, then merge for LLM reasoning.

Document Understanding Accuracy

Table extraction accuracy across approaches (measured on a proprietary benchmark of 500 financial documents):

Approach	Simple Tables	Complex/Merged Cells	Nested Tables
Vision-only (image)	~92% cell accuracy	~78%	~55%
Text extraction (pdfplumber)	~98% (native PDF)	~85%	~70%
Hybrid (text + vision)	~98%	~91%	~78%
Specialized doc AI (Azure Document Intelligence)	~99%	~95%	~88%

The takeaway: for high-volume structured extraction from standardized document types (invoices, tax forms, medical records), specialized document AI services still outperform general-purpose multi-modal models. Multi-modal models win on flexibility — they handle novel document types without training.

Video Understanding

How Video Processing Works

No major provider processes video as a continuous stream of frames at the model level. Video understanding is implemented as:

Frame sampling: Extract frames at fixed intervals (typically 1–2 fps)
Audio extraction: Separate audio track processed independently
Joint reasoning: Frame images and audio/transcript tokens are interleaved in the context window

Gemini has the most mature video support, accepting video files directly and handling the frame extraction internally:

# Gemini 3.5 Flash — direct video understanding
import google.generativeai as genai
model = genai.GenerativeModel("gemini-3.5-flash")

video_file = genai.upload_file("product_demo.mp4")
# Wait for processing
video_file = genai.get_file(video_file.name)

response = model.generate_content([
    "Describe each step shown in this product demo. Note any UI elements that appear.",
    video_file
])

Gemini processes video at roughly 1 fps by default, producing 258 tokens per frame. A 10-minute video generates ~154,800 visual tokens plus audio tokens — a large but manageable portion of the context window for Gemini 3.5 Flash.

Video Token Economics

Duration	Frames (1fps)	Visual Tokens	Audio Tokens	Total Tokens
1 min	60	~15,480	~1,920	~17,400
10 min	600	~154,800	~19,200	~174,000
60 min	3,600	~928,800	~115,200	~1,044,000

A 60-minute video consumes roughly 1M tokens — feasible with Gemini’s large context windows but expensive. For cost optimization, reduce frame rate to 0.5 fps or 0.2 fps for content where visual changes are slow (lectures, interviews).

For OpenAI and Anthropic, video understanding requires manual frame extraction:

import cv2, base64

def extract_frames(video_path, fps=1):
    """Extract frames at specified fps, return as base64."""
    cap = cv2.VideoCapture(video_path)
    video_fps = cap.get(cv2.CAP_PROP_FPS)
    frame_interval = int(video_fps / fps)
    
    frames = []
    frame_count = 0
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        if frame_count % frame_interval == 0:
            _, buffer = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 85])
            frames.append(base64.b64encode(buffer).decode())
        frame_count += 1
    cap.release()
    return frames

Video processing pipeline: frames and audio are extracted separately and interleaved in the context window.

Provider Comparison Matrix

Input Modality Support

Modality	OpenAI (GPT-5.6 Sol)	Anthropic (Claude Opus 4.8 / Sonnet 5)	Google (Gemini 3.5 Flash)
Images (JPEG, PNG, WebP, GIF)	✅	✅	✅
PDF (native)	✅	✅	✅
Audio (native input)	✅ (voice mode)	❌ (external ASR required)	✅
Video (native input)	❌ (manual frames)	❌ (manual frames)	✅
Spreadsheets/CSV	❌ (text extraction)	❌ (text extraction)	✅ (Sheets integration)

Output Modality Support

Modality	OpenAI	Anthropic	Google
Text	✅	✅	✅
Structured JSON	✅ (JSON mode, structured outputs)	✅ (tool use, JSON mode)	✅ (JSON mode)
Images	✅ (DALL·E integration)	❌	✅ (Imagen integration)
Audio/Speech	✅ (TTS API, voice mode)	❌	✅ (Gemini TTS)

Quality Rankings by Task (July 2026)

Based on independent evaluations and production experience:

Task	Best	Second	Notes
OCR accuracy	Claude Opus 4.8	Gemini 3.5 Flash	Claude excels on dense/small text
Chart/graph reading	Gemini 3.5 Flash	GPT-5.6 Sol	Gemini handles complex visualizations best
Document layout understanding	Claude Opus 4.8	Gemini 3.5 Flash	Claude’s native PDF processing is strong
Video temporal reasoning	Gemini 3.5 Flash	—	Only native video provider
Audio understanding (non-English)	Gemini 3.5 Flash	GPT-5.6 Sol (via Whisper)	Gemini covers more languages natively
Multi-image comparison	GPT-5.6 Sol	Claude Opus 4.8	GPT handles many-image prompts well
Handwriting recognition	Claude Opus 4.8	GPT-5.6 Sol	Claude is more consistent on degraded handwriting

Route different content types to specialized models based on the modality and task:

Router pattern: classify incoming content by modality and route to the most cost-effective model for each type.

# Multi-modal router implementation
from enum import Enum
from dataclasses import dataclass

class Modality(Enum):
    TEXT = "text"
    IMAGE = "image"
    AUDIO = "audio"
    PDF = "pdf"
    VIDEO = "video"

@dataclass
class ModelConfig:
    provider: str
    model: str
    max_tokens: int
    cost_per_1k_input: float

ROUTING_TABLE = {
    Modality.IMAGE: ModelConfig("anthropic", "claude-opus-4.8-20260501", 4096, 0.015),
    Modality.AUDIO: ModelConfig("google", "gemini-3.5-flash", 8192, 0.000075),
    Modality.PDF: ModelConfig("anthropic", "claude-sonnet-5-20260630", 4096, 0.002),
    Modality.VIDEO: ModelConfig("google", "gemini-3.5-flash", 8192, 0.000075),
    Modality.TEXT: ModelConfig("openai", "gpt-4.1-nano", 4096, 0.0001),
}

def route_content(content_type: Modality, complexity: str = "standard"):
    """Select model based on content type and complexity."""
    config = ROUTING_TABLE[content_type]
    if complexity == "high" and content_type == Modality.PDF:
        # Upgrade to Opus for complex documents
        config = ModelConfig("anthropic", "claude-opus-4.8-20260501", 4096, 0.015)
    return config

Pattern 2: Cascade Extraction

For document processing where accuracy matters more than latency, use a cascade: fast extraction first, then targeted re-extraction of low-confidence regions.

Cascade extraction: fast model handles the full document, expensive model only processes uncertain regions.

import json
from dataclasses import dataclass

@dataclass
class ExtractionResult:
    fields: dict
    low_confidence_regions: list  # [(page, bbox, field_name)]
    
async def cascade_extract(pdf_pages: list[str], schema: dict) -> dict:
    """Two-pass extraction: fast then targeted."""
    
    # Pass 1: Fast extraction with Sonnet 5
    fast_result = await extract_with_model(
        model="claude-sonnet-5-20260630",
        pages=pdf_pages,
        prompt=f"Extract these fields: {json.dumps(schema)}. "
               f"For each field, include a confidence score (0-1).",
    )
    
    # Identify low-confidence fields
    low_conf = [
        (field, info) for field, info in fast_result.items()
        if info.get("confidence", 1.0) < 0.85
    ]
    
    if not low_conf:
        return {k: v["value"] for k, v in fast_result.items()}
    
    # Pass 2: Re-extract only uncertain fields with Opus
    targeted_result = await extract_with_model(
        model="claude-opus-4.8-20260501",
        pages=pdf_pages,
        prompt=f"Focus specifically on these fields which were unclear: "
               f"{[f[0] for f in low_conf]}. Look very carefully at the document.",
    )
    
    # Merge results
    final = {k: v["value"] for k, v in fast_result.items()}
    for field, info in targeted_result.items():
        if field in dict(low_conf):
            final[field] = info["value"]
    
    return final

Standard RAG indexes text chunks. Multi-modal RAG indexes images, tables, and diagrams alongside text, retrieving the right modality for each query.

Multi-modal RAG: index text and visual elements separately, retrieve both modalities, pass to a vision-language model.

The key implementation detail is the indexing pipeline. For each document:

Extract text chunks (standard chunking strategies)
Extract images, charts, and tables as separate items
Generate text descriptions of visual elements using a vision model
Embed both the original text chunks and the generated descriptions
Store the original visual content alongside the embeddings for retrieval

from dataclasses import dataclass

@dataclass
class MultiModalChunk:
    content_type: str  # "text", "image", "table"
    text: str  # Original text or generated description
    image_data: bytes | None  # Original image if visual
    page_number: int
    bbox: tuple | None  # Bounding box on page
    embedding: list[float] | None = None

async def index_document(pdf_path: str) -> list[MultiModalChunk]:
    """Extract and index all modalities from a document."""
    chunks = []
    
    # Text chunks
    text_chunks = extract_text_chunks(pdf_path)
    for tc in text_chunks:
        chunks.append(MultiModalChunk(
            content_type="text", text=tc.text, image_data=None,
            page_number=tc.page, bbox=None
        ))
    
    # Visual elements
    visual_elements = extract_visual_elements(pdf_path)  # tables, charts, figures
    for ve in visual_elements:
        # Generate description using vision model
        description = await describe_visual(ve.image_data)
        chunks.append(MultiModalChunk(
            content_type=ve.type, text=description,
            image_data=ve.image_data, page_number=ve.page, bbox=ve.bbox
        ))
    
    # Embed all chunks (using their text/description)
    texts = [c.text for c in chunks]
    embeddings = await embed_batch(texts)
    for chunk, emb in zip(chunks, embeddings):
        chunk.embedding = emb
    
    return chunks

At retrieval time, when a chunk with content_type="image" or content_type="table" is retrieved, pass the original image_data as an image content block to the LLM, not just the text description. The description is for embedding similarity; the original image is for accurate reasoning.

For complex documents where the model needs to decide what to examine more closely — financial reports with appendices, technical manuals with cross-references, medical records with images.

Agentic document processing: the model uses tools to zoom into regions, extract data, and cross-reference sections.

This pattern uses tool calling (function calling) to let the model request specific pages, zoom into regions, or cross-reference sections. The model receives a low-resolution overview of the full document, then uses tools to request high-resolution views of specific areas.

Token Costs and Optimization

Image Cost Comparison

Cost to process a single 1024×1024 image:

Provider/Model	Tokens Consumed	Approximate Cost
GPT-5.6 Sol (high detail)	~765 tokens (4 tiles × 170 + 85)	Varies by pricing tier
GPT-4.1 Nano (high detail)	~765 tokens	Lowest cost option for vision
Claude Sonnet 5	~1,000 tokens	Cost depends on introductory vs. standard pricing
Claude Opus 4.8	~1,000 tokens	Highest per-image cost
Gemini 3.5 Flash	~516 tokens (2 tiles)	Lowest per-token cost

Optimization Strategies

1. Resolution-aware preprocessing: Resize images to the minimum resolution that preserves the information the model needs. For text extraction, 150 DPI is usually sufficient. For fine visual details, 200–300 DPI.

from PIL import Image

def optimize_image_for_api(image_path: str, task: str = "general") -> bytes:
    """Resize image based on task requirements."""
    img = Image.open(image_path)
    
    # Target sizes by task
    targets = {
        "general": (1024, 1024),
        "ocr": (2048, 2048),      # Need higher res for text
        "classification": (512, 512),  # Low res sufficient
        "thumbnail": (256, 256),   # Minimal token usage
    }
    
    max_size = targets.get(task, (1024, 1024))
    img.thumbnail(max_size, Image.LANCZOS)
    
    # Use JPEG at 85% quality — good compression, minimal quality loss
    buffer = io.BytesIO()
    img.save(buffer, format="JPEG", quality=85)
    return buffer.getvalue()

2. Detail mode selection (OpenAI-specific): Use detail: low for classification tasks, image captioning, or content moderation. Use detail: high only when the model needs to read text or identify fine details. This alone can reduce image token costs by 90%+.

3. Frame rate reduction for video: Most video understanding tasks work well at 0.5 fps or even 0.2 fps. A lecture recorded at 0.2 fps (one frame every 5 seconds) captures slide changes while using 5× fewer tokens than 1 fps.

4. Batch document processing: For high-volume document processing, use provider-specific batch APIs that offer reduced pricing:

# OpenAI Batch API for document processing
# 50% cost reduction, 24-hour completion window
batch_requests = []
for i, doc_images in enumerate(documents):
    batch_requests.append({
        "custom_id": f"doc-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4.1-nano",  # Budget model for batch
            "messages": [{
                "role": "user",
                "content": [
                    {"type": "text", "text": extraction_prompt},
                    *[{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img}", "detail": "high"}} 
                      for img in doc_images]
                ]
            }],
            "max_tokens": 2048
        }
    })

Failure Modes and Edge Cases

Hallucinated Text in Images

All vision models occasionally “read” text that isn’t there, especially in:

Low-contrast regions near actual text
Watermarked or textured backgrounds
Areas where text has been redacted or obscured

Mitigation: cross-validate extracted text against traditional OCR (Tesseract, Azure Document Intelligence) for high-stakes applications.

Coordinate and Bounding Box Inaccuracy

When asked to provide bounding boxes or pixel coordinates for objects in images, models are approximate at best. Typical error margins are 5–15% of the image dimension. For applications requiring precise localization (UI testing, automated clicking), use dedicated object detection models (YOLO, Grounding DINO) or Gemini’s computer use capability, which was specifically trained for UI element localization.

For precise localization, combine VLM semantic understanding with dedicated detector accuracy.

Audio-Visual Sync Issues

When processing video with both visual and audio content, models can misattribute what’s said to the wrong visual context. A speaker discussing “the chart on the next slide” while the current frame shows a different chart leads to incorrect associations. Mitigation: include timestamps in frame labels and instruct the model to align audio timestamps with frame timestamps.

Token Limit Surprises

Image token consumption is often larger than expected. A 10-page PDF with moderate graphics can consume 20,000–40,000 tokens in image form — before any text instructions. For long documents, this leaves less room for complex reasoning in the output.

Monitor actual token usage (not just estimated) and implement circuit breakers:

def estimate_multimodal_tokens(images: list, text: str, provider: str) -> int:
    """Estimate total token count before API call."""
    text_tokens = len(text) // 4  # rough approximation
    
    if provider == "openai":
        # High detail: 170 per 512x512 tile + 85 base per image
        image_tokens = sum(
            count_openai_tiles(img) * 170 + 85
            for img in images
        )
    elif provider == "anthropic":
        # ~1.6 tokens per pixel / 1000 (roughly)
        image_tokens = sum(
            int(img.width * img.height * 1.6 / 1000)
            for img in images
        )
    elif provider == "google":
        image_tokens = sum(
            count_gemini_tiles(img) * 258
            for img in images
        )
    
    return text_tokens + image_tokens

# Use before API calls
estimated = estimate_multimodal_tokens(images, prompt, "anthropic")
if estimated > 180_000:  # leave room for output
    # Reduce resolution or split into multiple calls
    images = [downsample(img, factor=2) for img in images]

Format-Specific Gotchas

HEIC/HEIF images: Not supported by any provider. Convert to JPEG or PNG before sending.
Animated GIFs: OpenAI processes only the first frame. Gemini processes multiple frames. Claude processes the first frame.
SVG: Not supported as image input. Render to PNG first.
TIFF: Limited support. Convert to PNG.
Audio in video: OpenAI and Anthropic ignore audio tracks when processing video frames as images. Only Gemini extracts and processes audio from video files.
Transparent PNGs: Models sometimes hallucinate content in transparent regions, interpreting the transparency as white or the application’s background color.

A production multi-modal pipeline for document processing that handles mixed content types, optimizes costs, and provides structured output:

End-to-end multi-modal processing pipeline with cost estimation, routing, and validation.

import asyncio
from pathlib import Path

class MultiModalPipeline:
    """Production pipeline for processing mixed-modality documents."""
    
    def __init__(self):
        self.providers = {
            "anthropic": AnthropicClient(),
            "openai": OpenAIClient(),
            "google": GeminiClient(),
        }
    
    async def process(self, file_path: str, task: str, schema: dict) -> dict:
        path = Path(file_path)
        modality = self._classify(path)
        
        # Preprocess based on modality
        content = await self._preprocess(path, modality, task)
        
        # Estimate tokens and select model
        estimated_tokens = self._estimate_tokens(content, modality)
        model_config = self._select_model(modality, estimated_tokens, task)
        
        # Process with selected model
        result = await self._call_model(model_config, content, task, schema)
        
        # Validate output against schema
        validated = self._validate(result, schema)
        
        return validated
    
    def _classify(self, path: Path) -> Modality:
        suffix_map = {
            ".pdf": Modality.PDF,
            ".jpg": Modality.IMAGE, ".jpeg": Modality.IMAGE,
            ".png": Modality.IMAGE, ".webp": Modality.IMAGE,
            ".mp3": Modality.AUDIO, ".wav": Modality.AUDIO,
            ".mp4": Modality.VIDEO, ".webm": Modality.VIDEO,
        }
        return suffix_map.get(path.suffix.lower(), Modality.TEXT)
    
    def _select_model(self, modality: Modality, tokens: int, task: str):
        """Cost-aware model selection."""
        if modality == Modality.VIDEO:
            return ("google", "gemini-3.5-flash")  # Only native video
        
        if modality == Modality.AUDIO:
            return ("google", "gemini-3.5-flash")  # Best native audio
        
        if task == "ocr" and tokens < 50_000:
            return ("anthropic", "claude-sonnet-5-20260630")  # Good OCR, lower cost
        
        if task == "complex_extraction":
            return ("anthropic", "claude-opus-4.8-20260501")  # Best accuracy
        
        # Default: balance cost and quality
        if tokens > 100_000:
            return ("google", "gemini-3.5-flash")  # Handles large contexts cheaply
        
        return ("anthropic", "claude-sonnet-5-20260630")

Handling Multi-Page Documents at Scale

For batch processing thousands of documents, the pipeline needs queuing, retries, and cost tracking:

import asyncio
from collections import defaultdict

class BatchDocumentProcessor:
    def __init__(self, max_concurrent: int = 10):
        self.semaphore = asyncio.Semaphore(max_concurrent)
        self.costs = defaultdict(float)
        self.pipeline = MultiModalPipeline()
    
    async def process_batch(self, files: list[str], task: str, schema: dict):
        tasks = [
            self._process_with_tracking(f, task, schema)
            for f in files
        ]
        results = await asyncio.gather(*tasks, return_exceptions=True)
        
        # Separate successes from failures
        successes = []
        failures = []
        for file, result in zip(files, results):
            if isinstance(result, Exception):
                failures.append((file, str(result)))
            else:
                successes.append((file, result))
        
        return {
            "results": successes,
            "failures": failures,
            "costs": dict(self.costs),
        }
    
    async def _process_with_tracking(self, file: str, task: str, schema: dict):
        async with self.semaphore:
            for attempt in range(3):  # Retry logic
                try:
                    result = await self.pipeline.process(file, task, schema)
                    # Track costs per provider
                    provider = result.get("_provider", "unknown")
                    self.costs[provider] += result.get("_cost", 0)
                    return result
                except RateLimitError:
                    await asyncio.sleep(2 ** attempt)
                except Exception as e:
                    if attempt == 2:
                        raise

For self-hosted deployments, several open-weight models provide competitive multi-modal capabilities:

Model	Parameters	Modalities	Hosting Requirements	Strengths
Qwen3.6-27B	27B	Vision + Text	1× A100 80GB or 2× A6000	Strong OCR, multilingual
Llama 4 Scout	~109B MoE	Vision + Text	Multi-GPU (4× A100)	10M context, document processing
DeepSeek-VL2	MoE	Vision + Text	2× A100	Chart understanding

Running open-source multi-modal models with vLLM:

# Serve Qwen multi-modal model with vLLM
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3.6-27B \
    --tensor-parallel-size 2 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code

Run vLLM ≥ 0.24.0: earlier versions carry an unauthenticated RCE in the multimodal video path (CVE-2026-22778, CVSS 9.8) and a ReDoS in structured outputs fixed in 0.24.0.

The quality gap between open-source and frontier multi-modal models is narrower for vision tasks than for pure text reasoning. For OCR and basic image understanding, models like Qwen3.6-27B achieve 85–90% of frontier model performance at a fraction of the cost when self-hosted.

Deployment decision: API for best quality, self-hosted for cost/privacy, hybrid for the best of both.

Summary

Multi-modal AI in production is less about any single model’s vision or audio capabilities and more about the engineering around it: preprocessing inputs to minimize token waste, routing content to the right model for each modality, validating outputs, and handling the failure modes that emerge when models process real-world documents, images, and audio.

Key technical points:

Image tokens are expensive and resolution-dependent. Always preprocess images to the minimum resolution that preserves task-relevant information. OpenAI’s detail: low mode saves 90%+ tokens when fine detail isn’t needed.
Native PDF processing (Anthropic, Google) extracts both text layers and visual elements, producing better results than image-only approaches at lower token cost for text-heavy documents.
Gemini 3.5 Flash is the only provider with native video and audio processing. For OpenAI and Anthropic, video requires manual frame extraction, and Anthropic has no native audio input.
Cascade extraction (fast model first, expensive model for low-confidence regions) reduces cost by 40–60% compared to using the best model for everything, with minimal quality loss.
Multi-modal RAG requires indexing visual elements with generated text descriptions for embedding, but passing original images to the LLM at retrieval time.
Open-source multi-modal models (Qwen3.6, Llama 4 Scout) close 85–90% of the gap to frontier models on vision tasks, making self-hosting viable for high-volume, privacy-sensitive workloads.
Cross-validate vision model OCR with traditional OCR for high-stakes text extraction. Vision models hallucinate text in low-contrast regions and near redactions.