Office Hours — Can trivial LLM calls be compiled into conventional data pipelines?

Can trivial LLM calls be compiled into conventional data pipelines?

The short answer: sometimes, but you’re probably asking the wrong question. The real issue isn’t whether you can compile LLM calls into SQL or Airflow—you can. It’s whether you should, and what you lose when you do.

Why the Temptation Exists

When you’re running the same prompt against a million records, it feels wasteful to call an LLM API that many times. The latency adds up. The costs add up. Your data engineering instinct says: batch it, optimize it, treat it like a traditional ETL job. That’s reasonable. But LLMs aren’t SQL operations. They don’t have the determinism or cost structure that justify that architecture.

What Actually Breaks

The first problem is determinism. A SQL query on the same input always returns the same output. An LLM call with temperature=0 probably does, but not actually. You can hit temperature drift from model updates, API routing differences, or just the inherent variance in sampling. When you’re running a million-record batch overnight, discovering that 0.3% of outputs changed because OpenAI rolled out a minor model revision is a production incident waiting to happen.

The second problem is cost opacity. Traditional pipelines have fixed unit economics: one query, one computation, one cost. LLM calls have variable output token counts. A classification task might output 10 tokens for “yes” and 500 tokens for a rambling explanation. Multiply that variance across a million records and your batch job either overruns budget or you’ve added guardrails that force the model to be terse and less useful. Neither is great.

The third problem is observability. When a SQL job fails, you get a stack trace and a query plan. When an LLM batch job fails partway through, you have a half-processed dataset, incomplete logs, and no clear signal about whether it’s a transient API issue or a systematic prompt problem that needs fixing before resuming.

When It Actually Makes Sense

Compiled LLM pipelines work in a specific case: when your task is genuinely trivial, deterministic, and you’re willing to accept some variance. Examples that work:

Classification into a fixed set of categories with structured output enforced. Run it through GPT-4.1 Nano with max_tokens=5 and a JSON schema. Cost is predictable. Failures are obvious (invalid JSON). You can batch.
Deduplication or entity matching. If you’re comparing two strings and saying “are these the same entity,” you can batch that against Gemini 3.5 Flash with a cost guardrail. The risk of hallucination is low because the task has a tight objective function.
Sentiment labeling on a fixed scale (negative/neutral/positive). Again, structured output with guardrails makes this safe to batch.

What doesn’t work: anything that requires reasoning beyond binary/ternary decisions, anything where you need to inspect individual failures, anything where token count variance matters.

A Real Example: What Breaks

Say you’re labeling a million customer support tickets as “actionable” or “not actionable.” You write a batch job:

import anthropic

client = anthropic.Anthropic()

# Batch configuration
messages = []
for ticket in tickets:
    messages.append({
        "custom_id": ticket.id,
        "params": {
            "model": "claude-opus-5",
            "max_tokens": 10,
            "system": "You are a support triage system. Respond with ONLY 'actionable' or 'not actionable'.",
            "messages": [{"role": "user", "content": ticket.text}]
        }
    })

# Submit batch
batch = client.messages.create_batch(
    requests=messages
)

This looks clean. You submit 1M requests, get back results hours later, parse them. But here’s what happens in production:

0.2% of responses include extra reasoning (“actionable because…”) even though you said “ONLY”. Now you have a parsing problem.
Your token estimate was 5 tokens per response. It’s actually 7 on average. Your $500 budget becomes $700.
A model update ships mid-batch. The first 500K requests use Claude Opus 5 behavior from Tuesday; the last 500K use Wednesday’s behavior. Your labeling consistency drops 3%.
One customer’s ticket includes a jailbreak prompt. Claude refuses it. Now you have a null response to handle.

None of these are catastrophic, but they’re death by a thousand cuts. You’ve added complexity to your data pipeline to save latency and (you thought) cost. You’ve actually traded simplicity for brittleness.

The Better Pattern

Keep LLM calls in your application layer where you can reason about individual results. If you need to process a million records, do it:

Streaming, with per-record error handling and retry logic.
With per-record cost tracking and kill switches.
Against a cheaper model (GPT-4.1 Nano, Gemini 3.5 Flash, or Claude Haiku 4.5) with explicit fallback to a more capable model on failure.

Use something like Inngest or temporal for orchestration instead of Airflow. You get better observability, per-task retry logic, and explicit error propagation without the SQL-thinking overhead.

If you really do need batch processing, use the provider’s batch API (OpenAI Batch, Anthropic Batch, Google’s batch inference). These give you cost discounts (typically 50% off) in exchange for latency. The tradeoff is explicit: you wait 24 hours, you save money. You get streaming results, you pay full price. That clarity is valuable.

The Token Efficiency Framing Matters

If your actual concern is token cost, the right lever isn’t architecture—it’s model selection and prompt optimization. Switching from Claude Opus 5 to Claude Sonnet 5 for classification cuts costs by 80% with minimal accuracy loss. Using Claude Haiku 4.5 cuts them further. That’s a 1-line code change that gets you better economics than any pipeline optimization.

Databricks benchmarked this on their million-line codebase: GLM-5.2 (open-source, cheaper) matched Claude Opus 4.8 on coding tasks at 66% lower cost per call. Not per-token. Per-task. That’s the real savings signal.

Bottom line: Don’t compile LLM calls into batch data pipelines just because it feels like “good engineering.” Keep them in your application layer where you can reason about individual results, pick a cheaper model, and optimize prompts. Use batch APIs only if latency isn’t a constraint and the 50% cost discount justifies the wait.

Question via Hacker News