Where randomness meets reason

Office Hours — How do you design prompts and workflows that reliably work across different LLM models? A daily developer question about AI/LLMs, answered with a direct, opinionated take. 2026-07-22T12:00:00.000Z Office Hours Office Hours office-hoursq-and-apractical-ai

Office Hours — How do you design prompts and workflows that reliably work across different LLM models?

A daily developer question about AI/LLMs, answered with a direct, opinionated take.

office-hoursq-and-apractical-ai
Daily One question from the trenches, one opinionated answer.

How do you design prompts and workflows that reliably work across different LLM models?

The honest answer: you can’t design a prompt that works identically across every model, but you can architect systems that degrade gracefully when models change. The frontier has shifted from “write one perfect prompt” to “build routing and fallback logic that survives model volatility.”

The Core Problem

Model behavior is not transitive. A prompt that works reliably on Claude Opus 4.8 might produce different output on Gemini 3.5 Flash or GPT-5.6 Sol, even when all three are capable of the task. The differences come from training data, inference-time optimization trade-offs, and architectural choices you can’t see. Databricks discovered this the hard way: their eval suite that gave Claude Opus 4.8 a 92% success rate on production code tasks scored GPT-5.5 at 87% on identical work, not because GPT-5.5 was weaker, but because it reasons differently about their codebase patterns.

The real issue isn’t finding a universal prompt. It’s that you need to know which model is best for your specific task, and that answer changes as new models ship.

Build Model-Agnostic Task Definitions, Not Model-Specific Prompts

Stop writing prompts. Write task specifications with measurable inputs and outputs. A prompt is temporary; a task definition is structural.

Here’s a concrete pattern:

from dataclasses import dataclass
from enum import Enum

class TaskModel(Enum):
    FAST = "gemini-3.5-flash"
    BALANCED = "claude-sonnet-5"
    BEST = "claude-opus-4.8"

@dataclass
class ExtractionTask:
    """Task definition, not a prompt."""
    input_text: str
    fields_to_extract: list[str]
    required_fields: list[str]
    confidence_threshold: float = 0.8
    
    def validate_output(self, extracted: dict) -> bool:
        """Objective verification, not vibe-checking."""
        missing = set(self.required_fields) - set(extracted.keys())
        return len(missing) == 0 and extracted.get("_confidence", 0) >= self.confidence_threshold

# Call different models with the SAME task definition
task = ExtractionTask(
    input_text=doc_text,
    fields_to_extract=["name", "date", "amount"],
    required_fields=["name", "amount"]
)

# Try fast model first
result = call_model(TaskModel.FAST, task)
if not task.validate_output(result):
    # Escalate silently
    result = call_model(TaskModel.BALANCED, task)

This pattern does three things: separates intent (the task) from implementation (the model), makes success objective, and enables cost-aware routing. You’re not rewriting prompts per model. You’re defining what success looks like once, then letting models compete to achieve it.

Use Model-Specific Adapters, Not Universal Prompts

When you do need to call different models, wrap them in adapters that translate your task definition into model-appropriate prompts. Each model gets a template tuned to its strengths.

class ModelAdapter(ABC):
    @abstractmethod
    def format_extraction_prompt(self, task: ExtractionTask) -> str:
        pass

class ClaudeAdapter(ModelAdapter):
    def format_extraction_prompt(self, task: ExtractionTask) -> str:
        # Claude Opus responds well to structured XML and explicit JSON in examples
        return f"""Extract the following fields from the document.
Required fields: {', '.join(task.required_fields)}

<document>
{task.input_text}
</document>

Output valid JSON only:
{{"extracted": {{ ... }}, "confidence": 0.95}}"""

class GeminiAdapter(ModelAdapter):
    def format_extraction_prompt(self, task: ExtractionTask) -> str:
        # Gemini 3.5 Flash is faster with concise, direct prompts
        return f"""Extract: {', '.join(task.fields_to_extract)}

Document:
{task.input_text}

JSON response:"""

# Usage
adapter = ClaudeAdapter() if model == "opus" else GeminiAdapter()
prompt = adapter.format_extraction_prompt(task)

The key insight: the task stays constant. Only the wrapping changes. This is how Databricks handles multi-model routing in production—they define task semantics once, then maintain model-specific prompt variants that all target the same success criteria.

Test Against Your Actual Data, Not Benchmarks

Frontier benchmarks are broken (SWE-Bench Pro itself has ~30% corrupt tasks per OpenAI’s own analysis). Build a minimal eval on your workload.

def eval_model_on_task(model: str, task_batch: list[ExtractionTask], n_samples: int = 50) -> dict:
    """Real-world eval, not benchmark theater."""
    results = {"successes": 0, "failures": 0, "cost": 0.0, "latency": []}
    
    for task in task_batch[:n_samples]:
        start = time.time()
        response = call_model(model, task)
        latency = time.time() - start
        
        is_valid = task.validate_output(response)
        results["successes" if is_valid else "failures"] += 1
        results["cost"] += estimate_tokens(response) * get_model_price(model)
        results["latency"].append(latency)
    
    return {
        "model": model,
        "success_rate": results["successes"] / n_samples,
        "avg_latency_ms": sum(results["latency"]) / len(results["latency"]) * 1000,
        "cost_per_success": results["cost"] / results["successes"],
    }

# Run once, use results for routing decisions
for model in [TaskModel.FAST, TaskModel.BALANCED, TaskModel.BEST]:
    print(eval_model_on_task(model, your_pdfs))

Run this on 50-100 real examples from your pipeline. You’ll find which model actually wins on your data, not which one wins on Abstract Code Arena or whatever benchmark is trending. Kimi K3 might beat Claude on Code Arena Frontend but lag on your finance documents. You won’t know until you test.

Route Based on Cost and Task Complexity, Not Model Prestige

Daily Signal coverage from July 20 highlighted that orchestration layers, not model selection, drive cost. A three-tier routing strategy beats picking one model and sticking with it:

  1. Fast tier (Gemini 3.5 Flash): Simple classification, summaries under 500 tokens, tasks with high success margins.
  2. Balanced tier (Claude Sonnet 5): Moderate reasoning, multi-step extraction, edge cases from tier 1.
  3. Best tier (Claude Opus 4.8 or Fable 5): Complex reasoning, high-stakes decisions, tasks tier 2 failed.

Don’t start with Opus. Start with Flash. Escalate only when necessary. The Daily Signal from July 20 found that orchestration beats token optimization by a factor of 2-3x on the same workload.

def smart_route(task: ExtractionTask) -> str:
    """Route based on task difficulty, not prestige."""
    complexity = len(task.fields_to_extract) + len(task.required_fields)
    
    if complexity <= 3:
        return TaskModel.FAST  # Gemini Flash handles this fine
    elif complexity <= 6:
        return TaskModel.BALANCED  # Sonnet for moderate work
    else:
        return TaskModel.BEST  # Only Opus for the hard stuff

Document Your Failure Cases

Every model fails differently. Claude hallucinates confidently on dates. GPT-5.6 Sol over-extracts fields it thinks might be relevant. Gemini 3.5 Flash misses rare formats. Document these in comments tied to your validation logic.

def validate_extraction(extracted: dict, task: ExtractionTask) -> tuple[bool, str]:
    """Return success and a reason (for

*Question via [Hacker News](https://news.ycombinator.com/item?id=48478162)*