How token-aware rate limiting works for LLM APIs, why request-count limits fail, and implementation patterns across API gateways and proxy layers.

Orchestrating Long-Running AI Workflows: Durable Execution, Checkpointing, and Human-in-the-Loop for AI Pipelines

Every AI workflow that matters eventually outgrows a single request-response cycle. A document processing pipeline that extracts, classifies, summarizes, and routes content across multiple models. A code migration agent that analyzes a codebase over hours, proposes changes, waits for human approval, then applies them. A research workflow that spawns parallel searches, synthesizes results, and iterates until a quality threshold is met. These workflows run for minutes, hours, or days — and the standard approaches to building them (background jobs, task queues, cron-triggered scripts) fail in predictable and painful ways.

Durable execution frameworks solve this by treating workflow state as a first-class concern: every step is checkpointed, every failure is recoverable, and the orchestration logic reads like straightforward code rather than a state machine bolted onto a message broker.

Table of Contents

• Why Traditional Job Queues Break for AI Workloads
• Durable Execution: The Core Abstraction
• The Framework Landscape: Temporal, Inngest, Trigger.dev, and Others
• Workflow Patterns for AI Pipelines
• Checkpointing LLM State
• Human-in-the-Loop Patterns
• Error Handling and Retry Strategies for LLM Calls
• Cost Control in Long-Running Workflows
• Observability for AI Workflows
• Production Architecture
• Summary
• Further Reading

Why Traditional Job Queues Break for AI Workloads {#why-traditional-job-queues-break-for-ai-workloads}

A typical background job (Celery, Sidekiq, BullMQ) assumes tasks complete in seconds to low minutes. AI workflows violate this assumption in several ways:

Duration. A Claude Fable 5.1 agentic coding session can run for 30+ minutes. A multi-document research pipeline with iterative retrieval and synthesis can run for hours. Worker timeouts, connection limits, and deployment cycles all conspire against long-running tasks.

Non-determinism. The same LLM call with identical inputs can produce different outputs. A workflow that retries a failed step by replaying from the beginning may get different intermediate results, invalidating downstream state.

Cost of re-execution. If a 15-step pipeline fails on step 14 and the only recovery option is replaying from step 1, the cost — both in tokens and wall-clock time — is prohibitive. At $10/M input tokens for Fable 5.1 or GPT-6 Astra, re-executing 13 successful LLM calls because of one transient failure burns real money.

Branching and parallelism. AI workflows frequently fan out (run multiple models in parallel, search multiple sources concurrently) and fan in (aggregate results, pick the best output). Task queues can model this with chord/group primitives, but the coordination logic becomes brittle fast.

Human gates. Many production AI workflows require human approval at critical junctures — before applying code changes, before sending customer communications, before committing financial transactions. A workflow must be able to pause indefinitely, persist its state, and resume when a human acts.

Traditional job queues and AI workflow requirements collide at predictable failure modes that durable execution addresses.

Durable Execution: The Core Abstraction {#durable-execution-the-core-abstraction}

Durable execution is a programming model where workflow code executes like normal sequential code, but the runtime transparently persists the result of each step. If the process crashes, the runtime replays the workflow from the last checkpoint, skipping steps whose results were already recorded.

The key insight: the workflow function itself is the source of truth for orchestration logic. No separate state machine definition, no DAG configuration file, no visual workflow builder. The code is the workflow.

Here is the mental model:

The durable execution loop: code executes, the runtime persists step results, and on failure, replay skips already-completed steps.

Three properties make this work for AI:

1. Deterministic replay. The runtime replays the workflow function but substitutes recorded results for completed steps instead of re-executing them. This means the workflow doesn’t re-call an LLM for a step that already succeeded — it uses the cached response.

2. Indefinite duration. Workflows can sleep for hours or days (waiting for a human approval, for example) without consuming a worker thread. The state lives in durable storage, and the workflow resumes when its wake condition is met.

3. Automatic retries with backoff. Each step can have its own retry policy. A rate-limited LLM API call (HTTP 429) retries with exponential backoff; a validation failure doesn’t retry at all.

The Framework Landscape: Temporal, Inngest, Trigger.dev, and Others {#the-framework-landscape}

Framework	Language Support	Hosting Model	State Storage	AI-Specific Features	Pricing Model
Temporal	Go, Java, Python, TypeScript, .NET	Self-hosted or Temporal Cloud	Cassandra, PostgreSQL, MySQL	None built-in; strong primitives	Cloud: per-action pricing
Inngest	TypeScript, Python, Go	Managed cloud or self-hosted	Built-in (managed)	AI middleware, streaming step support	Free tier + usage-based
Trigger.dev	TypeScript	Managed cloud or self-hosted	Built-in (managed)	AI SDK integrations, realtime logs	Free tier + usage-based
Restate	TypeScript, Java, Kotlin, Go, Python	Self-hosted or Restate Cloud	RocksDB-backed embedded	Virtual objects for stateful agents	Open source (BSL)
Hatchet	TypeScript, Python, Go	Self-hosted or managed	PostgreSQL	Concurrency control, rate limiting	Open source + managed

Temporal

The most mature durable execution framework. Temporal models workflows as functions that call activities (individual steps). Each activity execution is recorded in an event history. On replay, the workflow function re-executes, but activity calls return their previously recorded results instead of running again.

Temporal’s strength for AI workloads is its expressiveness: workflows are real code with loops, conditionals, and error handling. Its weakness is operational complexity — running Temporal requires a cluster (server + database + workers), and the learning curve for its replay semantics is steep.

# Temporal workflow for a multi-model document processing pipeline
from temporalio import workflow, activity
from datetime import timedelta

@activity.defn
async def extract_text(document_url: str) -> str:
    # OCR or PDF extraction
    ...

@activity.defn
async def classify_document(text: str) -> str:
    # LLM call to classify document type
    response = await anthropic.messages.create(
        model="claude-sonnet-5",
        messages=[{"role": "user", "content": f"Classify this document:\n{text[:4000]}"}]
    )
    return response.content[0].text

@activity.defn
async def summarize(text: str, doc_type: str) -> str:
    # LLM call with type-specific prompt
    ...

@activity.defn
async def route_document(summary: str, doc_type: str) -> str:
    # Determine destination based on classification
    ...

@workflow.defn
class DocumentPipeline:
    @workflow.run
    async def run(self, document_url: str) -> dict:
        text = await workflow.execute_activity(
            extract_text, document_url,
            start_to_close_timeout=timedelta(minutes=5),
            retry_policy=RetryPolicy(maximum_attempts=3)
        )
        
        doc_type = await workflow.execute_activity(
            classify_document, text,
            start_to_close_timeout=timedelta(minutes=2),
            retry_policy=RetryPolicy(maximum_attempts=3)
        )
        
        summary = await workflow.execute_activity(
            summarize, text, doc_type,
            start_to_close_timeout=timedelta(minutes=5),
        )
        
        destination = await workflow.execute_activity(
            route_document, summary, doc_type,
            start_to_close_timeout=timedelta(minutes=1),
        )
        
        return {"type": doc_type, "summary": summary, "routed_to": destination}

If the process crashes after classify_document completes but before summarize finishes, Temporal replays the workflow: extract_text and classify_document return their recorded results instantly, and summarize re-executes from scratch.

Inngest

Inngest takes a different approach: instead of replay-based determinism, it uses a step-function model where each step.run() call is a separate execution unit. The framework manages the execution graph, and each step’s result is persisted server-side.

Inngest has invested specifically in AI workflow primitives. Its step.ai namespace provides first-class support for LLM calls with automatic token tracking, and its streaming step support allows workflows to stream intermediate LLM output to clients while still checkpointing the final result.

// Inngest function for iterative research with quality gate
import { inngest } from "./client";

export const researchWorkflow = inngest.createFunction(
  { id: "research-pipeline", retries: 2 },
  { event: "research/started" },
  async ({ event, step }) => {
    const query = event.data.query;
    
    // Step 1: Generate search queries
    const searchQueries = await step.run("generate-queries", async () => {
      const response = await openai.chat.completions.create({
        model: "gpt-5.6-terra",
        messages: [{ role: "user", content: `Generate 5 search queries for: ${query}` }],
      });
      return JSON.parse(response.choices[0].message.content);
    });
    
    // Step 2: Execute searches in parallel
    const searchResults = await Promise.all(
      searchQueries.map((q: string, i: number) =>
        step.run(`search-${i}`, async () => {
          return await searchAPI.search(q);
        })
      )
    );
    
    // Step 3: Synthesize with quality scoring
    let synthesis = await step.run("synthesize", async () => {
      const response = await anthropic.messages.create({
        model: "claude-sonnet-5",
        messages: [{ role: "user", content: `Synthesize: ${JSON.stringify(searchResults)}` }],
      });
      return response.content[0].text;
    });
    
    // Step 4: Quality check — may loop
    const quality = await step.run("quality-check", async () => {
      const response = await openai.chat.completions.create({
        model: "gpt-5.6-terra",
        messages: [{ role: "user", content: `Rate quality 1-10: ${synthesis}` }],
      });
      return parseInt(response.choices[0].message.content);
    });
    
    if (quality < 7) {
      // Wait for human review instead of auto-iterating indefinitely
      const approval = await step.waitForEvent("human-review", {
        event: "research/reviewed",
        timeout: "24h",
      });
      if (approval?.data.action === "retry") {
        // Re-trigger with refined query
        await step.sendEvent("retry-research", {
          name: "research/started",
          data: { query: approval.data.refinedQuery },
        });
      }
    }
    
    return { synthesis, quality };
  }
);

Trigger.dev

Trigger.dev focuses on TypeScript-first developer experience with strong integration into the Vercel/Next.js ecosystem. Its v3 architecture runs tasks in long-lived cloud workers with no timeout limits, making it suitable for extended AI operations. It provides real-time log streaming during task execution, which is valuable for debugging multi-step AI pipelines.

Restate

Restate introduces “virtual objects” — durable, addressable entities with methods that execute with exactly-once semantics. This maps well to stateful AI agents: each agent is a virtual object that maintains conversation history, tool state, and execution progress across arbitrarily many interactions.

The durable execution landscape: each framework trades off operational complexity against AI-specific ergonomics.

Workflow Patterns for AI Pipelines {#workflow-patterns-for-ai-pipelines}

Pattern 1: Sequential Chain with Typed Checkpoints

The simplest pattern. Each LLM call is a checkpointed step, and the output type flows into the next step. If any step fails, only that step re-executes.

Sequential chain: each step checkpointed, failure at any point only re-executes from that step.

Pattern 2: Fan-Out / Fan-In with Model Routing

Send the same input to multiple models in parallel, then aggregate results. This is common for consensus-based workflows (e.g., three models classify a document independently, and the majority vote wins) or for best-of-N generation.

// Fan-out to multiple models, fan-in with voting
const classifications = await Promise.all([
  step.run("classify-claude", () => 
    classify(text, "claude-sonnet-5")),
  step.run("classify-gpt", () => 
    classify(text, "gpt-5.6-terra")),
  step.run("classify-gemini", () => 
    classify(text, "gemini-3.8-flash")),
]);

const finalClass = await step.run("vote", () => 
  majorityVote(classifications)
);

Each step.run is independently checkpointed. If the GPT call fails and retries, the Claude and Gemini results are already persisted and won’t re-execute.

Pattern 3: Iterative Refinement with Budget Caps

An LLM generates output, a quality evaluator scores it, and if the score is below threshold, the workflow loops with feedback. A budget cap prevents infinite iteration.

@workflow.defn
class RefinementWorkflow:
    @workflow.run
    async def run(self, task: str) -> dict:
        budget_remaining = 500_000  # token budget
        attempt = 0
        max_attempts = 5
        
        while attempt < max_attempts and budget_remaining > 0:
            result = await workflow.execute_activity(
                generate, task, attempt,
                start_to_close_timeout=timedelta(minutes=10),
            )
            budget_remaining -= result["tokens_used"]
            
            score = await workflow.execute_activity(
                evaluate, result["output"],
                start_to_close_timeout=timedelta(minutes=2),
            )
            budget_remaining -= score["tokens_used"]
            
            if score["value"] >= 8:
                return {"output": result["output"], "score": score["value"], 
                        "attempts": attempt + 1}
            
            task = f"{task}\n\nPrevious attempt scored {score['value']}/10. "
                   f"Feedback: {score['feedback']}"
            attempt += 1
        
        return {"output": result["output"], "score": score["value"],
                "attempts": attempt, "budget_exhausted": budget_remaining <= 0}

Pattern 4: Supervisor with Delegated Sub-Workflows

A supervisor workflow analyzes the task, breaks it into sub-tasks, spawns child workflows for each, monitors their progress, and synthesizes results. This maps to the supervisor agent orchestration pattern but with durable execution guarantees.

Supervisor pattern: the parent workflow spawns child workflows and aggregates their results, all with durable checkpointing.

Each child workflow runs independently with its own retry policies and checkpoints. If Worker B fails after 20 minutes, Workers A and C continue unaffected, and Worker B retries from its last checkpoint.

Pattern 5: Event-Driven Pipeline with External Triggers

Workflows that react to external events — a webhook from a CI system, a Slack message, a database change — and coordinate AI processing in response. The workflow starts, processes some steps, then pauses waiting for an external signal.

// Trigger.dev: code review workflow that waits for CI
export const codeReviewTask = task({
  id: "ai-code-review",
  run: async (payload: { prUrl: string }) => {
    const diff = await fetchPRDiff(payload.prUrl);
    
    const review = await generateReview(diff);
    await postReviewComment(payload.prUrl, review);
    
    // Wait for CI to complete (up to 2 hours)
    const ciResult = await waitForEvent({
      event: `ci/completed/${payload.prUrl}`,
      timeout: "2h",
    });
    
    if (ciResult.status === "failed") {
      const diagnosis = await diagnoseCIFailure(ciResult.logs, diff);
      await postReviewComment(payload.prUrl, diagnosis);
    }
    
    return { review, ciResult };
  },
});

Checkpointing LLM State {#checkpointing-llm-state}

Checkpointing LLM calls requires deciding what to persist and at what granularity. There are tradeoffs at each level.

What to Checkpoint

Level	Persisted Data	Recovery Capability	Storage Cost
Response only	Final LLM output text	Can skip the call on replay	Low
Full response	Output + token counts + model + finish reason	Can skip call + audit trail	Low-medium
Request + response	Full prompt + full response	Can debug, compare, replay to different model	Medium-high
Streaming chunks	Each SSE chunk as it arrives	Can resume mid-stream	High

For most production AI workflows, persisting the full response object (including token usage) is the right default. It enables cost tracking, debugging, and audit trails without the storage overhead of persisting every request payload.

Persisting request payloads makes sense when: (a) prompts are dynamically generated and hard to reproduce, (b) regulatory requirements demand full audit trails, or (c) the workflow needs the ability to retry a failed step against a different model using the same prompt.

Streaming with Checkpoints

Long LLM responses — especially from models with 128K output limits like Fable 5.1 or GPT-6 Astra — can take minutes to complete. Streaming the response to the user while also checkpointing the final result requires careful coordination.

// Inngest: stream to client while checkpointing final result
const analysis = await step.run("long-analysis", async () => {
  const stream = await anthropic.messages.create({
    model: "claude-fable-5.1",
    max_tokens: 32000,
    stream: true,
    messages: [{ role: "user", content: analysisPrompt }],
  });
  
  let fullResponse = "";
  for await (const event of stream) {
    if (event.type === "content_block_delta") {
      fullResponse += event.delta.text;
      // Emit to client via SSE or WebSocket
      await emitProgress({ partial: event.delta.text });
    }
  }
  
  // The return value is what gets checkpointed
  return fullResponse;
});

The workflow framework checkpoints the final fullResponse string. If the process restarts after this step completed, replay returns the checkpointed string without re-calling the LLM. The streaming was a side effect that doesn’t need to be replayed.

Conversation History as Workflow State

For multi-turn agent workflows, the conversation history grows with each step. Rather than storing the full history in each checkpoint, store it once and reference it by ID.

Conversation history stored in an append-only log, referenced by ID in workflow checkpoints, avoiding duplication.

Human-in-the-Loop Patterns {#human-in-the-loop-patterns}

Production AI workflows frequently need human judgment at specific points. Durable execution frameworks handle this naturally because a workflow can pause for an arbitrarily long time and resume when a signal arrives.

Approval Gates

The most common pattern. The workflow reaches a decision point, notifies a human, and waits for approval.

# Temporal: human approval for generated content
@workflow.defn
class ContentGenerationWorkflow:
    def __init__(self):
        self._approval_result = None
    
    @workflow.signal
    async def approve(self, decision: dict):
        self._approval_result = decision
    
    @workflow.run
    async def run(self, brief: str) -> dict:
        draft = await workflow.execute_activity(
            generate_content, brief,
            start_to_close_timeout=timedelta(minutes=10),
        )
        
        # Notify human reviewer
        await workflow.execute_activity(
            send_review_notification, draft,
            start_to_close_timeout=timedelta(seconds=30),
        )
        
        # Wait for approval signal — can be hours or days
        await workflow.wait_condition(
            lambda: self._approval_result is not None,
            timeout=timedelta(days=7),
        )
        
        if self._approval_result is None:
            return {"status": "timed_out"}
        
        if self._approval_result["approved"]:
            await workflow.execute_activity(
                publish_content, draft,
                start_to_close_timeout=timedelta(minutes=1),
            )
            return {"status": "published", "content": draft}
        else:
            # Re-generate with feedback
            revised = await workflow.execute_activity(
                generate_content, 
                f"{brief}\n\nFeedback: {self._approval_result['feedback']}",
                start_to_close_timeout=timedelta(minutes=10),
            )
            return {"status": "revised", "content": revised}

The workflow is consuming zero compute resources while waiting for the human signal. The state lives in Temporal’s persistence layer (Cassandra or PostgreSQL), and the workflow resumes on the next available worker when the signal arrives.

Escalation Chains

When a human doesn’t respond within a timeout, escalate to the next person or fall back to an automated decision.

Escalation chain: the workflow automatically escalates unanswered approvals, with a final automated fallback for low-risk decisions.

Collaborative Editing

The AI generates a draft, a human edits it, the AI refines based on the edits, and the cycle repeats. This pattern is common in Claude Cowork-style workflows where the AI is a persistent collaborator.

const collaborativeEdit = inngest.createFunction(
  { id: "collaborative-doc" },
  { event: "doc/started" },
  async ({ event, step }) => {
    let document = await step.run("initial-draft", () =>
      generateDraft(event.data.brief)
    );
    
    for (let round = 0; round < 5; round++) {
      // Send draft to human, wait for edits
      await step.run(`notify-${round}`, () =>
        sendForReview(document, event.data.reviewerEmail)
      );
      
      const edits = await step.waitForEvent(`edits-${round}`, {
        event: "doc/edited",
        match: "data.docId",
        timeout: "48h",
      });
      
      if (!edits || edits.data.action === "approve") break;
      
      // AI refines based on human edits
      document = await step.run(`refine-${round}`, () =>
        refineDocument(document, edits.data.changes, edits.data.comments)
      );
    }
    
    return { finalDocument: document };
  }
);

Error Handling and Retry Strategies for LLM Calls {#error-handling-and-retry-strategies}

LLM API calls fail in distinct categories, and each category needs a different retry strategy.

Error Type	HTTP Status	Retry?	Strategy
Rate limited	429	Yes	Exponential backoff, respect Retry-After header
Server error	500, 502, 503	Yes	Exponential backoff, max 3-5 attempts
Context length exceeded	400	No (or with modification)	Truncate input, switch to longer-context model
Content filtered	400 (safety)	Maybe	Rephrase prompt, or escalate to human
Authentication	401, 403	No	Fix credentials, alert ops
Timeout	—	Yes	Increase timeout, switch to faster model
Malformed output	200 (but unparseable)	Yes	Retry with stricter prompt, add JSON mode

Model Fallback Chains

When a primary model fails consistently, fall back to an alternative. This is especially important for availability — no single provider has 100% uptime.

@activity.defn
async def llm_call_with_fallback(prompt: str, task_type: str) -> dict:
    models = [
        {"provider": "anthropic", "model": "claude-sonnet-5", "timeout": 60},
        {"provider": "openai", "model": "gpt-5.6-terra", "timeout": 60},
        {"provider": "google", "model": "gemini-3.8-flash", "timeout": 45},
    ]
    
    last_error = None
    for model_config in models:
        try:
            result = await call_model(prompt, model_config)
            return {
                "output": result.text,
                "model_used": model_config["model"],
                "tokens": result.usage,
            }
        except RateLimitError:
            last_error = "rate_limited"
            continue
        except ServerError:
            last_error = "server_error"
            continue
        except ContentFilterError as e:
            # Don't fallback for safety blocks — they're likely 
            # to trigger on other models too
            raise ApplicationError(f"Content filtered: {e}", non_retryable=True)
    
    raise ApplicationError(f"All models failed. Last error: {last_error}")

Model fallback chain: each model is tried in sequence, with non-retryable errors (content filtering) short-circuiting the chain.

Parsing Retries

LLM structured output — even with JSON mode enabled — sometimes fails to parse. A common pattern is to retry with a more explicit prompt before giving up.

const result = await step.run("extract-entities", async () => {
  for (let attempt = 0; attempt < 3; attempt++) {
    const response = await openai.chat.completions.create({
      model: "gpt-5.6-terra",
      response_format: { type: "json_object" },
      messages: [{ 
        role: "user", 
        content: attempt === 0 
          ? basePrompt 
          : `${basePrompt}\n\nYou MUST respond with valid JSON matching this schema: ${schema}. Previous attempt failed to parse.`
      }],
    });
    
    try {
      return JSON.parse(response.choices[0].message.content);
    } catch (e) {
      if (attempt === 2) throw new Error("Failed to parse after 3 attempts");
    }
  }
});

Cost Control in Long-Running Workflows {#cost-control-in-long-running-workflows}

Long-running AI workflows can accumulate substantial costs, especially with iterative patterns that loop unpredictably. Durable execution frameworks provide the infrastructure to implement cost controls as first-class workflow logic.

Token Budgets

Assign each workflow run a token budget. Track cumulative usage across steps and halt execution when the budget is exhausted.

@workflow.defn
class BudgetedWorkflow:
    def __init__(self):
        self._total_input_tokens = 0
        self._total_output_tokens = 0
        self._budget_usd = 0.0
        self._max_budget_usd = 5.0  # $5 per run
    
    def _track_cost(self, usage: dict, model: str):
        # Pricing per 1M tokens (input/output)
        pricing = {
            "claude-sonnet-5": (2.0, 10.0),
            "gpt-5.6-terra": (2.0, 12.0),
            "gemini-3.8-flash": (0.75, 3.75),
        }
        input_rate, output_rate = pricing.get(model, (5.0, 15.0))
        cost = (usage["input"] * input_rate + usage["output"] * output_rate) / 1_000_000
        self._budget_usd += cost
        self._total_input_tokens += usage["input"]
        self._total_output_tokens += usage["output"]
    
    def _check_budget(self):
        if self._budget_usd >= self._max_budget_usd:
            raise ApplicationError(
                f"Budget exhausted: ${self._budget_usd:.2f} / ${self._max_budget_usd:.2f}",
                non_retryable=True,
            )

Model Tiering Within Workflows

Use expensive models only for steps that need them. Route simple classification to Gemini 3.8 Flash at $0.75/M input (introductory pricing through December 31, 2026; $1.50/M thereafter), use Claude Sonnet 5 for nuanced analysis, and reserve Fable 5.1 or GPT-6 Astra for complex reasoning steps.

Prompt Caching

For workflows that make multiple LLM calls with overlapping context (e.g., the same system prompt or document), prompt caching reduces costs. Anthropic charges $0.25/M for cache reads on Fable 5.1 (vs. $10/M for uncached input). OpenAI’s GPT-6 Astra charges cached input at a discount as well.

Structure workflows to maximize cache hits by keeping the static context (system prompt, reference documents) at the front of the message array and varying only the final user message across steps.

Observability for AI Workflows {#observability-for-ai-workflows}

Standard workflow observability (step durations, success/failure rates) is necessary but insufficient for AI pipelines. AI-specific metrics include:

Token usage per step and per run. Track cumulative input/output tokens across all steps, broken down by model. This feeds into cost dashboards and budget alerts.

Output quality scores. If the workflow includes evaluation steps (quality scoring, validation), these scores should be tracked as time-series metrics to detect drift.

Model latency by provider. LLM API latency varies by provider, model, time of day, and input length. Track P50/P95/P99 latency per model per step to inform model routing decisions.

Retry rates by error type. A spike in 429 retries indicates approaching rate limits. A spike in parse failures suggests a model behavior change.

AI workflow observability: workflow engines emit standard traces plus AI-specific metrics (tokens, quality, cost) to specialized dashboards.

Integration with LLM observability tools is straightforward. Langfuse, Helicone, and Braintrust all accept OpenTelemetry spans or custom trace data. Wrap each LLM call in a trace that captures the model, prompt hash, token usage, latency, and the workflow run ID for correlation.

# Wrapping an LLM activity with observability
@activity.defn
async def llm_with_tracing(prompt: str, model: str) -> dict:
    span = tracer.start_span("llm_call", attributes={
        "model": model,
        "prompt_hash": hashlib.sha256(prompt.encode()).hexdigest()[:16],
        "workflow_run_id": activity.info().workflow_run_id,
    })
    
    start = time.monotonic()
    try:
        response = await call_model(prompt, model)
        latency = time.monotonic() - start
        
        span.set_attributes({
            "input_tokens": response.usage.input_tokens,
            "output_tokens": response.usage.output_tokens,
            "latency_ms": int(latency * 1000),
            "finish_reason": response.stop_reason,
        })
        
        return {
            "output": response.content[0].text,
            "usage": {
                "input": response.usage.input_tokens,
                "output": response.usage.output_tokens,
            },
            "model": model,
            "latency_ms": int(latency * 1000),
        }
    except Exception as e:
        span.set_status(StatusCode.ERROR, str(e))
        raise
    finally:
        span.end()

Production Architecture {#production-architecture}

A production AI workflow system has several components beyond the workflow engine itself.

Production architecture: the API layer triggers workflows, the engine dispatches to workers, workers call LLMs through a gateway, and the engine persists state.

Worker Sizing

AI workflow workers are I/O-bound (waiting on LLM API responses), not CPU-bound. A single worker can handle many concurrent workflow executions. Size worker pools based on concurrent LLM calls, not CPU cores.

For Temporal, this means tuning max_concurrent_activities and max_concurrent_workflow_tasks. A worker with max_concurrent_activities=50 can have 50 LLM API calls in flight simultaneously, which is usually plenty.

Deployment Considerations

Worker versioning. When deploying new workflow code, existing in-flight workflows are running the old version. Temporal handles this with task queue versioning — old workflows complete on old workers, new workflows start on new workers. Inngest and Trigger.dev handle this at the platform level.

Idempotency. Every step that has side effects (sending an email, posting a Slack message, writing to a database) must be idempotent, because replays and retries can re-execute steps. Use idempotency keys derived from the workflow run ID and step name.

Secrets rotation. Long-running workflows may outlive API key rotations. Store secrets in a vault (AWS Secrets Manager, HashiCorp Vault) and fetch them at activity execution time, not at workflow start time. This way, a workflow that runs for days picks up rotated credentials automatically.

When to Use Which Framework

Temporal is the right choice for teams that need maximum control over execution semantics, have polyglot codebases (Go, Java, Python, TypeScript), and can absorb the operational overhead of running a Temporal cluster (or paying for Temporal Cloud). It’s the most battle-tested option for mission-critical workflows.

Inngest is the right choice for TypeScript/Python teams that want managed infrastructure, AI-specific primitives, and a lower operational bar. Its step-function model is easier to reason about than Temporal’s replay semantics, at the cost of less flexibility.

Trigger.dev is the right choice for TypeScript-only teams already in the Vercel ecosystem that want tight integration with their existing deployment pipeline.

Restate is the right choice for teams building stateful agent systems where the “virtual object” abstraction maps naturally to their domain model. Its embedded state (no external database required for state) reduces operational complexity.

Hatchet is worth evaluating for teams that need fine-grained concurrency control and rate limiting at the workflow engine level, particularly for high-throughput AI pipelines that need to respect per-model token-per-minute limits.

Summary {#summary}

Long-running AI workflows need durable execution because LLM calls are expensive to re-execute, AI pipelines run for minutes to days, and human-in-the-loop patterns require indefinite pauses. The core abstraction is straightforward: write sequential workflow code, let the runtime checkpoint each step’s result, and on failure, replay skips already-completed steps.

Key decisions for production deployments:

• Checkpoint granularity. Persist full LLM response objects (including token usage) as the default. Add request payloads when audit or model-switching flexibility justifies the storage cost.
• Retry policies. Differentiate by error type: exponential backoff for rate limits and server errors, no retry for content filtering, parsing retries with stricter prompts for malformed output.
• Model fallback. Chain multiple providers for availability. Route by task complexity to control costs.
• Human gates. Use the framework’s signal/wait primitives to pause workflows indefinitely for human decisions. Implement escalation chains with timeouts.
• Cost controls. Token budgets per workflow run, model tiering within workflows, and prompt caching across steps.
• Observability. Track tokens, cost, latency, and quality scores per step and per run. Correlate with workflow run IDs for end-to-end debugging.

The gap between “AI demo” and “AI in production” is largely an orchestration gap. Durable execution frameworks close it by making the hard problems — state persistence, failure recovery, human coordination, cost control — infrastructure concerns rather than application code concerns.

Further Reading {#further-reading}

• Temporal documentation — Core concepts, SDKs for Go/Java/Python/TypeScript, and production deployment guides
• Inngest documentation — Step function model, AI-specific middleware, and streaming step support
• Trigger.dev v3 documentation — Long-running tasks, realtime logs, and deployment to managed cloud workers
• Restate documentation — Virtual objects, durable execution model, and stateful service patterns
• Hatchet GitHub repository — Open-source workflow engine with concurrency control and rate limiting
• Temporal AI samples — Python samples including LLM orchestration patterns
• Anthropic prompt caching documentation — Cache read pricing and implementation details for reducing costs in multi-step workflows
• Langfuse documentation — Open-source LLM observability with trace correlation for workflow engines
• “Building Reliable AI Agents with Temporal” (Temporal blog) — Patterns for durable AI agent orchestration