How orchestrating long-running AI workflows differs from traditional job queues — durable execution, checkpointing, human-in-the-loop, and practical patterns with Temporal, Inngest, and Trigger.dev.

Orchestrating Long-Running AI Workflows: Durable Execution, Checkpointing, and Human-in-the-Loop for AI Pipelines

A single LLM API call takes seconds. A useful AI workflow — one that researches, reasons, acts, and waits for human approval — takes minutes, hours, or days. The gap between those two timescales is where most production AI systems break. Traditional job queues assume tasks are short-lived and idempotent. AI workflows are neither: they accumulate expensive intermediate state (tool call results, retrieved documents, partial generations), depend on external systems with unpredictable latency (model APIs, human reviewers, third-party tools), and fail in ways that make simple retries dangerous (a retried agentic action might duplicate a real-world side effect).

Durable execution engines — Temporal, Inngest, Trigger.dev, and their equivalents — solve this by persisting workflow state at each step, so a process that crashes at minute 47 resumes from minute 47 rather than restarting from scratch. This post covers how these systems work, how to apply them to AI-specific patterns, and where the sharp edges are.

Why Traditional Job Queues Fail for AI Workloads
Durable Execution: The Core Abstraction
The Orchestration Engines Compared
Checkpointing LLM State
Human-in-the-Loop Patterns
Error Handling and Retry Strategies for LLM Calls
Multi-Step Agent Workflows
Cost Control in Long-Running Workflows
Observability and Debugging
Patterns That Work in Production
Summary
Further Reading

Why Traditional Job Queues Fail for AI Workloads

A Celery worker, a Bull queue, or an AWS Lambda triggered by SQS all share the same model: receive message, execute function, return result. This works when the function runs in under 15 minutes, has no intermediate state worth preserving, and can be safely retried from the beginning on failure.

AI workflows violate all three assumptions:

Property	Traditional Job	AI Workflow
Duration	Seconds to minutes	Minutes to days
Intermediate state	Disposable	Expensive (retrieved docs, partial reasoning, tool outputs)
Retry safety	Usually idempotent	Side effects (emails sent, code deployed, APIs called)
Failure modes	Crash or timeout	Rate limits, content filters, model degradation, human delay
Cost of restart	Negligible	$0.50–$50+ in wasted tokens

Consider a workflow that: (1) retrieves 200 documents from a vector database, (2) calls Claude Sonnet 5 to synthesize findings across three passes, (3) generates a draft report, (4) sends it to a human reviewer, (5) incorporates feedback, and (6) publishes the final version. If the process crashes during step 5 and restarts from step 1, it re-embeds queries, re-retrieves documents, and re-runs synthesis — burning tokens and wall-clock time on work already completed.

Traditional job queues restart from scratch — every prior LLM call is wasted.

Durable Execution: The Core Abstraction

Durable execution engines persist the result of each step in a workflow to a database. When the workflow resumes after a crash, it replays the recorded results for completed steps (without re-executing them) and picks up at the first unfinished step.

The key insight: the workflow function itself looks like normal sequential code, but the runtime intercepts each “activity” (external call, timer, signal) and records its outcome. On replay, recorded outcomes are returned immediately instead of re-executing the activity.

Durable execution replays completed steps from stored results and resumes at the point of failure.

This model has direct consequences for AI workflows:

LLM call results are persisted. A $2 synthesis call that completed before a crash is not re-executed. The saved output is replayed.
Human wait times are free. A workflow can sleep for days waiting on human approval. The engine persists the timer and resumes the workflow when the signal arrives.
Partial progress survives deployment. Deploying a new version of the orchestration code doesn’t kill running workflows — they continue from their last checkpoint.

Determinism Constraints

Durable execution engines require workflow functions to be deterministic: given the same inputs and replayed activity results, they must follow the same execution path. This means no Math.random(), no Date.now(), no direct HTTP calls inside workflow code. All non-deterministic operations must be wrapped as activities (which the engine records and replays).

For AI workflows, this constraint means:

LLM calls must be activities, not inline code in the workflow function
Any decision based on LLM output must use the persisted output, not re-query the model
Random sampling (e.g., choosing a subset of documents) must use a seeded RNG or be wrapped as an activity

The Orchestration Engines Compared

Three engines dominate the space for AI workflow orchestration: Temporal, Inngest, and Trigger.dev. Each makes different tradeoffs.

Feature	Temporal	Inngest	Trigger.dev
Architecture	Self-hosted cluster + workers	Managed cloud (self-host option)	Managed cloud (self-host option)
Language SDKs	Go, Java, Python, TypeScript, .NET	TypeScript, Python, Go	TypeScript
Persistence	Cassandra, PostgreSQL, MySQL	Built-in (cloud) or PostgreSQL	Built-in (cloud) or PostgreSQL
Max workflow duration	Unlimited (continue-as-new for history)	Unlimited	Unlimited
Human-in-the-loop	Signals + queries	Wait-for-event	Wait-for-event
Pricing model	Self-host: infra cost. Cloud: per-action	Per-step execution	Per-step execution
AI-specific features	None built-in (you build them)	AI middleware, streaming support	AI SDK integration, realtime
Operational complexity	High (run your own cluster)	Low (managed)	Low (managed)
Maturity	Production since 2020, battle-tested at scale	Production since 2023	Production since 2023

The three main durable execution engines occupy different points on the control-vs-simplicity spectrum.

Temporal

Temporal is the most mature option. Originally developed at Uber as Cadence, it handles millions of concurrent workflows in production at companies like Netflix, Snap, and Coinbase. The Python SDK (temporalio) is the most relevant for AI workloads.

A Temporal workflow for an AI pipeline:

# workflow.py
from temporalio import workflow
from datetime import timedelta

@workflow.defn
class ResearchWorkflow:
    def __init__(self):
        self.human_feedback = None

    @workflow.run
    async def run(self, query: str) -> dict:
        # Step 1: Retrieve documents (persisted)
        docs = await workflow.execute_activity(
            retrieve_documents,
            query,
            start_to_close_timeout=timedelta(seconds=30),
        )

        # Step 2: Synthesize with LLM (persisted, expensive)
        synthesis = await workflow.execute_activity(
            synthesize_with_llm,
            args=[query, docs],
            start_to_close_timeout=timedelta(minutes=5),
            retry_policy=RetryPolicy(
                maximum_attempts=3,
                backoff_coefficient=2.0,
                non_retryable_error_types=["ContentFilterError"],
            ),
        )

        # Step 3: Wait for human review (can sleep for days)
        await workflow.execute_activity(
            notify_reviewer, synthesis,
            start_to_close_timeout=timedelta(seconds=10),
        )
        await workflow.wait_condition(lambda: self.human_feedback is not None)

        # Step 4: Incorporate feedback
        final = await workflow.execute_activity(
            revise_with_feedback,
            args=[synthesis, self.human_feedback],
            start_to_close_timeout=timedelta(minutes=5),
        )
        return final

    @workflow.signal
    async def submit_feedback(self, feedback: str):
        self.human_feedback = feedback

# activities.py
from temporalio import activity
import anthropic

@activity.defn
async def synthesize_with_llm(query: str, docs: list[str]) -> str:
    client = anthropic.AsyncAnthropic()
    response = await client.messages.create(
        model="claude-sonnet-5",
        max_tokens=4096,
        messages=[{
            "role": "user",
            "content": f"Synthesize findings for: {query}\n\nDocuments:\n" +
                       "\n---\n".join(docs)
        }],
    )
    return response.content[0].text

The critical property: if the worker crashes after synthesize_with_llm completes but before notify_reviewer runs, the workflow resumes by replaying the stored synthesis result, then proceeds to notify the reviewer. The LLM call is not re-executed.

Inngest

Inngest takes an event-driven approach. Functions are triggered by events and broken into steps, each of which is individually retried and persisted. The model is simpler than Temporal’s — no separate worker infrastructure, no cluster to manage.

// inngest/functions/research.ts
import { inngest } from "./client";

export const researchWorkflow = inngest.createFunction(
  { id: "research-workflow" },
  { event: "research/started" },
  async ({ event, step }) => {
    const docs = await step.run("retrieve-documents", async () => {
      return await retrieveDocuments(event.data.query);
    });

    const synthesis = await step.run("synthesize", async () => {
      return await synthesizeWithLLM(event.data.query, docs);
    });

    await step.run("notify-reviewer", async () => {
      await sendReviewNotification(synthesis);
    });

    // Wait for human feedback (up to 7 days)
    const feedback = await step.waitForEvent("wait-for-feedback", {
      event: "research/feedback-submitted",
      match: "data.workflowId",
      timeout: "7d",
    });

    if (!feedback) {
      return { status: "timed_out" };
    }

    const final = await step.run("revise", async () => {
      return await reviseWithFeedback(synthesis, feedback.data.feedback);
    });

    return { status: "completed", result: final };
  }
);

Inngest’s step.ai middleware provides built-in support for wrapping LLM calls with automatic token tracking and cost attribution, though it’s limited to supported providers.

Trigger.dev

Trigger.dev targets TypeScript-first teams building AI applications. Its v3 runtime supports long-running tasks with checkpointing, and it integrates with the Vercel AI SDK for streaming.

// trigger/research.ts
import { task, wait } from "@trigger.dev/sdk/v3";

export const researchTask = task({
  id: "research-workflow",
  run: async (payload: { query: string; workflowId: string }) => {
    const docs = await retrieveDocuments(payload.query);

    const synthesis = await synthesizeWithLLM(payload.query, docs);

    await notifyReviewer(synthesis);

    // Wait for human input via webhook
    const feedback = await wait.forRequest<{ feedback: string }>({
      id: `feedback-${payload.workflowId}`,
      timeout: "7d",
    });

    const final = await reviseWithFeedback(synthesis, feedback.feedback);

    return { result: final };
  },
});

Checkpointing LLM State

The naive approach to checkpointing wraps each LLM call as an activity and stores the complete response. This works, but AI workflows introduce specific checkpointing challenges.

Multi-Turn Conversations

An agent that makes 15 tool calls in sequence accumulates a conversation history that grows with each turn. Each turn’s LLM call depends on the full history of previous turns. Checkpointing each turn individually means storing the growing message array at each step.

Each turn in a multi-turn agent loop checkpoints the full conversation history.

The state grows quickly. A 15-turn conversation with tool calls can accumulate 50,000–100,000 tokens of context. Storing this is cheap (it’s text), but there’s a subtlety: if the workflow code changes between checkpoints — say, the system prompt is updated mid-flight — the replayed conversation history may be inconsistent with the new prompt. Temporal handles this with versioning; Inngest and Trigger.dev handle it by treating each step as an opaque recorded result.

Streaming Responses

Streaming creates a tension with checkpointing. The user wants tokens as they’re generated; the orchestration engine wants a complete result to persist. Two approaches:

Buffer and persist. Stream tokens to the user in real-time, but also buffer the full response. Once streaming completes, persist the buffered result as the step output. This is the simplest approach and works with all three engines.
Checkpoint at stream boundaries. For very long generations (128K output tokens from Claude Fable 5.1, for instance), persist partial results at intervals. This requires custom logic — none of the engines support this natively.

@activity.defn
async def stream_and_persist(query: str, docs: list[str]) -> str:
    client = anthropic.AsyncAnthropic()
    buffer = []

    async with client.messages.stream(
        model="claude-sonnet-5",
        max_tokens=8192,
        messages=[{"role": "user", "content": f"Synthesize: {query}\n\n{docs}"}],
    ) as stream:
        async for text in stream.text_stream:
            buffer.append(text)
            # Optionally: push to a side channel for real-time display
            await push_to_websocket(text)

    full_response = "".join(buffer)
    return full_response  # This gets persisted by the engine

KV Cache Considerations

LLM providers maintain KV caches for recently-seen prefixes. If a workflow resumes after a crash and re-sends the same prefix (system prompt + accumulated context), the provider may still have the prefix cached, reducing cost. But cache lifetimes are short — typically 5–10 minutes for most providers. A workflow that pauses for human review will not benefit from KV cache on resume. This is unavoidable; the cost savings from not re-running prior steps still dominate.

Human-in-the-Loop Patterns

Human-in-the-loop (HITL) is where durable execution engines provide the most value over traditional approaches. A workflow that needs human approval can sleep indefinitely without consuming compute resources.

Pattern 1: Approval Gate

The simplest HITL pattern: the workflow pauses, a human approves or rejects, and the workflow continues.

An approval gate pauses the workflow until a human signals a decision.

Implementation in Temporal:

@workflow.defn
class ApprovalWorkflow:
    def __init__(self):
        self.decision = None

    @workflow.run
    async def run(self, content: str) -> dict:
        await workflow.execute_activity(
            send_approval_request, content,
            start_to_close_timeout=timedelta(seconds=10),
        )

        # Sleep until signal arrives — no compute consumed
        await workflow.wait_condition(
            lambda: self.decision is not None,
            timeout=timedelta(days=7),
        )

        if self.decision is None:
            return {"status": "timed_out"}
        if self.decision == "rejected":
            return {"status": "rejected"}

        await workflow.execute_activity(
            publish_content, content,
            start_to_close_timeout=timedelta(seconds=30),
        )
        return {"status": "published"}

    @workflow.signal
    async def approve(self, decision: str):
        self.decision = decision

The Temporal worker process can be scaled down to zero while waiting. The workflow state lives in the Temporal server’s database. When the signal arrives (via an API call from a web UI, Slack bot, or email link), the server schedules the workflow for execution on a worker.

A human reviews output, provides feedback, and the workflow revises — potentially multiple times.

Iterative refinement loops between generation and human review until approval.

@workflow.run
async def run(self, query: str) -> dict:
    draft = await workflow.execute_activity(
        generate_draft, query,
        start_to_close_timeout=timedelta(minutes=5),
    )

    max_revisions = 5
    for i in range(max_revisions):
        await workflow.execute_activity(
            send_for_review, args=[draft, i],
            start_to_close_timeout=timedelta(seconds=10),
        )

        self.feedback = None
        await workflow.wait_condition(lambda: self.feedback is not None)

        if self.feedback.get("approved"):
            return {"status": "approved", "draft": draft, "revisions": i}

        draft = await workflow.execute_activity(
            revise_draft,
            args=[draft, self.feedback["comments"]],
            start_to_close_timeout=timedelta(minutes=5),
        )

    return {"status": "max_revisions_reached", "draft": draft}

Pattern 3: Escalation

The workflow tries automated processing first, escalates to a human only if confidence is low or the output fails validation.

Escalation routes low-confidence outputs to humans, keeping the happy path fully automated.

This pattern is particularly effective with Claude Code Auto Mode, where the safety classifier catches 89% of dangerous commands. Workflow orchestration provides the structure: if the classifier flags a command, the workflow pauses and routes to a human approver rather than blindly retrying.

Error Handling and Retry Strategies for LLM Calls

LLM API calls fail in diverse ways, and each failure mode demands a different retry strategy.

Failure Mode	HTTP Status	Retry?	Strategy
Rate limit	429	Yes	Exponential backoff with jitter, respect `Retry-After`
Server overload	529 (Anthropic), 503	Yes	Backoff, consider fallback model
Content filter	400 with `content_filter`	No	Route to human, or rephrase prompt
Context too long	400 with `context_length_exceeded`	No	Truncate input, switch to longer-context model
Invalid request	400	No	Fix request, don’t retry
Timeout	N/A	Yes	Increase timeout, consider model with faster inference
Model degradation	200 (but bad output)	Maybe	Re-run with temperature=0, or escalate

The last category — model degradation — is the hardest. The API returns 200, the response is syntactically valid, but the content is wrong or incoherent. Detecting this requires output validation, which can itself involve an LLM call (adding cost and latency).

from temporalio.common import RetryPolicy

# Different retry policies for different failure modes
llm_retry_policy = RetryPolicy(
    initial_interval=timedelta(seconds=1),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(minutes=2),
    maximum_attempts=5,
    non_retryable_error_types=[
        "ContentFilterError",
        "ContextLengthError",
        "InvalidRequestError",
    ],
)

Fallback Chains

When the primary model is unavailable or rate-limited, a fallback chain tries progressively cheaper/faster models:

A fallback chain tries progressively cheaper models when the primary is unavailable.

@activity.defn
async def llm_with_fallback(prompt: str) -> dict:
    models = [
        ("anthropic", "claude-opus-5", {"max_tokens": 4096}),
        ("anthropic", "claude-sonnet-5", {"max_tokens": 4096}),
        ("google", "gemini-3.8-flash", {"max_output_tokens": 4096}),
    ]

    last_error = None
    for provider, model, params in models:
        try:
            result = await call_llm(provider, model, prompt, **params)
            return {"model_used": model, "content": result}
        except (RateLimitError, ServerOverloadError) as e:
            last_error = e
            continue

    raise AllModelsUnavailableError(f"All fallbacks exhausted: {last_error}")

Important: the workflow engine persists the result of this activity, including which model was actually used. Downstream steps can inspect result["model_used"] to adjust behavior (e.g., apply stricter validation if a weaker fallback model was used).

Idempotency Keys

For activities with side effects (sending emails, creating database records, calling external APIs), use idempotency keys derived from the workflow ID and step name. This ensures that retries don’t produce duplicate side effects.

@activity.defn
async def send_email(workflow_id: str, step: str, recipient: str, body: str):
    idempotency_key = f"{workflow_id}:{step}"
    await email_service.send(
        to=recipient,
        body=body,
        idempotency_key=idempotency_key,  # Provider deduplicates
    )

Multi-Step Agent Workflows

The most complex AI workflows involve agents that plan, execute tools, observe results, and iterate. These map naturally onto durable execution, but require careful structuring.

Agent Loop as a Workflow

An agent loop — plan → act → observe → repeat — translates to a workflow where each iteration is a set of persisted activities:

An agent loop maps each plan-act-observe cycle to persisted workflow steps.

@workflow.defn
class AgentWorkflow:
    @workflow.run
    async def run(self, task: str) -> dict:
        messages = [{"role": "user", "content": task}]
        max_iterations = 20
        total_tokens = 0

        for i in range(max_iterations):
            # Plan: LLM decides next action
            response = await workflow.execute_activity(
                agent_step,
                args=[messages],
                start_to_close_timeout=timedelta(minutes=3),
                retry_policy=llm_retry_policy,
            )

            total_tokens += response["usage"]["total_tokens"]
            messages = response["updated_messages"]

            # Check if agent is done
            if response["stop_reason"] == "end_turn":
                return {
                    "result": response["content"],
                    "iterations": i + 1,
                    "total_tokens": total_tokens,
                }

            # Execute tool calls
            for tool_call in response.get("tool_calls", []):
                tool_result = await workflow.execute_activity(
                    execute_tool,
                    tool_call,
                    start_to_close_timeout=timedelta(minutes=10),
                    retry_policy=tool_retry_policy,
                )
                messages.append({
                    "role": "tool",
                    "tool_use_id": tool_call["id"],
                    "content": tool_result,
                })

            # Cost guard
            if total_tokens > 500_000:
                return {"status": "token_budget_exceeded", "iterations": i + 1}

        return {"status": "max_iterations_reached"}

Continue-As-New for Long Agent Runs

Temporal (and similar engines) stores workflow history in a database. A workflow with hundreds of steps accumulates a large history that slows down replay. Temporal’s solution is continue_as_new: the workflow completes with its current state and immediately starts a new execution, resetting the history.

For agent workflows, this means periodically checkpointing the full message array and continuing with a fresh history:

if i > 0 and i % 50 == 0:
    # History is getting large — continue as new
    workflow.continue_as_new(
        args=[{"messages": messages, "iteration_offset": i, "total_tokens": total_tokens}]
    )

Inngest and Trigger.dev don’t have this concern — they persist each step individually rather than maintaining a replay log, so history growth isn’t a problem in the same way.

Parallel Tool Execution

When an agent requests multiple independent tool calls, executing them in parallel reduces latency:

# Parallel tool execution in Temporal
tool_handles = []
for tool_call in response["tool_calls"]:
    handle = workflow.start_activity(
        execute_tool,
        tool_call,
        start_to_close_timeout=timedelta(minutes=10),
    )
    tool_handles.append((tool_call["id"], handle))

for tool_id, handle in tool_handles:
    result = await handle
    messages.append({
        "role": "tool",
        "tool_use_id": tool_id,
        "content": result,
    })

Cost Control in Long-Running Workflows

Long-running AI workflows can silently accumulate large costs. A runaway agent loop hitting Claude Fable 5.1 at $10/$50 per M tokens can burn through hundreds of dollars before anyone notices.

Token Budgets

Every AI workflow should have a token budget — a hard cap on total tokens consumed across all LLM calls in the workflow.

class TokenBudget:
    def __init__(self, max_input: int, max_output: int):
        self.max_input = max_input
        self.max_output = max_output
        self.used_input = 0
        self.used_output = 0

    def record(self, input_tokens: int, output_tokens: int):
        self.used_input += input_tokens
        self.used_output += output_tokens

    def remaining_fraction(self) -> float:
        input_frac = 1 - (self.used_input / self.max_input)
        output_frac = 1 - (self.used_output / self.max_output)
        return min(input_frac, output_frac)

    def exceeded(self) -> bool:
        return self.used_input > self.max_input or self.used_output > self.max_output

The budget should be a first-class part of workflow state, persisted alongside other checkpointed data. When the budget is 80% consumed, the workflow can switch to a cheaper model. When exceeded, it stops.

Model Routing by Step

Not every step in a workflow needs the most capable model. A pattern that works well: use a frontier model for the critical reasoning step, and a cheaper model for summarization, formatting, and validation.

Step	Suggested Model Tier	Rationale
Document retrieval + reranking	Embedding model only	No generation needed
Initial analysis / planning	Frontier (Opus 5, GPT-6 Astra)	Accuracy matters most
Tool call generation	Mid-tier (Sonnet 5, Gemini 3.8 Flash)	Structured output, lower complexity
Output formatting	Budget (Haiku 4.5, GPT-6 Luna)	Template-filling, low reasoning
Validation / fact-checking	Mid-tier + structured output	Balance accuracy and cost

Prompt Caching

For workflows that make multiple LLM calls with overlapping prefixes (same system prompt, same retrieved documents), prompt caching reduces input token costs. Claude’s prompt caching charges $0.30/M for cache writes and $0.20/M for cache reads (on Sonnet 5), compared to $2/M for non-cached input. A workflow that makes 10 calls with the same 10,000-token system prompt saves ~$0.18 — modest per-workflow, but meaningful at scale.

The workflow engine doesn’t manage prompt caching directly. It’s handled at the LLM call level:

@activity.defn
async def llm_with_caching(system: str, messages: list) -> dict:
    client = anthropic.AsyncAnthropic()
    response = await client.messages.create(
        model="claude-sonnet-5",
        max_tokens=4096,
        system=[{
            "type": "text",
            "text": system,
            "cache_control": {"type": "ephemeral"},
        }],
        messages=messages,
    )
    return {
        "content": response.content[0].text,
        "usage": {
            "input_tokens": response.usage.input_tokens,
            "cache_read_tokens": response.usage.cache_read_input_tokens,
            "output_tokens": response.usage.output_tokens,
        },
    }

Observability and Debugging

Long-running workflows fail in opaque ways. An agent loop that produces wrong output after 12 iterations is harder to debug than a single API call that returns an error.

What to Track

Metric	Why It Matters
Token usage per step	Cost attribution, budget monitoring
Model used per step	Detect unexpected fallbacks
Latency per step	Identify bottlenecks (LLM latency vs. tool latency)
Workflow duration	End-to-end SLA tracking
Step retry count	Detect flaky providers
Human wait time	Identify process bottlenecks
Output quality scores	Detect model degradation (requires eval)

Temporal has built-in visibility APIs — query workflow state, list workflows by status, search by custom attributes. Inngest and Trigger.dev provide dashboards with step-level execution traces.

For LLM-specific observability, tools like Langfuse, Helicone, and Braintrust integrate at the LLM call level rather than the workflow level. The bridge between them: tag each LLM call with the workflow ID and step name, so traces can be correlated.

@activity.defn
async def tracked_llm_call(
    workflow_id: str, step_name: str, model: str, messages: list
) -> dict:
    # Langfuse trace
    trace = langfuse.trace(
        name=step_name,
        metadata={"workflow_id": workflow_id},
    )

    generation = trace.generation(
        name=f"{step_name}_generation",
        model=model,
        input=messages,
    )

    response = await call_llm(model, messages)

    generation.end(
        output=response["content"],
        usage={
            "input": response["usage"]["input_tokens"],
            "output": response["usage"]["output_tokens"],
        },
    )

    return response

LLM observability tools correlate with workflow engines via shared workflow IDs and step names.

Patterns That Work in Production

After working through the mechanics, some patterns consistently prove effective for production AI workflows.

1. Fan-Out/Fan-In for Document Processing

Process a batch of documents in parallel, then synthesize results. Each document’s processing is an independent activity, so failures are isolated.

@workflow.run
async def process_documents(self, doc_ids: list[str]) -> dict:
    # Fan out: process each document in parallel
    handles = [
        workflow.start_activity(
            process_single_document, doc_id,
            start_to_close_timeout=timedelta(minutes=5),
        )
        for doc_id in doc_ids
    ]

    results = []
    failures = []
    for doc_id, handle in zip(doc_ids, handles):
        try:
            result = await handle
            results.append(result)
        except Exception as e:
            failures.append({"doc_id": doc_id, "error": str(e)})

    # Fan in: synthesize all successful results
    synthesis = await workflow.execute_activity(
        synthesize_results, results,
        start_to_close_timeout=timedelta(minutes=5),
    )

    return {"synthesis": synthesis, "failures": failures}

2. Scheduled Recurring Workflows

A workflow that runs daily — monitoring a data source, generating reports, checking for drift — is a natural fit for durable execution. Unlike cron jobs, the workflow maintains state across runs.

@workflow.defn
class DailyMonitorWorkflow:
    @workflow.run
    async def run(self, config: dict) -> None:
        while True:
            report = await workflow.execute_activity(
                generate_daily_report, config,
                start_to_close_timeout=timedelta(minutes=30),
            )

            if report["anomalies_detected"]:
                await workflow.execute_activity(
                    alert_team, report,
                    start_to_close_timeout=timedelta(seconds=30),
                )

            # Sleep until next run
            await workflow.sleep(timedelta(hours=24))

3. Multi-Agent Supervisor

A supervisor workflow spawns multiple agent sub-workflows, each handling a different aspect of a task, then merges their results. Durable execution ensures that if one sub-agent fails, the supervisor can retry just that agent without re-running the others.

A supervisor workflow runs child agent workflows in parallel, merging results with individual retry isolation.

4. Checkpoint-and-Resume for Expensive Pipelines

For pipelines where each step costs $1+ in token spend, aggressive checkpointing ensures that money isn’t wasted on re-execution:

// Inngest: each step.run is individually checkpointed
const pipeline = inngest.createFunction(
  { id: "expensive-pipeline" },
  { event: "pipeline/start" },
  async ({ event, step }) => {
    // $2-3 in tokens — persisted
    const analysis = await step.run("deep-analysis", async () => {
      return await callOpus5(event.data.corpus);
    });

    // $1-2 in tokens — persisted
    const structured = await step.run("structure-output", async () => {
      return await extractStructuredData(analysis);
    });

    // $0.50 in tokens — persisted
    const summary = await step.run("generate-summary", async () => {
      return await callSonnet5(structured);
    });

    // Free — side effect only
    await step.run("store-results", async () => {
      await db.insert("results", { ...summary, pipeline_id: event.data.id });
    });

    return summary;
  }
);

If the pipeline crashes after structure-output, the resume replays the stored analysis and structured results, then executes only generate-summary and store-results. The $3-5 spent on the first two steps is preserved.

Summary

Durable execution transforms AI workflows from fragile chains of API calls into resilient, checkpointed processes that survive crashes, scale to multi-day human review cycles, and preserve expensive intermediate results.

Key takeaways:

Temporal is the right choice for teams that need maximum control, multi-language support, and battle-tested reliability at scale. The operational burden is real — plan for a dedicated cluster.
Inngest and Trigger.dev trade some flexibility for dramatically lower operational complexity. For TypeScript-centric teams building AI products, they’re the faster path to production.
Every LLM call in a workflow should be an individually persisted step. The cost of re-executing a $2 synthesis call after a crash adds up across thousands of workflow runs.
Human-in-the-loop is a first-class concern, not an afterthought. Durable execution engines make it trivial to pause a workflow for days and resume on human input, with zero compute cost during the wait.
Token budgets, model routing per step, and fallback chains are essential cost and reliability controls. Without them, a runaway agent loop or a provider outage becomes an expensive incident.
Connect workflow step metadata (workflow ID, step name) to LLM observability tools for end-to-end visibility from business workflow to individual token spend.
Use continue-as-new (Temporal) or step-based persistence (Inngest/Trigger.dev) to handle agent loops with hundreds of iterations without unbounded history growth.

Orchestrating Long-Running AI Workflows: Durable Execution, Checkpointing, and Human-in-the-Loop for AI Pipelines

Orchestrating Long-Running AI Workflows: Durable Execution, Checkpointing, and Human-in-the-Loop for AI Pipelines

Table of Contents

Why Traditional Job Queues Fail for AI Workloads

Durable Execution: The Core Abstraction

Determinism Constraints

The Orchestration Engines Compared

Temporal

Inngest

Trigger.dev

Checkpointing LLM State

Multi-Turn Conversations

Streaming Responses

KV Cache Considerations

Human-in-the-Loop Patterns

Pattern 1: Approval Gate

Pattern 2: Iterative Refinement

Pattern 3: Escalation

Error Handling and Retry Strategies for LLM Calls

Fallback Chains

Idempotency Keys

Multi-Step Agent Workflows

Agent Loop as a Workflow

Continue-As-New for Long Agent Runs

Parallel Tool Execution

Cost Control in Long-Running Workflows

Token Budgets

Model Routing by Step

Prompt Caching

Observability and Debugging

What to Track

Patterns That Work in Production

1. Fan-Out/Fan-In for Document Processing

2. Scheduled Recurring Workflows

3. Multi-Agent Supervisor

4. Checkpoint-and-Resume for Expensive Pipelines

Summary

Further Reading