Office Hours — How do you handle long-running LLM requests and API latency in user-facing applications?

How do you handle long-running LLM requests and API latency in user-facing applications?

Long-running LLM requests are a real problem in production. A single request to GPT-5.6 Sol or Claude Opus 4.8 can take 15–45 seconds depending on output length and load, and that’s before network variance. If you’re waiting synchronously for that response in a user-facing application, you’ll timeout, lose users, or burn through your infrastructure costs on idle connections. This is one of the highest-impact operational problems you’ll hit, and the solution isn’t sexy—it’s architectural.

The Synchronous Timeout Wall

If your user-facing app calls an LLM API and waits for the response, you hit latency limits fast. HTTP request timeouts are typically 30 seconds; LLM APIs can exceed that. You also have queuing latency on the provider’s side (especially during peak hours), network round-trip time, and then actual model inference. Add reasoning tokens or a long context window, and you’re easily looking at 60+ seconds per request.

The knee-jerk fix—increasing timeouts—is wrong. It just moves the problem downstream. Your load balancer times out, your reverse proxy times out, or your client gives up anyway. You’re now burning server connections on requests that won’t complete.

Queue Everything: Request-Response to Request-Polling

The standard pattern is to decouple the user request from the LLM call. Instead of synchronous waiting, you queue the request, return a ticket immediately, and let the user poll for results.

# FastAPI example: user-facing endpoint
@app.post("/summarize")
async def submit_summary(doc: UploadFile):
    job_id = str(uuid.uuid4())
    # Queue the job, return immediately
    task_queue.enqueue(
        "summarize_document",
        job_id=job_id,
        doc_content=await doc.read()
    )
    return {"job_id": job_id, "status_url": f"/jobs/{job_id}"}

@app.get("/jobs/{job_id}")
async def check_job(job_id: str):
    result = cache.get(f"job:{job_id}")
    if result is None:
        return {"status": "pending"}
    return {"status": "complete", "result": result}

# Background worker: processes LLM request
def summarize_document(job_id, doc_content):
    response = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=2048,
        messages=[{"role": "user", "content": f"Summarize: {doc_content}"}]
    )
    cache.set(f"job:{job_id}", response.content[0].text, ex=3600)

This decouples user latency from LLM latency completely. The user gets a 200 response in 50ms, and the LLM request happens in the background. The user polls /jobs/{job_id} every 500ms or 2 seconds depending on UX expectations.

The trade-off is you now need to manage job state (database or Redis), handle job failures gracefully, and give users a way to know they should keep polling. For most production cases, this is non-negotiable.

Streaming: When You Can’t Queue

If the user interaction requires real-time feedback—like a chat interface—queuing doesn’t work. You need streaming. Both Claude and GPT-5.6 Sol support streaming responses via Server-Sent Events (SSE). The key is to stream directly to the client instead of waiting for the full response.

from fastapi.responses import StreamingResponse
import anthropic

@app.post("/chat")
async def stream_chat(messages: list):
    client = anthropic.Anthropic()
    
    def event_generator():
        with client.messages.stream(
            model="claude-opus-4-8",
            max_tokens=2048,
            messages=messages
        ) as stream:
            for text in stream.text_stream:
                yield f"data: {text}\n\n"
    
    return StreamingResponse(event_generator(), media_type="text/event-stream")

The client receives tokens as they arrive, so the perceived latency is much lower. The first token might appear in 1–2 seconds instead of 30. This is especially important for conversational interfaces where users expect real-time feedback.

The catch: streaming doesn’t work for all use cases. If you need to batch-process structured outputs or you’re doing downstream processing on the complete response, streaming adds complexity. You have to buffer the stream or process tokens incrementally.

Batching and Off-Peak Scheduling

For non-interactive workloads, batch requests and send them during off-peak hours to get better availability and sometimes lower latency. Anthropic and OpenAI both offer batch APIs that cost less than real-time inference but process requests with higher latency (hours or days).

# Use batch API for non-urgent summarization
batch_requests = [
    {
        "custom_id": f"doc-{i}",
        "params": {
            "model": "claude-opus-4-8",
            "max_tokens": 1024,
            "messages": [{"role": "user", "content": doc}]
        }
    }
    for i, doc in enumerate(documents)
]

batch_response = client.beta.messages.batches.create(requests=batch_requests)
# Check results after a few hours

If your use case is “summarize 10,000 documents overnight,” batch APIs are 30–50% cheaper and reduce provider load. For user-facing work, this doesn’t apply.

Caching: Request and Response

Prompt caching (both Claude and GPT-5.6 Sol support this) reduces latency and cost for repeated contexts. If you’re analyzing the same legal document repeatedly, the system caches the processed context tokens after the first request.

# Claude: explicit cache headers
response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": legal_document_text,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": "Summarize the liability clause"}]
)

The first request processes the full document. Subsequent requests within 5 minutes (default TTL) reuse the cached tokens, cutting latency by 2–3x and cost by 50% or more. This is free money if your workload involves repeated analysis of the same content.

Also cache at the application level. Store successful responses in Redis or your database keyed by input. A user asking the same question twice shouldn’t hit the LLM API twice.

Parallel Requests and Speculation

If you’re making multiple independent LLM calls, parallelize them instead of chaining them. Use asyncio.gather() in Python or equivalent concurrency primitives in your language.

async def process_batch(items):
    tasks = [
        client.messages.create_async(
            model="claude-opus-4-8",
            max_tokens=512,
            messages=[{"role": "user", "content": item}]
        )
        for item in items
    ]
    results = await asyncio.gather(*tasks)
    return results

If you’re speculating on what the user will ask next, you can pre-compute LLM responses for the most likely next steps and cache them. This is rarely worth the complexity unless you have a very narrow action space.

Timeouts and Graceful Degradation

Always set explicit timeouts on LLM API calls. Don’t rely on framework defaults.

import httpx

client = anthropic.Anthropic(
    timeout=httpx.Timeout(60.0, connect=10.0)
)

try:
    response = client.messages.create(...)
except httpx.TimeoutException:
    # Return cached fallback, shorter summary, or user-friendly error
    return {"result": "Request took too long. Please try again."}

If an LLM request times out, don’t just fail. Return a cached fallback if available, or degrade gracefully (shorter output, simpler logic, or a human escalation). Users would rather get a degraded

Question via Hacker News