Every major AI provider handles streaming differently. This guide covers SSE, WebSockets, HTTP chunked transfer, and the implementation details that make AI UIs feel responsive.

Streaming, SSE, and Real-Time AI: How Streaming Responses Work and How to Build Responsive AI UIs

Streaming is the difference between a chat interface that feels alive and one that feels broken. When a model takes 8 seconds to generate a full response, users staring at a blank screen for that duration will assume the application has crashed. Streaming the same response token-by-token creates perceived latency under 500ms — even though total generation time is identical.

The underlying mechanisms — Server-Sent Events, WebSockets, HTTP chunked transfer encoding — each carry different tradeoffs in complexity, compatibility, and failure handling. Every major LLM provider has made different choices, and the details matter when building production systems.

Why Streaming Matters for LLM Applications
The Three Transport Mechanisms
How LLM Providers Implement Streaming
Server-Sent Events in Depth
Parsing Streaming Responses
Building Responsive AI UIs
Backpressure, Buffering, and Flow Control
Error Handling and Reconnection
Streaming with Tool Use and Structured Output
Streaming Through Proxies and Gateways
Performance: Time to First Token vs Total Latency
Summary
Further Reading

Why Streaming Matters for LLM Applications {#why-streaming-matters}

Two latency numbers define the user experience of an LLM application: time to first token (TTFT) and total generation time. Without streaming, the user sees nothing until the entire response is complete. With streaming, they see the first token after TTFT — typically 200-800ms depending on provider, model, and prompt length — and then watch text arrive continuously.

Streaming exposes first-token latency separately from total generation time — the user reads while the model generates.

The perceptual impact is large. Research on typing indicators in messaging apps shows that visible progressive output reduces perceived wait time by 30-50%, even when actual completion time is unchanged. For LLM applications specifically, streaming enables users to start reading, decide early if the response is on track, and cancel bad generations before they consume the full token budget.

There’s also a practical cost angle: if a user sees the first 200 tokens of a streaming response and recognizes it’s wrong, they can cancel. Without streaming, the full 2,000-token response completes server-side before the user gets any signal.

The Three Transport Mechanisms {#transport-mechanisms}

Three HTTP-based approaches dominate LLM streaming. Each has a distinct architecture.

HTTP Chunked Transfer Encoding

The simplest mechanism. The server sends an HTTP response with Transfer-Encoding: chunked and writes data to the connection incrementally. Each chunk is prefixed with its size in hexadecimal. The connection stays open until the server sends a zero-length chunk.

This is the foundational layer — SSE builds on top of it. Raw chunked encoding provides no retry semantics, no event structure, and no built-in reconnection. Most LLM APIs don’t use raw chunked encoding directly; they layer SSE on top.

Server-Sent Events (SSE)

SSE is a W3C standard (text/event-stream content type) that defines a structured format for server-to-client streaming over HTTP. Key properties:

Unidirectional: server → client only
Text-based with a defined field format (data:, event:, id:, retry:)
Built-in reconnection with Last-Event-ID
Works through HTTP/1.1 and HTTP/2 without special negotiation
Native browser support via EventSource API

SSE is what nearly every LLM provider uses for streaming completions.

WebSockets

A full-duplex, bidirectional protocol established via HTTP upgrade handshake. After upgrade, communication happens over a persistent TCP connection with its own framing protocol.

WebSockets are overkill for standard LLM streaming (which is inherently unidirectional — the client sends a prompt, the server streams tokens back). They add complexity: connection management, ping/pong heartbeats, binary framing, and no built-in reconnection. However, WebSockets become relevant for real-time voice/audio streaming and multi-turn conversational interfaces where bidirectional communication is genuinely needed.

Each transport mechanism builds on or diverges from the previous one in complexity and capability.

Feature	SSE	WebSocket	Chunked HTTP
Direction	Server → Client	Bidirectional	Server → Client
Protocol	HTTP	WS (upgrade from HTTP)	HTTP
Reconnection	Built-in (`Last-Event-ID`)	Manual	None
Browser API	`EventSource`	`WebSocket`	`fetch` + `ReadableStream`
Proxy compatibility	Excellent	Moderate (upgrade required)	Excellent
Binary data	No (text only)	Yes	Yes
Max connections (browser)	6 per domain (HTTP/1.1)	No practical limit	6 per domain (HTTP/1.1)
LLM provider adoption	Universal	Rare (voice/real-time only)	Underlying layer

How LLM Providers Implement Streaming {#provider-implementations}

Every major provider uses SSE for text/chat completions, but the event formats, field names, and edge-case behaviors differ.

OpenAI

Set "stream": true in the request body. The response is text/event-stream with data: lines containing JSON objects. Each chunk has the same structure as the non-streaming response but with partial delta content instead of full message content.

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1720300000,"model":"gpt-5.5","choices":[{"index":0,"delta":{"content":" Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1720300000,"model":"gpt-5.5","choices":[{"index":0,"delta":{"content":" world"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1720300000,"model":"gpt-5.5","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Key details:

The delta object contains only the new content for that chunk, not the accumulated message
finish_reason is null on all chunks except the last
The stream terminates with data: [DONE] (a literal string, not JSON)
With "stream_options": {"include_usage": true}, the final chunk before [DONE] includes token usage statistics
Function/tool call arguments stream as partial JSON strings in delta.tool_calls[].function.arguments

Anthropic

Anthropic uses SSE with named event types, which is more structured than OpenAI’s single-event approach.

event: message_start
data: {"type":"message_start","message":{"id":"msg_abc","type":"message","role":"assistant","content":[],"model":"claude-sonnet-5","stop_reason":null,"usage":{"input_tokens":25,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" world"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":12}}

event: message_stop
data: {"type":"message_stop"}

Key differences from OpenAI:

Uses the SSE event: field to distinguish event types (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop)
Content is organized into numbered content blocks, which matters for tool use (each tool call is a separate content block)
Input token usage is reported in message_start; output token usage in message_delta
No [DONE] sentinel — message_stop is the terminal event

Google (Gemini)

Gemini’s streaming API returns JSON objects separated by newlines (or SSE depending on the endpoint). The REST API uses chunked JSON array syntax:

[{"candidates":[{"content":{"parts":[{"text":"Hello"}],"role":"model"}}]},
{"candidates":[{"content":{"parts":[{"text":" world"}],"role":"model"}}],"finishReason":"STOP","usageMetadata":{"promptTokenCount":10,"candidatesTokenCount":12}}]

The Gemini API SDKs abstract this into an async iterator. The AI Studio endpoint and Vertex AI endpoint have slightly different envelope structures, which is a common source of bugs when switching between them.

xAI (Grok)

Grok’s API follows OpenAI’s streaming format almost exactly — same data: line structure, same delta field, same [DONE] sentinel. This is intentional API compatibility; most OpenAI client libraries work with Grok by changing the base URL.

The basic streaming flow is the same across all providers — the differences are in event structure and field naming.

Server-Sent Events in Depth {#sse-in-depth}

Since SSE is the universal transport, understanding its specification matters for debugging and building custom parsers.

The Wire Format

An SSE stream is UTF-8 text with events separated by blank lines. Each event consists of one or more fields:

field: value\n
field: value\n
\n

The defined fields are:

Field	Purpose
`data`	Event payload. Multiple `data:` lines within one event are concatenated with newlines.
`event`	Event type name. If omitted, the event type is `"message"`.
`id`	Event ID. Sent as `Last-Event-ID` header on reconnect.
`retry`	Reconnection time in milliseconds.

Lines starting with : are comments (often used as keepalive heartbeats).

: keepalive

event: content_block_delta
data: {"type":"text_delta","text":"token"}

Browser EventSource API

The native browser API is simple but limited:

const source = new EventSource('/stream');

source.onmessage = (event) => {
  // Fires for events with no 'event' field (type "message")
  console.log(event.data);
};

source.addEventListener('content_block_delta', (event) => {
  // Fires for events with event: content_block_delta
  const data = JSON.parse(event.data);
  console.log(data.delta.text);
});

source.onerror = (event) => {
  // EventSource auto-reconnects by default
  console.error('SSE error', event);
};

The problem: EventSource only supports GET requests. LLM APIs require POST with a JSON body. This makes EventSource unusable directly with any LLM API. The workaround is fetch with ReadableStream, which is what every modern LLM client library actually uses.

Fetch + ReadableStream (the Real Way)

const response = await fetch('https://api.openai.com/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${apiKey}`,
  },
  body: JSON.stringify({
    model: 'gpt-5.5',
    messages: [{ role: 'user', content: 'Explain streaming' }],
    stream: true,
  }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  
  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('\n');
  buffer = lines.pop(); // Keep incomplete line in buffer
  
  for (const line of lines) {
    if (line.startsWith('data: ')) {
      const data = line.slice(6);
      if (data === '[DONE]') return;
      const parsed = JSON.parse(data);
      const content = parsed.choices[0]?.delta?.content;
      if (content) process.stdout.write(content);
    }
  }
}

The { stream: true } option on TextDecoder handles multi-byte UTF-8 characters that might be split across chunks — a subtle but critical detail. Without it, emoji and non-ASCII characters can render as garbage.

Parsing Streaming Responses {#parsing-streaming-responses}

Parsing SSE sounds simple until edge cases appear. Production parsers need to handle:

Chunk boundaries splitting JSON — a single data: line might arrive across two TCP segments
Multiple events per chunk — a single read() call might return 10 events at once (common during fast generation)
Empty lines as event separators — consecutive \n\n sequences
UTF-8 continuation bytes — a 4-byte emoji split across chunks
Heartbeat comments — lines starting with : that should be silently consumed

Five-stage parsing pipeline. Each stage handles a different class of edge case.

A Robust SSE Parser

type SSEEvent = {
  event: string;
  data: string;
  id?: string;
  retry?: number;
};

function* parseSSE(text: string): Generator<SSEEvent> {
  const events = text.split(/\n\n/);
  
  for (const eventBlock of events) {
    if (!eventBlock.trim()) continue;
    
    let event = 'message';
    let dataLines: string[] = [];
    let id: string | undefined;
    let retry: number | undefined;
    
    for (const line of eventBlock.split('\n')) {
      if (line.startsWith(':')) continue; // Comment
      
      const colonIndex = line.indexOf(':');
      if (colonIndex === -1) continue;
      
      const field = line.slice(0, colonIndex);
      // Spec: if the character after colon is a space, skip it
      const value = line[colonIndex + 1] === ' '
        ? line.slice(colonIndex + 2)
        : line.slice(colonIndex + 1);
      
      switch (field) {
        case 'event': event = value; break;
        case 'data': dataLines.push(value); break;
        case 'id': id = value; break;
        case 'retry': retry = parseInt(value, 10); break;
      }
    }
    
    if (dataLines.length > 0) {
      yield {
        event,
        data: dataLines.join('\n'),
        id,
        retry,
      };
    }
  }
}

This handles the full SSE spec, including multiple data: lines per event (concatenated with newlines), comment lines, and the optional space after the colon. The caller still needs to buffer incomplete events at chunk boundaries.

Provider SDK Streaming

In practice, most developers use provider SDKs that handle parsing internally:

# OpenAI Python SDK
import openai

client = openai.OpenAI()
stream = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Explain streaming"}],
    stream=True,
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

# Anthropic Python SDK
import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain streaming"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

The Anthropic SDK’s stream.text_stream is a convenience iterator that extracts just the text deltas. The full event stream is available via stream.__stream__ for cases where tool use events or metadata are needed.

Building Responsive AI UIs {#responsive-ai-uis}

The frontend rendering strategy matters as much as the transport layer. Naive implementations that append text character-by-character create layout thrashing and visible jank.

React Streaming Pattern

import { useState, useCallback, useRef } from 'react';

function useChatStream() {
  const [messages, setMessages] = useState<Message[]>([]);
  const [isStreaming, setIsStreaming] = useState(false);
  const abortRef = useRef<AbortController | null>(null);

  const send = useCallback(async (userMessage: string) => {
    abortRef.current = new AbortController();
    setIsStreaming(true);

    // Add user message and empty assistant message
    const assistantId = crypto.randomUUID();
    setMessages(prev => [
      ...prev,
      { id: crypto.randomUUID(), role: 'user', content: userMessage },
      { id: assistantId, role: 'assistant', content: '' },
    ]);

    const response = await fetch('/api/chat', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ message: userMessage }),
      signal: abortRef.current.signal,
    });

    const reader = response.body!.getReader();
    const decoder = new TextDecoder();
    let buffer = '';

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      buffer += decoder.decode(value, { stream: true });
      const lines = buffer.split('\n');
      buffer = lines.pop()!;

      for (const line of lines) {
        if (!line.startsWith('data: ') || line === 'data: [DONE]') continue;
        const parsed = JSON.parse(line.slice(6));
        const token = parsed.choices[0]?.delta?.content;
        if (token) {
          setMessages(prev =>
            prev.map(m =>
              m.id === assistantId
                ? { ...m, content: m.content + token }
                : m
            )
          );
        }
      }
    }

    setIsStreaming(false);
  }, []);

  const cancel = useCallback(() => {
    abortRef.current?.abort();
    setIsStreaming(false);
  }, []);

  return { messages, isStreaming, send, cancel };
}

Rendering Performance

Updating React state on every token (which might arrive every 20-50ms during fast generation) can cause performance problems. Three mitigation strategies:

1. Batch state updates with requestAnimationFrame:

const pendingTokens = useRef('');
const rafRef = useRef<number>();

function appendToken(token: string) {
  pendingTokens.current += token;
  
  if (!rafRef.current) {
    rafRef.current = requestAnimationFrame(() => {
      const batch = pendingTokens.current;
      pendingTokens.current = '';
      rafRef.current = undefined;
      
      setMessages(prev =>
        prev.map(m =>
          m.id === assistantId
            ? { ...m, content: m.content + batch }
            : m
        )
      );
    });
  }
}

This collapses multiple tokens into a single React render per animation frame (~16ms). On fast streams, this can reduce renders from 50/second to 60/second while batching 2-5 tokens per render.

2. Use CSS content-visibility: auto for long message lists to skip rendering off-screen messages.

3. Render Markdown incrementally. Full markdown parsing on every token is expensive. Libraries like marked or markdown-it can parse incrementally, but most implementations re-parse the entire accumulated string. For long responses, this becomes O(n²). The pragmatic solution: render raw text during streaming, apply full markdown formatting only after the stream completes or during pauses in token delivery.

Batching tokens into animation frames prevents render thrashing during fast generation.

Scroll Behavior

Auto-scrolling to the bottom as tokens arrive seems obvious but is full of UX pitfalls. If the user scrolls up to re-read earlier content, auto-scroll yanks them back down on every token. The standard solution:

function useAutoScroll(containerRef: RefObject<HTMLElement>, isStreaming: boolean) {
  const isUserScrolledUp = useRef(false);

  useEffect(() => {
    const container = containerRef.current;
    if (!container) return;

    const onScroll = () => {
      const { scrollTop, scrollHeight, clientHeight } = container;
      // User is "at bottom" if within 50px of the end
      isUserScrolledUp.current = scrollHeight - scrollTop - clientHeight > 50;
    };

    container.addEventListener('scroll', onScroll, { passive: true });
    return () => container.removeEventListener('scroll', onScroll);
  }, []);

  useEffect(() => {
    if (isStreaming && !isUserScrolledUp.current) {
      containerRef.current?.scrollTo({
        top: containerRef.current.scrollHeight,
        behavior: 'instant', // 'smooth' causes jank during streaming
      });
    }
  });
}

Use behavior: 'instant' during streaming. 'smooth' creates a queued animation that can’t keep up with token delivery, resulting in jerky scrolling that lags behind the content.

Backpressure, Buffering, and Flow Control {#backpressure}

When a server generates tokens faster than the client can consume them, backpressure becomes relevant. With SSE over HTTP/2, TCP flow control handles this automatically — if the client stops reading, the TCP receive window fills up, and the server’s writes block.

In practice, LLM token generation speed (typically 30-150 tokens/second for frontier models) is slow enough that backpressure rarely triggers for direct client connections. It becomes relevant in:

Proxy/gateway architectures where an intermediary buffers the full stream before forwarding
Server-side processing that does expensive work per token (real-time translation, TTS)
Fan-out scenarios where one stream feeds multiple clients

Node.js Backpressure with Streams

import { Transform } from 'node:stream';

// Transform that applies backpressure correctly
const tokenProcessor = new Transform({
  transform(chunk, encoding, callback) {
    // Do expensive work per chunk
    const processed = expensiveOperation(chunk);
    this.push(processed);
    callback(); // Signal readiness for next chunk
  },
  highWaterMark: 16, // Buffer only 16 chunks before applying backpressure
});

// Pipe from upstream SSE to downstream processing
upstreamResponse.body
  .pipe(tokenProcessor)
  .pipe(clientResponse);

Error Handling and Reconnection {#error-handling}

Streaming connections fail in multiple ways, and each failure mode requires different handling.

Failure Mode	Symptom	Recovery
Network drop	`read()` returns error	Reconnect with accumulated context
Server timeout	No data for N seconds	Keepalive timeout detection
Mid-stream error	Error event in stream	Parse error, show partial content
Rate limit	429 during stream	Retry with exponential backoff
Context length exceeded	Error after partial generation	Truncate input, retry
Server overload	503 or stream stall	Switch provider / retry

Anthropic’s Streaming Error Events

Anthropic can emit an error event mid-stream:

event: error
data: {"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}}

This is distinct from an HTTP error status code. The stream started successfully (200 OK) and then failed during generation. The client has partial content that it should preserve and display.

Timeout Detection

Most providers send heartbeat comments (: ping or : keepalive) to prevent intermediary proxies from closing idle connections. A robust client should implement its own timeout:

function streamWithTimeout(url: string, body: object, timeoutMs = 30000) {
  return new Promise<void>((resolve, reject) => {
    let timer: ReturnType<typeof setTimeout>;

    const resetTimer = () => {
      clearTimeout(timer);
      timer = setTimeout(() => {
        reader.cancel();
        reject(new Error(`No data received for ${timeoutMs}ms`));
      }, timeoutMs);
    };

    // ... fetch and read loop ...
    // Call resetTimer() after each successful read()
  });
}

Set the timeout generously for thinking models. Claude Opus 4.8 with extended thinking can take 60+ seconds before emitting the first content token (thinking tokens may or may not stream depending on configuration).

Error recovery flow for streaming connections. Partial content should always be preserved and shown.

Reconnection Strategy

SSE’s Last-Event-ID header enables reconnection after drops, but no LLM provider supports resuming a generation from a specific event ID. The token generation is stateful (depends on KV cache) and can’t be resumed after a connection drop.

Practical reconnection for LLM streaming means: save the partial response, re-send the original prompt with the partial response appended (as an assistant message prefix, if the API supports it), and continue generation. Anthropic supports this via the messages format — include the partial response as an assistant message, and the model continues from there. OpenAI’s messages API supports the same pattern.

# Resuming after a dropped connection (Anthropic)
partial_response = "The three main causes of "  # What we received before the drop

messages = [
    {"role": "user", "content": "What causes inflation?"},
    {"role": "assistant", "content": partial_response},  # Partial response as prefix
]

# The model continues from where it left off
with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=messages,
) as stream:
    for text in stream.text_stream:
        partial_response += text

This isn’t free — the input tokens now include the partial response, so there’s a cost penalty proportional to how much was generated before the drop.

Streaming with Tool Use and Structured Output {#streaming-tool-use}

Streaming interacts with tool use and structured output in non-obvious ways.

Tool Call Streaming

When a model decides to call a tool, the tool call arguments stream as partial JSON. This creates a problem: the arguments aren’t valid JSON until the stream completes.

data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"lo"}}]}}]}
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"catio"}}]}}]}
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"n\": "}}]}}]}
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"Par"}}]}}]}
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"is\"}"}}]}}]}

Accumulate the argument string across chunks and parse JSON only after finish_reason is "tool_calls". Some applications show a “Calling search…” indicator while arguments stream, which requires detecting the tool call intent from the first chunk (which contains the function name) before arguments are complete.

Anthropic Content Blocks

Anthropic’s streaming design handles tool use more elegantly. Each tool call is a separate content block with its own start/delta/stop lifecycle:

event: content_block_start
data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_abc","name":"search","input":{}}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"query\":"}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":" \"weather\"}"}}

event: content_block_stop
data: {"type":"content_block_stop","index":1}

The content_block_start event tells the client immediately which tool is being called, before any arguments arrive. This enables showing “Using search tool…” instantly.

Anthropic’s content block model gives clean boundaries between text and tool use during streaming.

JSON Mode and Streaming

Structured output (JSON mode) complicates streaming because partial JSON isn’t useful to display. The typical approach: stream the response, buffer it entirely, parse the complete JSON on finish. This negates the UX benefit of streaming, but the latency improvement from TTFT still helps — the client knows generation has started even if it can’t show partial results.

Some applications parse partial JSON for progress indication. A streaming JSON object like {"title": "Complete", "sections": [{"heading": "Intro can be parsed to show “Generated title… working on sections…” without displaying raw JSON fragments.

Streaming Through Proxies and Gateways {#proxies-and-gateways}

Production deployments rarely have direct client-to-provider connections. Reverse proxies, API gateways, CDNs, and load balancers sit in the path, and each can break streaming.

Common Problems

Nginx buffering: By default, Nginx buffers proxy responses. This converts a streaming response into a batch response — the client sees nothing until the entire response is buffered.

location /api/chat {
    proxy_pass http://backend;
    
    # Disable buffering for SSE
    proxy_buffering off;
    proxy_cache off;
    
    # Disable gzip (interferes with streaming)
    proxy_set_header Accept-Encoding "";
    
    # Prevent timeout during long generations
    proxy_read_timeout 300s;
    
    # Required for HTTP/1.1 keepalive
    proxy_http_version 1.1;
    proxy_set_header Connection "";
}

CloudFlare/CDN buffering: Most CDNs buffer responses for caching. For SSE endpoints, CDN caching must be bypassed. CloudFlare requires Cache-Control: no-cache and X-Accel-Buffering: no headers. Even then, some CDN configurations silently buffer small chunks.

AWS ALB: Application Load Balancers support streaming but have a default idle timeout of 60 seconds. Thinking models that pause before generating can hit this. Increase to 300-600 seconds for LLM workloads.

API Gateway (AWS): AWS API Gateway’s REST API type does NOT support streaming. The HTTP API type supports it but with a 10MB response payload limit and 30-second integration timeout (extendable to 30 minutes with custom configuration as of 2025). For long LLM responses, this is tight. Many teams use Lambda Function URLs or direct ALB → ECS/Lambda instead.

Every layer between client and provider must be configured for streaming passthrough.

Server-Side Streaming Proxy in Node.js

import express from 'express';

const app = express();

app.post('/api/chat', async (req, res) => {
  // Set SSE headers
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('X-Accel-Buffering', 'no'); // Nginx
  res.flushHeaders();

  const upstream = await fetch('https://api.anthropic.com/v1/messages', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'x-api-key': process.env.ANTHROPIC_API_KEY!,
      'anthropic-version': '2023-06-01',
    },
    body: JSON.stringify({
      model: 'claude-sonnet-5',
      max_tokens: 1024,
      stream: true,
      messages: req.body.messages,
    }),
  });

  // Pipe upstream SSE directly to client
  const reader = upstream.body!.getReader();
  
  // Detect client disconnect
  req.on('close', () => {
    reader.cancel();
  });

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    
    // Optionally transform, log, or filter events here
    res.write(new TextDecoder().decode(value));
    
    // Force flush (important for Node.js)
    if (typeof (res as any).flush === 'function') {
      (res as any).flush();
    }
  }

  res.end();
});

The res.flushHeaders() call is critical. Without it, Node.js may buffer the initial headers along with the first chunk of data, adding latency to TTFT.

Performance: Time to First Token vs Total Latency {#performance}

TTFT depends on multiple factors:

Network round-trip to the provider (50-200ms depending on region)
Prompt processing (prefill) — proportional to input token count. Long prompts with 100K+ tokens can add seconds to TTFT.
Queue wait time — during peak load, providers queue requests. This can add hundreds of milliseconds to seconds.
Model size — larger models generally have higher TTFT
Thinking/reasoning — extended thinking models (thinking variants) may delay first visible content by seconds or tens of seconds while reasoning

TTFT is the sum of network latency, queue wait, and prefill time. Queue wait is the most variable component.

Measuring TTFT Accurately

async function measureStreamPerformance(url: string, body: object) {
  const requestStart = performance.now();
  
  const response = await fetch(url, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(body),
  });
  
  const headerTime = performance.now();
  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  
  let ttft: number | null = null;
  let totalTokens = 0;
  let buffer = '';
  
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    
    if (ttft === null) {
      ttft = performance.now() - requestStart;
    }
    
    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split('\n');
    buffer = lines.pop()!;
    
    for (const line of lines) {
      if (line.startsWith('data: ') && line !== 'data: [DONE]') {
        totalTokens++;
      }
    }
  }
  
  const totalTime = performance.now() - requestStart;
  const generationTime = totalTime - (ttft ?? 0);
  const tokensPerSecond = totalTokens / (generationTime / 1000);
  
  return {
    ttft: Math.round(ttft ?? 0),
    totalTime: Math.round(totalTime),
    headerLatency: Math.round(headerTime - requestStart),
    totalTokens,
    tokensPerSecond: Math.round(tokensPerSecond * 10) / 10,
  };
}

Note that totalTokens here counts SSE events, not actual tokens — some events contain multiple tokens, and some contain metadata without content. For accurate token counts, use the usage statistics from the final event.

Streaming vs Non-Streaming Token Costs

Streaming adds negligible overhead to token costs. The same number of tokens are generated regardless of streaming mode. However, there’s a subtle infrastructure cost: streaming connections are held open for the full generation duration (potentially 10-30 seconds), consuming a connection slot the entire time. For high-concurrency servers, this means more open connections per unit of throughput compared to non-streaming batch requests.

Some providers charge identically for streaming and non-streaming. Others may internally batch non-streaming requests more efficiently, but this is transparent to the user.

Voice and Real-Time Audio Streaming

Voice interfaces introduce bidirectional requirements that SSE can’t satisfy. OpenAI’s Realtime API and Google’s Gemini Live API both use WebSockets for voice-to-voice streaming:

Audio input streams from client to server continuously
Text and audio output stream from server to client simultaneously
Interruption detection requires the server to receive audio while sending audio
Sub-200ms round-trip latency is required for natural conversation

Voice AI requires full-duplex WebSocket connections — SSE cannot support bidirectional audio streaming.

The WebSocket protocol for voice APIs typically uses a JSON envelope for control messages (session configuration, function calls, turn management) and binary frames for audio data. This mixed text/binary framing is another reason WebSockets are necessary here — SSE is text-only.

For text chat, SSE remains the better choice. It’s simpler, works through more infrastructure, and auto-reconnects. The common mistake is using WebSockets for text chat “because it seems more modern” — this adds connection management complexity without benefit.

Summary {#summary}

SSE is the universal transport for LLM text streaming. Every major provider uses it. WebSockets are only necessary for bidirectional use cases like voice.
The browser EventSource API can’t be used with LLM APIs because it only supports GET. Use fetch + ReadableStream.
Event formats differ across providers. OpenAI uses data: lines with [DONE] sentinel. Anthropic uses named event: types with structured content blocks. Grok mirrors OpenAI’s format. Google returns chunked JSON.
Buffering is the most common production issue. Nginx, CDNs, and API gateways all default to buffering responses. Every layer must be explicitly configured for streaming passthrough.
Batch UI updates with requestAnimationFrame to prevent render thrashing during fast token generation. Defer full markdown parsing until stream completion.
Partial JSON from tool calls must be accumulated and parsed only after the tool call completes. Anthropic’s content block model provides cleaner boundaries than OpenAI’s delta-based approach.
TTFT is the metric that matters for perceived responsiveness. It’s the sum of network latency, queue wait, and prefill time — and it’s independent of total generation time.
Error recovery means re-sending the prompt with accumulated partial output as context. No provider supports true mid-stream resume via SSE Last-Event-ID.
Auto-scroll must respect user scroll position. Use behavior: 'instant' during streaming and stop auto-scrolling when the user scrolls up.