How to Stream LLM Responses in a Web App

The short answer

Streaming means the user sees the answer word by word instead of waiting for the whole response. The chain has three links:

1. Backend → LLM API: enable streaming mode in the provider's SDK and receive chunks as they are generated.
2. Backend → browser: forward the chunks in a streaming HTTP response, usually in Server-Sent Events (SSE) format.
3. Browser → UI: read the stream and append text to the message on screen.

Never call the LLM API directly from the browser: the API key would be exposed. The backend is always in the middle.

SSE or a raw fetch stream

For chat, the most practical choice is fetch with a POST request and an SSE-formatted response: you can send message history in the body and still keep a clear event structure.

Backend: forwarding the stream

The example uses the standard Fetch API Response and ReadableStream, which works in modern Node.js runtimes and route handlers. llm.stream stands in for your provider SDK's streaming call.

export async function POST(req: Request) {
  const { messages } = await req.json();
  const encoder = new TextEncoder();
  const send = (obj: unknown) => encoder.encode(`data: ${JSON.stringify(obj)}\n\n`);

  const body = new ReadableStream({
    async start(controller) {
      try {
        for await (const text of llm.stream(messages, { signal: req.signal })) {
          controller.enqueue(send({ text }));
        }
        controller.enqueue(send({ done: true }));
      } catch {
        controller.enqueue(send({ error: "generation_failed" }));
      } finally {
        controller.close();
      }
    },
  });

  return new Response(body, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache, no-transform",
      "X-Accel-Buffering": "no",
    },
  });
}

Key points: pass the request's abort signal to the LLM call so generation stops when the user leaves, and send errors as a regular event, because the HTTP status is already 200 by the time a mid-stream failure happens.

Frontend: reading and cancelling

const controller = new AbortController();

async function ask(messages: Message[], onText: (t: string) => void) {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
    signal: controller.signal,
  });
  if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);

  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;
    const events = buffer.split("\n\n");
    buffer = events.pop() ?? "";
    for (const event of events) {
      if (!event.startsWith("data: ")) continue;
      const msg = JSON.parse(event.slice(6));
      if (msg.error) throw new Error(msg.error);
      if (msg.text) onText(msg.text);
    }
  }
}
// "Stop" button: controller.abort();

A network chunk does not equal one event: a chunk may contain half an event or several. That is why the code keeps a buffer and only parses complete events separated by a blank line.

Rendering markdown as it arrives

• Re-render the whole accumulated text, not each token separately: markdown depends on context (lists, code blocks).
• Throttle updates, for example once per animation frame, to avoid re-rendering on every small chunk.
• An unclosed code block or table mid-stream is normal; a good markdown renderer handles it, and the final render fixes the layout.
• Sanitize HTML. Model output is untrusted; do not inject raw HTML into the page.
• Keep auto-scroll only while the user is at the bottom of the chat.

Errors and proxy buffering

The most common complaint is "streaming works locally but the answer appears all at once in production". The cause is almost always buffering:

• Nginx buffers proxied responses by default. Disable it with proxy_buffering off; for the route or send the X-Accel-Buffering: no header.
• Compression (gzip/brotli) at the proxy or CDN may collect data before sending. Exclude text/event-stream.
• CDNs and load balancers may have idle timeouts. Send a periodic comment line such as : ping during long pauses.
• Serverless platforms may limit response duration; check your platform's streaming support.

Also handle: the user pressing "Stop" (keep the partial answer), network drops (show a retry button), and rate limit errors from the provider (show a clear message rather than an empty bubble).