LLM Streaming: SSE Implementation

Users Wait Too Long for Complete LLM Responses

Your self-hosted LLM generates a 500-token response in 3 seconds, but the user stares at a blank screen until the entire response is ready. Cloud LLM APIs stream tokens as they are generated, giving users instant feedback. Your vLLM or Ollama deployment on your GPU server can do the same using Server-Sent Events — the user sees the first word within 100 milliseconds of their request.

vLLM Streaming API

Enable streaming by adding "stream": true to your request. vLLM returns Server-Sent Events as tokens are generated:

# Streaming request to vLLM
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -N \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain how GPUs work"}],
    "stream": true,
    "temperature": 0.7
  }'

# Response arrives as SSE events:
# data: {"choices":[{"delta":{"content":"GPUs"},"index":0}]}
# data: {"choices":[{"delta":{"content":" are"},"index":0}]}
# data: {"choices":[{"delta":{"content":" specialized"},"index":0}]}
# ...
# data: [DONE]

Python Streaming Client

Consume the SSE stream in Python for backend applications:

import requests, json

def stream_llm_response(prompt, model="meta-llama/Meta-Llama-3.1-8B-Instruct"):
    response = requests.post(
        "http://localhost:8000/v1/chat/completions",
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "stream": True,
            "temperature": 0.7
        },
        stream=True
    )

    full_response = ""
    for line in response.iter_lines():
        if line:
            line = line.decode("utf-8")
            if line.startswith("data: "):
                data = line[6:]
                if data == "[DONE]":
                    break
                chunk = json.loads(data)
                token = chunk["choices"][0]["delta"].get("content", "")
                full_response += token
                print(token, end="", flush=True)  # Stream to terminal

    return full_response

# Usage
result = stream_llm_response("Explain how GPUs accelerate AI inference")

FastAPI Streaming Proxy

Build a streaming API endpoint that proxies to vLLM while adding authentication and rate limiting:

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import httpx

app = FastAPI()

async def stream_from_vllm(payload):
    async with httpx.AsyncClient() as client:
        async with client.stream(
            "POST",
            "http://localhost:8000/v1/chat/completions",
            json=payload,
            timeout=120.0
        ) as response:
            async for line in response.aiter_lines():
                if line.startswith("data: "):
                    yield f"{line}\n\n"
                    if line == "data: [DONE]":
                        break

@app.post("/api/chat")
async def chat_stream(request: Request):
    body = await request.json()
    payload = {
        "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
        "messages": body["messages"],
        "stream": True,
        "temperature": body.get("temperature", 0.7),
    }
    return StreamingResponse(
        stream_from_vllm(payload),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no"  # Disable Nginx buffering
        }
    )

Browser-Side SSE Consumption

Consume the stream in a web frontend for real-time token display:

<script>
async function streamChat(prompt) {
  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: {'Content-Type': 'application/json'},
    body: JSON.stringify({
      messages: [{role: 'user', content: prompt}]
    })
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  const outputEl = document.getElementById('response');
  outputEl.textContent = '';

  while (true) {
    const {done, value} = await reader.read();
    if (done) break;

    const text = decoder.decode(value);
    const lines = text.split('\n');
    for (const line of lines) {
      if (line.startsWith('data: ') && line !== 'data: [DONE]') {
        try {
          const data = JSON.parse(line.slice(6));
          const token = data.choices[0].delta.content || '';
          outputEl.textContent += token;
        } catch (e) {}
      }
    }
  }
}
</script>

Handling Stream Errors and Timeouts

Production streaming needs robust error recovery:

import asyncio, httpx

async def stream_with_recovery(payload, max_retries=3):
    for attempt in range(max_retries):
        try:
            async with httpx.AsyncClient() as client:
                async with client.stream(
                    "POST", "http://localhost:8000/v1/chat/completions",
                    json=payload, timeout=120.0
                ) as response:
                    if response.status_code != 200:
                        raise httpx.HTTPStatusError(
                            f"Status {response.status_code}",
                            request=response.request, response=response)

                    async for line in response.aiter_lines():
                        if line.startswith("data: "):
                            yield line + "\n\n"
                            if line == "data: [DONE]":
                                return
        except (httpx.ReadTimeout, httpx.ConnectError) as e:
            if attempt == max_retries - 1:
                yield f"data: {{\"error\": \"{str(e)}\"}}\n\n"
                return
            await asyncio.sleep(1)

# Nginx configuration for SSE passthrough
# proxy_buffering off;
# proxy_cache off;
# proxy_read_timeout 120s;
# add_header X-Accel-Buffering no;

SSE streaming transforms the user experience of self-hosted LLMs on your GPU server. The vLLM production guide covers Nginx proxy configuration for streaming. Check the LLM hosting section for architecture patterns, tutorials for deployment guides, and benchmarks for token generation speed per GPU.