Streaming LLM Responses in Django and React: A Practical Guide
A practical guide to streaming LLM responses in Django with Server-Sent Events, covering React integration, Gunicorn workers, Nginx buffering and Celery.
When a Django app first shipped an AI feature the naive way — POST, wait, render — users abandoned it despite accurate responses, simply because an 8-second wait before any output felt broken next to ChatGPT-trained expectations. The fix was streaming via Server-Sent Events, since LLM communication is one-directional and doesn't need WebSockets. On the server, Django's StreamingHttpResponse wraps a generator that yields SSE-formatted chunks as tokens arrive from the model; on the client, fetch with a ReadableStream reader is used instead of EventSource, since EventSource can't POST JSON bodies.
Three infrastructure details break streaming in production that work fine locally: Nginx buffers responses by default unless you set X-Accel-Buffering: no; Gunicorn's sync workers block for the entire stream duration, requiring a switch to gevent or gthread workers; and load balancers like AWS ALB have a 60-second idle timeout that can kill long-running LLM responses unless raised to around 300 seconds. When AI calls run through Celery, tasks can't stream directly, so chunks are written to Redis and a thin Django view relays them to the client using the same SSE pattern.
Streaming doesn't reduce actual latency — time-to-first-token stays the same, which matters for use cases like real-time autocomplete where a smaller or faster model is the real fix. It also doesn't solve mid-stream failures: if generation breaks after partial output, the client has already received those tokens, so teams typically just show an 'interrupted' notice and let users retry rather than build complex reconnect logic. The overall lesson is that streaming meaningfully improves perceived UX, but only if the surrounding infrastructure — proxies, workers, timeouts, and queues — is configured correctly.