How Netflix Built Its Own In-House LLM Serving Stack
Netflix details its in-house LLM serving architecture: vLLM engine choice, Triton packaging, OpenAI-compatible API, and production rollout issues.
Netflix runs large language model inference entirely in-house rather than relying on hosted APIs, integrating it into its existing production ML infrastructure. The AI Platform team detailed an architecture built around a gRPC-based JVM serving layer, a Model Scoring Service (MSS) backed by NVIDIA Triton, and routing that sends small CPU models in-process while larger GPU models go to a shared remote backend.
On engine choice, Netflix moved from TensorRT-LLM to vLLM after open-source engines closed the performance gap and workloads diversified to include embeddings, prefill-only retrieval, and autoregressive decoding. vLLM won on operational fit — faster iteration on custom architectures, extensibility for custom decoding logic, and easier debuggability. For packaging, the team chose Triton's vLLM backend, which generates I/O tensor specs dynamically at deploy time instead of freezing them in the artifact, decoupling model updates from frontend changes. Production still surfaced friction: Triton-vLLM API mismatches could break backend loading entirely, forcing strict version pinning.
On the API surface, an OpenAI-compatible HTTP frontend was added alongside gRPC so switching from a hosted model to a fine-tuned self-hosted one requires minimal code changes. Adopting Triton's OpenAI frontend exposed a bug where response_format was silently dropped, letting malformed JSON through without guided decoding — the team patched the frontend to fix it. On rollouts, the Red-Black deployment strategy revealed a coordination gap: when a new model version needs a schema change, old requests can still hit the new deployment mid-migration and fail.