How Netflix Runs Its Own LLM Serving Stack with vLLM and Triton
Netflix engineers explain how they built an in-house LLM serving stack using vLLM, Triton, and an OpenAI-compatible API, with real production lessons.
Unlike most organizations that consume LLMs through hosted APIs, Netflix runs the entire serving stack in-house — from model deployment to inference — inside its existing production environment. This piece details the engineering decisions behind that system: engine selection, model packaging, API surface design, and rollout strategy, all built on a unified JVM-based serving layer that supports both gRPC and direct HTTP paths, with GPU-heavy models routed to a Triton-backed Model Scoring Service.
The platform originally ran on TensorRT-LLM but switched to vLLM by summer 2025 as open-source engines closed the performance gap and Netflix's workload diversified into embedding generation, prefill-only inference, and custom constrained-decoding logic. Inside Triton, the team chose the vLLM backend over the Python backend for packaging, decoupling model artifacts from frontend upgrades — though version mismatches between Triton and vLLM caused real production outages requiring careful version pinning.
To keep LLMs from becoming special-cased services, Netflix added an OpenAI-compatible HTTP frontend alongside gRPC, easing the path from hosted models to self-hosted fine-tuned ones. Adopting Triton's OpenAI-compatible frontend surfaced a silent bug — the response_format field was dropped before reaching vLLM, disabling guided decoding without any error — which the team patched directly. Production also revealed coordination gaps in red-black deployments when schema changes accompany a new model version, causing in-flight requests to fail during migration windows.