In-House LLM Serving at Netflix: Design and Implementation
Netflix runs LLMs in-house, discussing key decisions on model deployment and API design.
Netflix manages LLMs fully in-house within its production environment. This article discusses critical decisions around model deployment, API design, and deployment strategies. Challenges and solutions that arose during production are shared.