TokenRouter: A Serving Engine for Token-Level LLM Routing
TokenRouter serves as a token-level LLM routing engine, enabling collaboration among models.
TokenRouter enables collaboration between small and large models within a single response through token-level routing. This system allows each model to operate at its own pace while utilizing a scheduler to efficiently manage incoming requests. Additionally, a user-friendly interface facilitates the definition of routing policies.
This synthesis was produced from its source by AI; there is no human editor or manual review step. How we work