Designing Kolibri: Architectural Choices and Their Impact
Explore the architectural design of Kolibri, analyzing FLOPs and parameter distribution. Use the interactive tool to configure your model.
This post explores the impact of architectural choices in autoregressive language models on training and deployment costs. It examines the allocation of parameters and FLOPs, how these scale with context length, and the size of sequence-mixer states. An interactive tool is also provided for users to configure their own models and compare them with recent open-weight architectures.
This synthesis was produced from its source by AI; there is no human editor or manual review step. How we work