I Trained a 30M-Parameter LLM from Scratch: The Scaling "Floor" Was a Mirage
A 30M parameter language model was trained from scratch, aiming to deepen understanding of transformers.
A small language model, approximately 30M parameters, was built from scratch as a learning project, trained on the TinyStories dataset. The project aims to deepen understanding of transformers by implementing core components manually and conducting controlled experiments, culminating in a technical write-up.