Scaling Laws Explained: From Kaplan to Chinchilla to Overtraining
A breakdown of LLM scaling laws from Kaplan to Chinchilla, and why modern models are deliberately overtrained to cut inference costs.
For years, labeled-data scarcity — not compute — was the real ceiling on model quality, tied to the classic bias-variance tradeoff. Self-supervision broke that ceiling by turning raw text into its own label, making compute the only real constraint left, and raising a new question: given a fixed compute budget, how should it be split between model size and data?
The training-cost formula C≈6×N×D governs every run, where N is parameters and D is tokens. OpenAI's 2020 Kaplan scaling laws suggested most compute should go toward bigger models, producing GPT-3 and PaLM: enormous parameter counts trained on comparatively little data. DeepMind's 2022 Chinchilla paper corrected the methodology and found parameters and tokens should scale equally — a smaller, better-fed Chinchilla model outperformed the much larger GPT-3.
But Chinchilla only optimizes training cost, ignoring inference, which is multiplied by every query a deployed model ever serves. That's why modern models like LLaMA 3 8B are deliberately overtrained far past the Chinchilla-optimal ratio: pay more upfront in training to shrink the model and cut per-query inference cost. For agentic pipelines where one request fans out into dozens of model calls, that tradeoff isn't optional — it's the architecture.