Data Improvements Drive AI Pretraining Progress
The impact of data improvements on AI pretraining from 2019 to 2025 was analyzed, showing they yield greater efficiency than model enhancements.
Research into the rapid advancements in AI over recent years reveals that a significant portion of progress stems from data rather than model enhancements. Between 2019 and 2025, findings indicate that data improvements have resulted in 3.24 times more compute efficiency gains compared to model improvements. This insight has implications for the economics of AI labs and the pace of future advancements.
This synthesis was produced from its source by AI; there is no human editor or manual review step. How we work