What is Scaling Law?
TL;DR
A predictable relationship between model size, data volume, compute, and performance that follows power-law patterns.
Scaling Law: Definition & Explanation
Scaling laws describe the observation that AI model performance improves predictably according to power laws as model parameter count, training data volume, and compute (FLOPs) increase. This was systematically demonstrated in OpenAI's research (Kaplan et al., 2020) and further refined in DeepMind's Chinchilla paper (2022). These laws provide theoretical backing that large-scale compute investments directly translate to performance gains, justifying the development of massive models like GPT-4 and Gemini. However, there are also arguments that scaling alone has limitations, with growing attention on approaches beyond scaling such as data quality, architectural improvements, and test-time compute.