Ever spent months training a massive language model, only to realize you could have gotten better results with half the parameters and twice the data? You’re not alone. For years, the industry chased bigger parameter counts, assuming that scaling laws guaranteed performance gains simply by throwing more hardware at the problem. But the landscape has shifted. Today, understanding how loss scales with parameters, data, and compute isn’t just academic-it’s the difference between a profitable product and a budget overrun.
If you are an engineer or ML practitioner planning your next pretraining run, you need more than intuition. You need a quantitative framework. This guide breaks down what modern scaling laws actually tell us about large language models (LLMs), moving beyond the hype to the math that dictates your GPU bill.
The Core Equation: What Scales?
At its heart, neural scaling law research tries to answer one question: How does test loss change when we increase model size, dataset size, or compute? The foundational work by OpenAI in 2020 established that these relationships follow power laws. In simple terms, if you double your resources, your error doesn't halve; it drops by a predictable, smaller fraction.
Practitioners characterize a model using four variables:
- N: Number of parameters (model size).
- D: Number of training tokens (dataset size).
- C: Total compute used (floating-point operations).
- L: Test cross-entropy loss (performance metric).
The original Kaplan et al. findings suggested that increasing N was the most efficient way to reduce loss. This led to the "scale is all you need" era, where labs like OpenAI built GPT-3 with 175 billion parameters trained on roughly 300 billion tokens. It worked, but it wasn't necessarily optimal.
The Chinchilla Correction: Balance is Key
In 2022, DeepMind dropped a bombshell paper titled "Training Compute-Optimal Large Language Models," commonly known as the Chinchilla paper. They argued that previous models were under-trained relative to their size. Their revised scaling laws showed that for a fixed compute budget C, you should scale parameters N and data D equally.
| Feature | Kaplan et al. (2020) | Hoffmann et al. / Chinchilla (2022) |
|---|---|---|
| Optimal Data Scaling | Sublinear ($D \propto N^{0.74}$) | Linear ($D \propto N$) |
| Compute Allocation | Favor larger models, less data | Equal growth in model size and data |
| Rule of Thumb | Maximize N first | ~20 tokens per parameter |
| Impact | GPT-3 architecture dominance | Shift to trillion-token datasets |
The Chinchilla law implies that if you want to increase your compute budget by 10x, you shouldn't just make your model 10x bigger. Instead, you should increase both your model size and your dataset size by roughly 3.1x each. This balanced approach often yields lower loss than asymmetric scaling. If you're starting a new project today, assume Chinchilla-style balanced scaling unless you have specific evidence to do otherwise.
Data Quality Trumps Quantity
Here is where many practitioners get burned. Scaling laws assume high-quality, diverse data. Recent studies, including a 2025 ACL paper, highlight "sub-scaling" phenomena. This happens when adding more data yields diminishing returns because the new data is redundant or low-quality.
Imagine you are scraping web text. The first 100 billion tokens might be unique articles. The next 100 billion might be duplicates or boilerplate. According to recent meta-analyses, doubling your token count on dense, repetitive data won't track the ideal power-law curve. You will hit a wall.
To avoid this trap:
- Deduplicate aggressively: Remove near-duplicates before they enter your training set.
- Balance domains: Ensure your mix of code, books, web, and scientific papers reflects your target use case.
- Prioritize quality over volume: A smaller, curated dataset often outperforms a massive, noisy one at the same compute cost.
How to Use Scaling Laws for Planning
You don't need to train your final model to know if it will work. You can predict its performance using smaller proxies. A 2025 study from MIT and IBM Watson AI Lab analyzed hundreds of models and found that careful fitting on small models can predict large-model performance with an absolute relative error as low as 4%.
Follow this workflow for accurate forecasting:
- Define your budget: Decide on your total compute FLOPs upfront.
- Train a ladder of models: Train at least five models of varying sizes within the same architecture family. Do not just train one small model and extrapolate wildly.
- Use intermediate checkpoints: Don't wait for the end of training. Capture loss values at multiple steps during training to fit the curve more robustly.
- Discard early noise: Ignore the first ~10 billion tokens of training data when fitting your scaling law, as early convergence is unstable.
This method allows you to simulate different $N$ and $D$ combinations virtually. You can ask questions like, "If I add 500 GPUs, should I make the model wider or train it longer?" without wasting weeks on failed runs.
Scaling Beyond Training: Inference-Time Compute
Scaling laws aren't just for training. They also apply to deployment. Known as inference-time scaling, this concept suggests that you can improve model accuracy after training by spending more compute during generation.
Techniques like chain-of-thought prompting, best-of-N sampling, or tree-of-thought reasoning increase test-time compute (TTC). Just like training scaling, there are diminishing returns here too. Running 100 samples instead of 10 might give you a marginal accuracy boost, but at a 10x latency cost.
For production systems, map out your TTC-quality tradeoffs. If your application can tolerate higher latency, inference-time scaling offers a way to squeeze more performance out of a static model without retraining. This is particularly useful when your base model is already at the limit of its architectural capacity.
Common Pitfalls to Avoid
Even with good laws, practitioners often misapply them. Here are three traps to watch out for:
1. Borrowing exponents across architectures. Scaling laws fitted on decoder-only transformers may not apply to encoder-decoder models or mixture-of-experts architectures. Always validate on your specific architecture family.
2. Ignoring optimization dynamics. Better optimizers (like AdamW with tuned betas) or learning rate schedules can shift the entire scaling curve downward. If you change your optimizer, refit your scaling laws.
3. Overlooking hardware constraints. Scaling laws predict loss, not throughput. A theoretically optimal model might require memory configurations that are inefficient on your specific GPU cluster. Factor in engineering realities like memory bandwidth and parallelism efficiency.
Key Takeaways for Practitioners
- Start Balanced: Assume equal scaling of parameters and data (Chinchilla style) until proven otherwise.
- Validate Locally: Fit scaling laws on your own data and architecture using a ladder of small models.
- Curate Data: Data quality is a multiplier. Poor data causes sub-scaling, breaking the power-law predictions.
- Plan for Inference: Consider test-time compute as a lever for performance improvement post-training.
- Treat Laws as Guides: Scaling laws are powerful first-order approximations, not crystal balls. Update them as you introduce new architectural features.
What is the Chinchilla scaling law?
The Chinchilla scaling law, proposed by DeepMind in 2022, states that for compute-optimal training, model size (parameters) and dataset size (tokens) should scale equally with the total compute budget. This contrasts with earlier laws that favored larger models with less data. It generally recommends training on approximately 20 tokens per parameter.
Do scaling laws apply to fine-tuning?
Classic scaling laws primarily describe pretraining from scratch. Fine-tuning dynamics differ significantly because the model starts from a converged state rather than random initialization. While similar power-law trends can exist, the exponents and constants are different, and the benefits of scaling are often less predictable due to catastrophic forgetting or task-specific constraints.
Can I use scaling laws to predict emergent abilities?
Not reliably. Standard scaling laws predict smooth improvements in loss (cross-entropy). Emergent abilities, such as complex reasoning or few-shot learning, often appear abruptly at certain model sizes and do not follow the same smooth power-law curves as loss. You need separate evaluations for these capabilities.
How much data do I need for my model?
Under current best practices derived from Chinchilla laws, aim for roughly 20 tokens per parameter for compute-optimal training. However, recent trends show many frontier models are trained on far more tokens (up to 100+ tokens per parameter) to maximize performance, accepting higher compute costs for better generalization.
Why do my scaling law predictions fail?
Predictions often fail due to poor data quality (high redundancy), architectural changes between proxy and target models, or inadequate fitting methodology. If your data contains many duplicates, the effective information content grows slower than the token count, causing sub-scaling. Always ensure your proxy models share the exact architecture and tokenizer of your target model.