Training Compute-Optimal Large Language Models — and why throwing more parameters at the problem was the wrong strategy all along.
In 2020 and 2021, the AI world had a simple mantra: bigger is better. GPT-3 had 175 billion parameters. Google's PaLM would push to 540 billion. Labs were racing to train the largest model they could afford, often spending tens of millions of dollars on a single training run.
There was just one problem. They were all doing it wrong.
In March 2022, a team at DeepMind published a paper that sent shockwaves through the AI industry. Their finding was as simple as it was devastating: virtually every large language model in existence had been significantly undertrained. These models had too many parameters and were trained on far too little data.
The proof? A model called Chinchilla — with just 70 billion parameters, less than half the size of DeepMind's own Gopher (280B) — outperformed Gopher on nearly every benchmark. It used the exact same compute budget. The only difference was how that compute was allocated: fewer parameters, more data.
This wasn't a marginal improvement. It was a paradigm shift. Let's start with a question to see what your intuition says.
You have a fixed compute budget of 5.76 × 10²³ FLOPs. How would you allocate it? Move the slider to choose your split between model size and training data.
To understand why the Chinchilla paper mattered, we need to rewind. The year is 2020. OpenAI has just released GPT-3, and the world is stunned. The model can write poetry, code, and even do basic math — all from a single prompt. The message was clear: scale up the parameters, and intelligence follows.
This belief wasn't unfounded. In January 2020, Kaplan et al. at OpenAI published "Scaling Laws for Neural Language Models" — a landmark paper that showed smooth, predictable improvements in model performance as you increased three things: parameters (N), data (D), and compute (C).
But Kaplan's paper had a particular conclusion that shaped the entire industry's strategy: it suggested that model performance was more sensitive to the number of parameters than to the amount of training data. The implication? If you had a bigger compute budget, you should mostly spend it on a bigger model.
And so began the great parameter arms race.
Click on each milestone to see how the race unfolded. Notice how parameters grew much faster than training data.
The prevailing wisdom was essentially: "We have X FLOPs. Let's make the model as big as possible, train it for one epoch on whatever data we have, and call it done." The Chinchilla paper showed this was leaving enormous performance on the table.
Here is the central claim of the Chinchilla paper, stated as plainly as possible:
Let's unpack what this means. If you double your compute budget, you shouldn't just double the model size. You should increase both the model size and the training data by roughly the same factor — about 1.4× each (since 1.4 × 1.4 ≈ 2).
The paper proposes that the optimal number of training tokens should be approximately 20× the number of parameters. A 10B parameter model should see ~200B tokens. A 70B model should see ~1.4T tokens. A 500B model would need ~10T tokens.
This was a radical departure from Kaplan's earlier findings, which suggested parameters should grow faster than data. Let's visualize what this means in practice.
Use the slider to increase compute budget and watch how the optimal allocation of parameters and data should grow together.
Think of it like cooking. Kaplan's earlier work was like saying: "If you want a better meal, just get a bigger oven." The Chinchilla paper says: "Actually, a slightly smaller oven with more ingredients makes a much better dish." The oven is your model; the ingredients are your data.
One of the most compelling aspects of the Chinchilla paper is that the authors didn't rely on a single methodology. They used three completely independent approaches to derive the optimal scaling law — and all three converged on the same answer. That's the kind of evidence that makes a paper hard to argue with.
Step through each approach to see how they derived the optimal scaling.
For each of several fixed compute budgets (from 10¹⁸ to 10²¹ FLOPs), they trained over 400 models ranging from 70M to 16B parameters.
For each budget level, they varied how much was allocated to model size vs. training data. Then they found which allocation produced the lowest loss — the "valley" of the loss curve.
Each compute level produces a U-shaped curve. Too few parameters wastes compute on unnecessary passes; too many leaves each parameter undertrained.
The convergence of all three approaches is what gave the paper its weight. Any one method might have had flaws or biases, but when three fundamentally different analytical strategies agree, you can be pretty confident in the result.
Theory is nice. But nothing convinces like a head-to-head battle. The Chinchilla paper didn't just propose a new scaling law — it built a model to prove it.
Gopher was DeepMind's state-of-the-art LLM: 280 billion parameters, trained on 300 billion tokens, using approximately 5.76 × 10²³ FLOPs of compute.
Chinchilla used the exact same compute budget. But instead of 280B parameters, it had just 70B — a quarter the size. The saved compute was redirected into training on 4.7× more data: 1.4 trillion tokens.
The result? Chinchilla uniformly outperformed Gopher. Not on one benchmark. On virtually all of them.
Click each benchmark to see how Chinchilla (70B) compared to Gopher (280B). Green = Chinchilla wins.
This wasn't just a theoretical exercise. It had immediate practical implications: a model 4× smaller is 4× cheaper to serve. Inference costs — the cost of actually running the model for users — scale roughly linearly with parameter count. Chinchilla delivered better performance and cost less to deploy.
Let's get into the equations — but gently. The core math is surprisingly elegant.
The total compute C for training a transformer is approximately:
Where N is the number of parameters and D is the number of training tokens. The factor of 6 comes from the forward and backward passes through the network (roughly 2 FLOPs per parameter per token for the forward pass, and 4 for backward).
The Chinchilla paper's key insight is that the loss function has a specific shape. If you fix compute C, there's a unique optimal pair (N*, D*) that minimizes loss. And that pair follows:
Both scale with the same exponent (0.50), confirming that parameters and data should grow at the same rate. The ratio D/N works out to roughly 20, meaning about 20 tokens per parameter.
Enter your compute budget (in FLOPs) and see the Chinchilla-optimal allocation. Or enter a model size to see how much data and compute you need.
Here's a fun exercise: plug in GPT-3's 175B parameters. According to the Chinchilla scaling law, it should have been trained on roughly 3.5 trillion tokens. It was actually trained on 300 billion — about 12× too few. That's a lot of wasted potential.
The Chinchilla paper didn't just present new results — it directly contradicted the most influential scaling paper in AI. The Kaplan et al. (2020) paper from OpenAI had argued for very different optimal scaling ratios.
Kaplan found that parameters should scale ~3× faster than data. Double your compute? Make the model about 5× bigger but only train on 1.5× more tokens. This led to the "bigger model, same data" approach that dominated the field.
Hoffmann et al. found the opposite: parameters and data should scale equally. So what went wrong with Kaplan's analysis?
Use the slider to increase compute budget and see how the two scaling laws prescribe very different allocations.
The key methodological differences that led Kaplan astray:
1. Learning rate schedules. Kaplan's experiments used a fixed learning rate schedule that wasn't adjusted to match the number of training tokens. When training for longer, you need to decay the learning rate more gradually. Without this, longer training runs look artificially worse — biasing the results toward bigger models with shorter training.
2. Not training to convergence. Many of Kaplan's smaller models may not have been trained long enough, making small models appear less capable than they actually are.
3. Narrow parameter range. Kaplan's experiments covered a narrower range of model sizes, making the extrapolation to very large scales less reliable.
These sound like minor technical details, but they compounded into a conclusion that misdirected billions of dollars of compute investment. Methodological rigor matters — especially when entire industries follow your prescriptions.
After the paper dropped, a delightful term entered the AI lexicon: the "Chinchilla tax." It refers to the extra inference cost you pay forever because you trained a model that's bigger than it needed to be.
Here's the logic: Suppose you trained a 280B parameter model (like Gopher) when a compute-optimal 70B model (like Chinchilla) would have been just as good or better. Every single API call, every chat response, every inference step now costs you 4× more than it should. Multiply that by millions of users and months of deployment, and you're talking about hundreds of millions of dollars in unnecessary inference costs.
The training run is a one-time cost. Inference runs forever.
Estimate how much extra you'd spend serving an oversized model vs. the compute-optimal one.
The "Chinchilla tax" is why this paper had such outsized impact on the industry. It wasn't just about training better models — it was about the economics of deployment. In a world where LLMs are served to millions of users, inference costs dominate. A smaller model with the same (or better!) capabilities is worth enormously more than a bigger one.
Some wags noted the irony: the entire industry had been voluntarily paying a massive Chinchilla tax for years — because they were following scaling laws that told them bigger was always better.
The Chinchilla paper didn't just start an academic debate. It fundamentally changed how every major AI lab allocates compute. The evidence is visible in every major model released after March 2022.
Hover over each model to see how the Chinchilla paper influenced its design. Notice how the tokens-per-parameter ratio increased dramatically.
The pattern is unmistakable. Before Chinchilla, labs scaled parameters 100× while barely increasing data 10×. After Chinchilla, the data budgets exploded: 1T, 2T, 15T tokens. Some teams deliberately "over-trained" their models on more data than Chinchilla-optimal, accepting slightly worse loss to get a smaller, cheaper model for deployment.
This strategy — training beyond the compute-optimal point for inference efficiency — has become known as inference-optimal training, and it's a direct intellectual descendant of the Chinchilla paper. You can't think about inference optimization until you first accept the framework that says "bigger isn't always better."
The Chinchilla paper is brilliant, but it's not without problems. The most immediate practical challenge is what some call the "data wall."
If a 1 trillion parameter model needs ~20 trillion tokens of high-quality training data, where exactly does all that data come from? The entire internet (as of 2022) was estimated to contain roughly 5–10 trillion tokens of high-quality text. That's a hard ceiling.
Several strategies have emerged to address this:
Select a model size to see how much data Chinchilla prescribes vs. how much quality data exists. Click each solution to learn more.
Other limitations worth noting:
The 20× ratio may shift with architecture changes. The Chinchilla experiments used standard dense Transformers. Mixture-of-Experts models, which activate only a fraction of parameters per token, have different compute profiles. The exact ratio might be different for different architectures.
Quality ≠ Quantity. The paper treats all tokens as equal. But a token from a carefully curated textbook isn't the same as a token from a random Reddit thread. Data quality modifies the effective scaling law in ways that are still being researched.
Downstream performance vs. loss. Chinchilla optimizes pre-training loss (perplexity). But what matters in practice is downstream task performance after fine-tuning. The mapping from loss to usefulness isn't always straightforward.
The Chinchilla paper is one of those rare works that instantly divides AI history into "before" and "after." Its influence extends far beyond the specific numbers it proposed.
It made data a first-class citizen. Before Chinchilla, the AI discourse was all about model size. After Chinchilla, labs started investing heavily in data pipelines, data quality, and data curation. Common Crawl refinement became as important as architecture innovation.
It enabled the open-source revolution. If bigger was always better, only the richest labs could compete. But Chinchilla showed that a well-trained 7B or 13B model could punch far above its weight. This opened the door for Meta's LLaMA, Mistral, and the entire open-source ecosystem.
It shifted the cost conversation from training to inference. Once you accept that a smaller model can match a larger one, the economics of deployment become paramount. This led to quantization, distillation, and inference-optimization becoming major research areas.
Five questions to check if the Chinchilla insight has really sunk in.
Perhaps the deepest legacy of the Chinchilla paper is philosophical. It taught the AI community that intuition about scaling can be dangerously wrong. The field's most prominent researchers, at the best-funded labs, had been following a suboptimal strategy for years — because a single paper's methodology was slightly off, and nobody questioned it seriously until Hoffmann et al. did the rigorous experiments.
If the smartest people in AI can be wrong about something this fundamental, what else might the field be getting wrong?
Let's compress everything into a quick reference.
Click on any row for a quick refresher on each key concept.
| Concept | Pre-Chinchilla | Post-Chinchilla |
|---|---|---|
| Scaling priority | Parameters first | Equal: params & data |
| Tokens per parameter | ~1-5× | ~20× (or more) |
| Key bottleneck | Compute & model size | Data quality & quantity |
| Inference cost | Ignored during training | Central design concern |
| Compute formula | C ≈ 6ND | C ≈ 6ND (same, better used) |
| Optimal scaling exponent | N ∝ C⁰·⁷³ (Kaplan) | N ∝ C⁰·⁵⁰ (Hoffmann) |
The Chinchilla paper is ultimately a story about humility in the face of data. The most expensive, prestigious models in the world were undertrained. The fix wasn't a better architecture or a clever trick — it was simply reading more books. There's a lesson in that for all of us.