Chinchilla AI.pdf

It challenges the current trend of scaling up model size while keeping data constant, arguing that large models are often under-trained.

Key Findings:

- Equal Scaling: Model size and number of training tokens should be scaled equally for optimal training. Doubling the model size should be accompanied by doubling the training tokens.
- Chinchilla: The authors train a predicted compute-optimal model, Chinchilla (70B parameters), on 1.4 trillion tokens, significantly outperforming larger models like Gopher (280B) and others on various benchmarks, including MMLU.
- Advantages of Chinchilla: Chinchilla uses less compute for fine-tuning and inference, making it more accessible for downstream applications. It achieves a state-of-the-art average accuracy of 67.5% on MMLU, surpassing Gopher by over 7%.

Methodology:

The researchers trained over 400 models ranging from 70M to 16B parameters on 5B to 500B tokens. They used a mathematical model to predict the optimal allocation of compute budget (FLOPs) between model size and training data, leading to different conclusions than previous work by Kaplan et al. (2020).