Pythia Suite.pdf

Pythia is a suite of 16 large language models (LLMs) with sizes ranging from 70M to 12B parameters, trained on public data in the same order. The suite provides 154 checkpoints for each model, along with tools to download and reconstruct their exact training dataloaders. Pythia is designed to facilitate research in areas such as memorization, term frequency effects, and reducing gender bias.

Key features of Pythia include:
Models span several orders of magnitude of model scale
All models were trained on the same data in the same order
The data and intermediate checkpoints are publicly available for study

The Pythia suite has been used to study various properties of LLMs, including:
Mitigating gender bias: modifying the frequency of gendered terms in pretraining data can reduce bias measures
Memorization: the location of a sequence in the training dataset does not influence the likelihood of it being memorized, and a Poisson point process is a good model for the occurrence of memorized sequences
Emergence of the impact of pretraining frequencies: the frequency of specific facts within a corpus influences a model's ability to apply that fact in response to a natural language question.

Pythia is available at https://github.com/EleutherAI/pythia, along with trained models, analysis code, training code, and training data.