MLCommons™ AlgoPerf: Training Algorithms Leaderboard

This repository hosts the official rolling leaderboard for the AlgoPerf: Training Algorithms benchmark by MLCommons. The benchmark measures neural network training speedups due to algorithmic improvements in training algorithms. The leaderboard tracks the aggregate performance of different algorithms on a variety of workloads and under two different tuning rulesets.

Note

If you want to submit to the AlgoPerf benchmark, please open a PR with your submission. The AlgoPerf working group will review your submission and potentially evaluate your submission on all workloads. For more details, see the How to Submit section.

Live Leaderboards

Leaderboard Version: 0.6 Last Updated: 2025-03-24 15:07 UTC Using Benchmark Version: latest

Tip

The leaderboard of the first AlgoPerf competition with more entries can be found here.

External Tuning Ruleset Leaderboard

In the external tuning ruleset, submission must provide workload-agnostic hyperparameter search spaces and they will get $5$ tuning trials per workload sampled from this search space.

Rank	Submission	Authors	Affiliation	Framework	Logs	Score
1.	Distributed Shampoo Based on the Distributed Shampoo algorithm of Anil et al. (2020) with an implementation tailored to leverage PyTorch performance optimizations. See Shi et al. (2023) for details. The submission uses a list of five hyperparameter settings.	Hao-Jun Shi, Tsung-Hsien Lee, Anna Cai, Shintaro Iwasaki, Wenyin Fu, Yuchen Hao, Mike Rabbat	Meta Platforms	PyTorch	💾	0.6244
2.	Baseline Baseline using NadamW (Dozat, 2016; Loshchilov & Hutter, 2019) and a linear learning rate warmup followed by a cosine decay (Dahl et al., 2023).			JAX	💾	0.4590

Self-Tuning Ruleset Leaderboard

In the self-tuning ruleset, submissions must be completely hyperparameter-free.

Note

The first self-tuning submissions are currently being scored.

How to Submit

To submit your algorithm for evaluation on the AlgoPerf leaderboard, please follow these steps:

1. Implement your algorithm in the AlgoPerf API: Have a look at our Getting Started Guide and the Technical Documentation.
2. Create a Pull Request: Fork this repository, create a new branch and add your submission code to a new folder within either submissions/external_tuning/ or submissions/self_tuning. Open a pull request (PR) to the evaluation branch of this repository. Make sure to fill out the PR template asking for information such as submission name, authors, affiliations, etc.
3. PR Review and Evaluation: The AlgoPerf working group will review your PR. Based on our available resources and the perceived potential of the method, it will be selected for a free evaluation and merged into the evaluation branch. The working group will run your submission on all workloads and push the results, as well as the updated leaderboard, to the mainbranch.

Scoring

The code that computes this leaderboard lives in scoring/. Given a directory of submission logs (such as those under previous_leaderboards/), it computes the performance profiles, time-to-target, AlgoPerf benchmark scores, and speedups used in the tables above. This code was moved here from the scoring/ directory of the algorithmic-efficiency repository so that the repository that hosts the leaderboard also owns the code that produces it.

Installation

The scoring code is self-contained. To run it, set up a fresh Python (>=3.11) environment, e.g. via conda or virtualenv:

python3 -m venv env && source env/bin/activate
pip3 install -e .          # installs the scoring tooling (numpy, pandas, scipy, ...)

Note

The scoring/workload_targets*.json files are generated from the benchmark definitions in algorithmic-efficiency (scoring/generate_workload_targets.py) and commited here. Each file is frozen for one benchmark version, carrying that version's base/held-out workload sets and per-workload targets. Regenerate and re-copy when a benchmark version changes the workloads or targets.

Regenerating the leaderboard

The current targets are the default. To score an older leaderboard, pass --workload_targets for that version's file (e.g. scoring/workload_targets_v05.json).

# External tuning ruleset
python -m scoring.score_submissions \
  --submission_directory previous_leaderboards/algoperf_v06/logs/external_tuning \
  --compute_performance_profiles \
  --output_dir scoring_results_external_tuning

# Self-tuning ruleset (add --self_tuning_ruleset)
python -m scoring.score_submissions \
  --submission_directory previous_leaderboards/algoperf_v06/logs/self_tuning \
  --compute_performance_profiles \
  --self_tuning_ruleset \
  --output_dir scoring_results_self_tuning

# Reproduce the v0.5 leaderboard (8 base + 6 held-out workloads)
python -m scoring.score_submissions \
  --workload_targets scoring/workload_targets_v05.json \
  --submission_directory previous_leaderboards/algoperf_v05/logs/external_tuning \
  --compute_performance_profiles \
  --output_dir scoring_results_v05_external

See the scoring methodology in the benchmark documentation for details on how scores are computed.

Citation

If you use the AlgoPerf benchmark in your research, please consider citing our paper.

Dahl, Schneider, Nado, et al.
> Benchmarking Neural Network Training Algorithms
> arXiv 2306.07179

@Misc{Dahl2023AlgoPerf,
  title         = {{Benchmarking Neural Network Training Algorithms}},
  author        = {Dahl, George E. and Schneider, Frank and Nado, Zachary and Agarwal, Naman and Sastry, Chandramouli Shama and Hennig, Philipp and Medapati, Sourabh and Eschenhagen, Runa and Kasimbeg, Priya and Suo, Daniel and Bae, Juhan and Gilmer, Justin and Peirson, Abel L. and Khan, Bilal and Anil, Rohan and Rabbat, Mike and Krishnan, Shankar and Snider, Daniel and Amid, Ehsan and Chen, Kongtao and Maddison, Chris J. and Vasudev, Rakshith and Badura, Michal and Garg, Ankush and Mattson, Peter},
  year          = {2023},
  archiveprefix = {arXiv},
  eprint        = {2306.07179},
}

If you use the results from the first AlgoPerf competition, please consider citing the results paper, as well as the relevant submissions:

Kasimbeg, Schneider, Eschenhagen, et al.
> Accelerating neural network training: An analysis of the AlgoPerf competition
ICLR 2025

@inproceedings{Kasimbeg2025AlgoPerfResults,
title           = {Accelerating neural network training: An analysis of the {AlgoPerf} competition},
author          = {Kasimbeg, Priya and Schneider, Frank and Eschenhagen, Runa and Bae, Juhan and Sastry, Chandramouli Shama and Saroufim, Mark and Boyuan, Feng and Wright, Less and Yang, Edward Z. and Nado, Zachary and Medapati, Sourabh and Hennig, Philipp and Rabbat, Michael and Dahl, George E.},
booktitle       = {The Thirteenth International Conference on Learning Representations},
year            = {2025},
url             = {https://openreview.net/forum?id=CtM5xjRSfm}
}