Today, MLCommons®, an open engineering consortium, released new results for MLPerf™ Training v1.1, the organization's machine learning training performance benchmark suite. MLPerf Training measures the time it takes to train machine learning models to a standard quality target in a variety of tasks including image classification, object detection, NLP, recommendation, and reinforcement learning.
MLPerf Training is a full system benchmark, testing machine learning models, software, and hardware. MLPerf creates a reliable and consistent way to track performance over time, and the fair and representative benchmarks create a “level playing field” where competition drives the industry forward, accelerating innovation. Compared to the previous submission round, the best benchmark results improved by up to 2.3X, showing substantial improvement in hardware, software, and system scale.
Similar to past MLPerf Training results, the submissions consist of two divisions: closed and open. Closed submissions use the same reference model to ensure a level playing field across systems, while participants in the open division are permitted to submit a variety of models. Submissions are additionally classified by availability within each division, including systems commercially available, in preview, and RDI.
MLPerf Training v1.1 results further MLCommons’ goal to provide benchmarks and metrics that level the industry playing field through the comparison of ML systems, software, and solutions. The latest benchmark round received submissions from 14 organizations and released over 185 peer-reviewed results for machine learning systems spanning from edge devices to data center servers. Submissions this round included software and hardware innovations from Azure, Baidu, Dell, Fujitsu, GIGABYTE, Google, Graphcore, HabanaLabs, HPE, Inspur, Lenovo, NVIDIA, Samsung, and Supermicro. To view the results, please visit https://mlcommons.org/en/training-normal-11/.
“We’re thrilled to have such broad participation in MLPerf Training,” said Victor Bittorf, Co-Chair of the MLPerf Training Working Group. “Congratulations to all of our participants in this round, especially the first-time submitters. It’s particularly exciting to see the advances in the Open Division.”
“Looking back to the first MLPerf Training round in 2018, it’s remarkable that performance has improved by 30X for some of our benchmarks,” said David Kanter, Executive Director of MLCommons. "That rapid increase in performance will ultimately unleash new machine learning innovations that will benefit society."
Additional information about the Training v1.1 benchmarks is available at https://mlcommons.org/en/training-normal-11/.
About MLCommons
MLCommons is an open engineering consortium with a mission to benefit society by accelerating innovation in machine learning. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding partners - global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.
For additional information on MLCommons and details on becoming a Member or Affiliate of the organization, please visit http://mlcommons.org/ or contact participation@mlcommons.org.
Press Contact:
press@mlcommons.org