Preparing a new MLCommons education and reproducibility workgroup to make it easier to run, customize and reproduce MLPerf benchmarks

The 1st presentation to help prepare a new MLCommons workgroup to make it easier to run, customize and reproduce MLPerf benchmarks. The mission: Develop an automated open-source workflow to make it easier to plug any real-world ML & AI tasks, models, data sets, software and hardware into the MLPerf benchmarking infrastructure. Use this workflow to help the newcomers learn how to customize and run MLPerf benchmarks across rapidly evolving software, hardware and data. Lower the barrier of entry for new MLPerf submitters and reduce their associated costs. Automate design space exploration of diverse ML/SW/HW stacks to trade off performance, accuracy, energy, size and costs; automate submission of Pareto-efficient configurations to MLPerf. Help end-users visualize all MLPerf results, reproduce them and deploy the most suitable ML/SW/HW stacks in production. Support reproducibility initiatives at ML and Systems conferences using rigorous MLPerf methodology and our educational toolkit