“MLPerf is being contributed to by many organizations, from tiny startups to giant corporations with up to 50 contributors per organization. It is simply astonishing that a small organization with only 3 MLPerf contributors has submitted nearly 3 times more results than all other organizations combined,” stated Dr Vijay Janapa Reddi, Associate Professor, Harvard University, and MLPerf Inference Co-chair. “Based on the success of the first submission round, we fully expect to receive thousands of results next year. Workflow automation will be critical not only for generating large volumes of high-quality results, but also for validating and finding the most optimal solutions in terms of performance, quality and cost.”
“Benchmarking modern day platforms with multiple software branches, libraries, toolchains, datasets, and test and device configurations may deliver a set of inconsistent results,” said Colin Osborne, director of engineering and distinguished engineer, Machine Learning Group, Arm. “Arm uses the Collective Knowledge (CK) framework to transform our multi-dimensional problem space into simplified building blocks and more manageable benchmark results.”
“We have been contributing to the MLPerf initiative since its official announcement in 2018Q2. Our automated, customizable and reproducible Collective Knowledge workflows for image classification and object detection were among the very first inference workloads included in MLPerf Inference. Eventually, we aim to automate all MLPerf workloads, and thus enable easy validation, interactive visualization and fair comparison of all submissions,” said Dr Anton Lokhmotov. “We are already working with several key customers, helping create highly competitive, credible and compliant submissions for the next MLPerf Inference v0.7 round. We believe that workflow automation will go far beyond benchmarking to accelerate time-to-market and slash development costs for innovative products in automotive, robotics, healthcare and smart infrastructure domains.”
About dividitidividiti is a UK-based high-tech company built upon decades of unique R&D experience of Dr Anton Lokhmotov (formerly manager of GPU Compute compilers at Arm with a PhD from the University of Cambridge) and Dr Grigori Fursin (formerly head of program optimization group at Intel’s Exascale Lab and senior tenured scientist at INRIA with a PhD from the University of Edinburgh).
Our pioneering techniques have enabled rigorous performance analysis and optimization for world-leading companies including Arm, Intel, and General Motors, and powered the world’s first machine-learning based compiler developed in the MILEPOST project with IBM. Our customizable workflow framework, Collective Knowledge (CK), is the only universal solution for continuous multi-objective performance analysis and optimization available under a permissive open-source license. By automating systematic and reproducible experimentation with ever evolving software and hardware, CK gives our partners a distinct competitive advantage, as confirmed by a growing number of users in industry and academia.