The MLPerf consortium has recently released over 500 validated inference benchmarking results from 14 organizations measuring how fast and how well a pre-trained computer system can classify images, detect objects, and translate sentences. Over 400 of these results were submitted by dividiti, a high-tech company based in Cambridge, UK.
“Our success in MLPerf Inference v0.5 is due to our unique open workflow automation technology called Collective Knowledge (CK)”, explains Dr Anton Lokhmotov, CEO and co-founder of dividiti. “We conducted literally hundreds of benchmarking experiments, followed by thousands of auditing experiments, with many combinations of machine learning models, libraries, frameworks and hardware platforms. Such experiments are notoriously hard to stage in an automated, portable and reproducible fashion, which explains why even well-resourced hardware vendors only submit a handful of results. In collaboration with Arm and the Polytechnical University of Milan, we staged experiments on systems ranging from Raspberry Pi class boards and Android phones to high-end workstations. Benchmarking anything anywhere is what we call omni-benchmarking.”“MLPerf is being contributed to by many organizations, from tiny startups to giant corporations with up to 50 contributors per organization. It is simply astonishing that a small organization with only 3 MLPerf contributors has submitted nearly 3 times more results than all other organizations combined,” stated Dr Vijay Janapa Reddi, Associate Professor, Harvard University, and MLPerf Inference Co-chair. “Based on the success of the first submission round, we fully expect to receive thousands of results next year. Workflow automation will be critical not only for generating large volumes of high-quality results, but also for validating and finding the most optimal solutions in terms of performance, quality and cost.”
“Benchmarking modern day platforms with multiple software branches, libraries, toolchains, datasets, and test and device configurations may deliver a set of inconsistent results,” said Colin Osborne, director of engineering and distinguished engineer, Machine Learning Group, Arm. “Arm uses the Collective Knowledge (CK) framework to transform our multi-dimensional problem space into simplified building blocks and more manageable benchmark results.”
“We have been contributing to the MLPerf initiative since its official announcement in 2018Q2. Our automated, customizable and reproducible Collective Knowledge workflows for image classification and object detection were among the very first inference workloads included in MLPerf Inference. Eventually, we aim to automate all MLPerf workloads, and thus enable easy validation, interactive visualization and fair comparison of all submissions,” said Dr Anton Lokhmotov. “We are already working with several key customers, helping create highly competitive, credible and compliant submissions for the next MLPerf Inference v0.7 round. We believe that workflow automation will go far beyond benchmarking to accelerate time-to-market and slash development costs for innovative products in automotive, robotics, healthcare and smart infrastructure domains.”
About dividiti
dividiti is a UK-based high-tech company built upon decades of unique R&D experience of Dr Anton Lokhmotov (formerly manager of GPU Compute compilers at Arm with a PhD from the University of Cambridge) and Dr Grigori Fursin (formerly head of program optimization group at Intel’s Exascale Lab and senior tenured scientist at INRIA with a PhD from the University of Edinburgh).Our pioneering techniques have enabled rigorous performance analysis and optimization for world-leading companies including Arm, Intel, and General Motors, and powered the world’s first machine-learning based compiler developed in the MILEPOST project with IBM. Our customizable workflow framework, Collective Knowledge (CK), is the only universal solution for continuous multi-objective performance analysis and optimization available under a permissive open-source license. By automating systematic and reproducible experimentation with ever evolving software and hardware, CK gives our partners a distinct competitive advantage, as confirmed by a growing number of users in industry and academia.
