When preparing that paper, I really suffered a lot from the continuously changing software and hardware stack when performing and processing huge amounts of experiments to build and train models which could predict optimizations. That experience eventually motivated me to continue my work on machine learning based optimization as a community effort [1,2] while sharing all my benchmarks, data sets, models, tools and scripts as customizable and reusable components. It also motivated me to develop an open-source framework and repository to crowdsource empirical experiments (such as multi-objective optimization of deep learning and other realistic workloads) across diverse hardware and input provided by volunteers which later became known as the Collective Knowledge (CK):
For example, CK now assists various Artifact Evaluation initiatives at the premier ACM conferences on parallel programming, architecture and code generation (CGO, PPoPP, PACT, SC), which aim to encourage sharing of code and data, and independently validate experimental results from published papers:
We also use CK to crowdsource benchmarking and optimizations of realistic workloads across embedded devices such as mobile phones and tablets, while publicly sharing all optimization statistics for further collaborative analysis and mining:
Hope you will also join our community effort to accelerate computer systems' research and enable cheap and efficient computing from IoT devices to supercomputers!
