Introducing ReQuEST: an Open Platform for Reproducible and
  Quality-Efficient Systems-ML Tournaments
  Co-designing efficient machine learning based systems across the whole
hardware/software stack to trade off speed, accuracy, energy and costs is
becoming extremely complex and time consuming. Researchers often struggle to
evaluate and compare different published works across rapidly evolving software
frameworks, heterogeneous hardware platforms, compilers, libraries, algorithms,
data sets, models, and environments.
  We present our community effort to develop an open co-design tournament
platform with an online public scoreboard. It will gradually incorporate best
research practices while providing a common way for multidisciplinary
researchers to optimize and compare the quality vs. efficiency Pareto
optimality of various workloads on diverse and complete hardware/software
systems. We want to leverage the open-source Collective Knowledge framework and
the ACM artifact evaluation methodology to validate and share the complete
machine learning system implementations in a standardized, portable, and
reproducible fashion. We plan to hold regular multi-objective optimization and
co-design tournaments for emerging workloads such as deep learning, starting
with ASPLOS'18 (ACM conference on Architectural Support for Programming
Languages and Operating Systems - the premier forum for multidisciplinary
systems research spanning computer architecture and hardware, programming
languages and compilers, operating systems and networking) to build a public
repository of the most efficient machine learning algorithms and systems which
can be easily customized, reused and built upon.
