OpenML Benchmarking Suites
  Machine learning research depends on objectively interpretable, comparable,
and reproducible algorithm benchmarks. We advocate the use of curated,
comprehensive suites of machine learning tasks to standardize the setup,
execution, and reporting of benchmarks. We enable this through software tools
that help to create and leverage these benchmarking suites. These are
seamlessly integrated into the OpenML platform, and accessible through
interfaces in Python, Java, and R. OpenML benchmarking suites (a) are easy to
use through standardized data formats, APIs, and client libraries; (b) come
with extensive meta-information on the included datasets; and (c) allow
benchmarks to be shared and reused in future studies. We then present a first,
carefully curated and practical benchmarking suite for classification: the
OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18). Finally,
we discuss use cases and applications which demonstrate the usefulness of
OpenML benchmarking suites and the OpenML-CC18 in particular.
