The Benchmark Lottery
  The world of empirical machine learning (ML) strongly relies on benchmarks in
order to determine the relative effectiveness of different algorithms and
methods. This paper proposes the notion of "a benchmark lottery" that describes
the overall fragility of the ML benchmarking process. The benchmark lottery
postulates that many factors, other than fundamental algorithmic superiority,
may lead to a method being perceived as superior. On multiple benchmark setups
that are prevalent in the ML community, we show that the relative performance
of algorithms may be altered significantly simply by choosing different
benchmark tasks, highlighting the fragility of the current paradigms and
potential fallacious interpretation derived from benchmarking ML methods. Given
that every benchmark makes a statement about what it perceives to be important,
we argue that this might lead to biased progress in the community. We discuss
the implications of the observed phenomena and provide recommendations on
mitigating them using multiple machine learning domains and communities as use
cases, including natural language processing, computer vision, information
retrieval, recommender systems, and reinforcement learning.
