A Troubling Analysis of Reproducibility and Progress in Recommender
  Systems Research
  The design of algorithms that generate personalized ranked item lists is a
central topic of research in the field of recommender systems. In the past few
years, in particular, approaches based on deep learning (neural) techniques
have become dominant in the literature. For all of them, substantial progress
over the state-of-the-art is claimed. However, indications exist of certain
problems in today's research practice, e.g., with respect to the choice and
optimization of the baselines used for comparison, raising questions about the
published claims. In order to obtain a better understanding of the actual
progress, we have tried to reproduce recent results in the area of neural
recommendation approaches based on collaborative filtering. The worrying
outcome of the analysis of these recent works-all were published at prestigious
scientific conferences between 2015 and 2018-is that 11 out of the 12
reproducible neural approaches can be outperformed by conceptually simple
methods, e.g., based on the nearest-neighbor heuristics. None of the
computationally complex neural methods was actually consistently better than
already existing learning-based techniques, e.g., using matrix factorization or
linear models. In our analysis, we discuss common issues in today's research
practice, which, despite the many papers that are published on the topic, have
apparently led the field to a certain level of stagnation.
