Are We Really Making Much Progress? A Worrying Analysis of Recent Neural
  Recommendation Approaches
  Deep learning techniques have become the method of choice for researchers
working on algorithmic aspects of recommender systems. With the strongly
increased interest in machine learning in general, it has, as a result, become
difficult to keep track of what represents the state-of-the-art at the moment,
e.g., for top-n recommendation tasks. At the same time, several recent
publications point out problems in today's research practice in applied machine
learning, e.g., in terms of the reproducibility of the results or the choice of
the baselines when proposing new models. In this work, we report the results of
a systematic analysis of algorithmic proposals for top-n recommendation tasks.
Specifically, we considered 18 algorithms that were presented at top-level
research conferences in the last years. Only 7 of them could be reproduced with
reasonable effort. For these methods, it however turned out that 6 of them can
often be outperformed with comparably simple heuristic methods, e.g., based on
nearest-neighbor or graph-based techniques. The remaining one clearly
outperformed the baselines but did not consistently outperform a well-tuned
non-neural linear ranking method. Overall, our work sheds light on a number of
potential problems in today's machine learning scholarship and calls for
improved scientific practices in this area. Source code of our experiments and
full results are available at:
https://github.com/MaurizioFD/RecSys2019_DeepLearning_Evaluation.
