Sources of Irreproducibility in Machine Learning: A Review
  Lately, several benchmark studies have shown that the state of the art in
some of the sub-fields of machine learning actually has not progressed despite
progress being reported in the literature. The lack of progress is partly
caused by the irreproducibility of many model comparison studies. Model
comparison studies are conducted that do not control for many known sources of
irreproducibility. This leads to results that cannot be verified by third
parties. Our objective is to provide an overview of the sources of
irreproducibility that are reported in the literature. We review the literature
to provide an overview and a taxonomy in addition to a discussion on the
identified sources of irreproducibility. Finally, we identify three lines of
further inquiry.
