Troubling Trends in Machine Learning Scholarship
  Collectively, machine learning (ML) researchers are engaged in the creation
and dissemination of knowledge about data-driven algorithms. In a given paper,
researchers might aspire to any subset of the following goals, among others: to
theoretically characterize what is learnable, to obtain understanding through
empirically rigorous experiments, or to build a working system that has high
predictive accuracy. While determining which knowledge warrants inquiry may be
subjective, once the topic is fixed, papers are most valuable to the community
when they act in service of the reader, creating foundational knowledge and
communicating as clearly as possible.
  Recent progress in machine learning comes despite frequent departures from
these ideals. In this paper, we focus on the following four patterns that
appear to us to be trending in ML scholarship: (i) failure to distinguish
between explanation and speculation; (ii) failure to identify the sources of
empirical gains, e.g., emphasizing unnecessary modifications to neural
architectures when gains actually stem from hyper-parameter tuning; (iii)
mathiness: the use of mathematics that obfuscates or impresses rather than
clarifies, e.g., by confusing technical and non-technical concepts; and (iv)
misuse of language, e.g., by choosing terms of art with colloquial connotations
or by overloading established technical terms.
  While the causes behind these patterns are uncertain, possibilities include
the rapid expansion of the community, the consequent thinness of the reviewer
pool, and the often-misaligned incentives between scholarship and short-term
measures of success (e.g., bibliometrics, attention, and entrepreneurial
opportunity). While each pattern offers a corresponding remedy (don't do it),
we also discuss some speculative suggestions for how the community might combat
these trends.
