Smoke Testing for Machine Learning: Simple Tests to Discover Severe
  Defects
  Machine learning is nowadays a standard technique for data analysis within
software applications. Software engineers need quality assurance techniques
that are suitable for these new kinds of systems. Within this article, we
discuss the question whether standard software testing techniques that have
been part of textbooks since decades are also useful for the testing of machine
learning software. Concretely, we try to determine generic and simple smoke
tests that can be used to assert that basic functions can be executed without
crashing. We found that we can derive such tests using techniques similar to
equivalence classes and boundary value analysis. Moreover, we found that these
concepts can also be applied to hyperparameters, to further improve the quality
of the smoke tests. Even though our approach is almost trivial, we were able to
find bugs in all three machine learning libraries that we tested and severe
bugs in two of the three libraries. This demonstrates that common software
testing techniques are still valid in the age of machine learning and that
considerations how they can be adapted to this new context can help to find and
prevent severe bugs, even in mature machine learning libraries.
