Failing Loudly: An Empirical Study of Methods for Detecting Dataset
  Shift
  We might hope that when faced with unexpected inputs, well-designed software
systems would fire off warnings. Machine learning (ML) systems, however, which
depend strongly on properties of their inputs (e.g. the i.i.d. assumption),
tend to fail silently. This paper explores the problem of building ML systems
that fail loudly, investigating methods for detecting dataset shift,
identifying exemplars that most typify the shift, and quantifying shift
malignancy. We focus on several datasets and various perturbations to both
covariates and label distributions with varying magnitudes and fractions of
data affected. Interestingly, we show that across the dataset shifts that we
explore, a two-sample-testing-based approach, using pre-trained classifiers for
dimensionality reduction, performs best. Moreover, we demonstrate that
domain-discriminating approaches tend to be helpful for characterizing shifts
qualitatively and determining if they are harmful.
