Beyond Average Performance -- exploring regions of deviating performance
  for black box classification models
  Machine learning models are becoming increasingly popular in different types
of settings. This is mainly caused by their ability to achieve a level of
predictive performance that is hard to match by human experts in this new era
of big data. With this usage growth comes an increase of the requirements for
accountability and understanding of the models' predictions. However, the
degree of sophistication of the most successful models (e.g. ensembles, deep
learning) is becoming a large obstacle to this endeavour as these models are
essentially black boxes. In this paper we describe two general approaches that
can be used to provide interpretable descriptions of the expected performance
of any black box classification model. These approaches are of high practical
relevance as they provide means to uncover and describe in an interpretable way
situations where the models are expected to have a performance that deviates
significantly from their average behaviour. This may be of critical relevance
for applications where costly decisions are driven by the predictions of the
models, as it can be used to warn end users against the usage of the models in
some specific cases.
