Do not explain without context: addressing the blind spot of model
  explanations
  The increasing number of regulations and expectations of predictive machine
learning models, such as so called right to explanation, has led to a large
number of methods promising greater interpretability. High demand has led to a
widespread adoption of XAI techniques like Shapley values, Partial Dependence
profiles or permutational variable importance. However, we still do not know
enough about their properties and how they manifest in the context in which
explanations are created by analysts, reviewed by auditors, and interpreted by
various stakeholders. This paper highlights a blind spot which, although
critical, is often overlooked when monitoring and auditing machine learning
models: the effect of the reference data on the explanation calculation. We
discuss that many model explanations depend directly or indirectly on the
choice of the referenced data distribution. We showcase examples where small
changes in the distribution lead to drastic changes in the explanations, such
as a change in trend or, alarmingly, a conclusion. Consequently, we postulate
that obtaining robust and useful explanations always requires supporting them
with a broader context.
