Feature relevance quantification in explainable AI: A causal problem
  We discuss promising recent contributions on quantifying feature relevance
using Shapley values, where we observed some confusion on which probability
distribution is the right one for dropped features. We argue that the confusion
is based on not carefully distinguishing between observational and
interventional conditional probabilities and try a clarification based on
Pearl's seminal work on causality. We conclude that unconditional rather than
conditional expectations provide the right notion of dropping features in
contradiction to the theoretical justification of the software package SHAP.
Parts of SHAP are unaffected because unconditional expectations (which we argue
to be conceptually right) are used as approximation for the conditional ones,
which encouraged others to `improve' SHAP in a way that we believe to be
flawed.
