Problems with Shapley-value-based explanations as feature importance
  measures
  Game-theoretic formulations of feature importance have become popular as a
way to "explain" machine learning models. These methods define a cooperative
game between the features of a model and distribute influence among these input
elements using some form of the game's unique Shapley values. Justification for
these methods rests on two pillars: their desirable mathematical properties,
and their applicability to specific motivations for explanations. We show that
mathematical problems arise when Shapley values are used for feature importance
and that the solutions to mitigate these necessarily induce further complexity,
such as the need for causal reasoning. We also draw on additional literature to
argue that Shapley values do not provide explanations which suit human-centric
goals of explainability.
