Predictive and Causal Implications of using Shapley Value for Model
  Interpretation
  Shapley value is a concept from game theory. Recently, it has been used for
explaining complex models produced by machine learning techniques. Although the
mathematical definition of Shapley value is straight-forward, the implication
of using it as a model interpretation tool is yet to be described. In the
current paper, we analyzed Shapley value in the Bayesian network framework. We
established the relationship between Shapley value and conditional
independence, a key concept in both predictive and causal modeling. Our results
indicate that, eliminating a variable with high Shapley value from a model do
not necessarily impair predictive performance, whereas eliminating a variable
with low Shapley value from a model could impair performance. Therefore, using
Shapley value for feature selection do not result in the most parsimonious and
predictively optimal model in the general case. More importantly, Shapley value
of a variable do not reflect their causal relationship with the target of
interest.
