Visualizing the Feature Importance for Black Box Models
  In recent years, a large amount of model-agnostic methods to improve the
transparency, trustability and interpretability of machine learning models have
been developed. We introduce local feature importance as a local version of a
recent model-agnostic global feature importance method. Based on local feature
importance, we propose two visual tools: partial importance (PI) and individual
conditional importance (ICI) plots which visualize how changes in a feature
affect the model performance on average, as well as for individual
observations. Our proposed methods are related to partial dependence (PD) and
individual conditional expectation (ICE) plots, but visualize the expected
(conditional) feature importance instead of the expected (conditional)
prediction. Furthermore, we show that averaging ICI curves across observations
yields a PI curve, and integrating the PI curve with respect to the
distribution of the considered feature results in the global feature
importance. Another contribution of our paper is the Shapley feature
importance, which fairly distributes the overall performance of a model among
the features according to the marginal contributions and which can be used to
compare the feature importance across different models.
