Bringing a Ruler Into the Black Box: Uncovering Feature Impact from
  Individual Conditional Expectation Plots
  As machine learning systems become more ubiquitous, methods for understanding
and interpreting these models become increasingly important. In particular,
practitioners are often interested both in what features the model relies on
and how the model relies on them--the feature's impact on model predictions.
Prior work on feature impact including partial dependence plots (PDPs) and
Individual Conditional Expectation (ICE) plots has focused on a visual
interpretation of feature impact. We propose a natural extension to ICE plots
with ICE feature impact, a model-agnostic, performance-agnostic feature impact
metric drawn out from ICE plots that can be interpreted as a close analogy to
linear regression coefficients. Additionally, we introduce an in-distribution
variant of ICE feature impact to vary the influence of out-of-distribution
points as well as heterogeneity and non-linearity measures to characterize
feature impact. Lastly, we demonstrate ICE feature impact's utility in several
tasks using real-world data.
