Unrestricted Permutation forces Extrapolation: Variable Importance
  Requires at least One More Model, or There Is No Free Variable Importance
  This paper reviews and advocates against the use of permute-and-predict (PaP)
methods for interpreting black box functions. Methods such as the variable
importance measures proposed for random forests, partial dependence plots, and
individual conditional expectation plots remain popular because they are both
model-agnostic and depend only on the pre-trained model output, making them
computationally efficient and widely available in software. However, numerous
studies have found that these tools can produce diagnostics that are highly
misleading, particularly when there is strong dependence among features. The
purpose of our work here is to (i) review this growing body of literature, (ii)
provide further demonstrations of these drawbacks along with a detailed
explanation as to why they occur, and (iii) advocate for alternative measures
that involve additional modeling. In particular, we describe how breaking
dependencies between features in hold-out data places undue emphasis on sparse
regions of the feature space by forcing the original model to extrapolate to
regions where there is little to no data. We explore these effects across
various model setups and find support for previous claims in the literature
that PaP metrics can vastly over-emphasize correlated features in both variable
importance measures and partial dependence plots. As an alternative, we discuss
and recommend more direct approaches that involve measuring the change in model
performance after muting the effects of the features under investigation.
