Model-agnostic Feature Importance and Effects with Dependent Features --
  A Conditional Subgroup Approach
  The interpretation of feature importance in machine learning models is
challenging when features are dependent. Permutation feature importance (PFI)
ignores such dependencies, which can cause misleading interpretations due to
extrapolation. A possible remedy is more advanced conditional PFI approaches
that enable the assessment of feature importance conditional on all other
features. Due to this shift in perspective and in order to enable correct
interpretations, it is therefore important that the conditioning is transparent
and humanly comprehensible. In this paper, we propose a new sampling mechanism
for the conditional distribution based on permutations in conditional
subgroups. As these subgroups are constructed using decision trees
(transformation trees), the conditioning becomes inherently interpretable. This
not only provides a simple and effective estimator of conditional PFI, but also
local PFI estimates within the subgroups. In addition, we apply the conditional
subgroups approach to partial dependence plots (PDP), a popular method for
describing feature effects that can also suffer from extrapolation when
features are dependent and interactions are present in the model. We show that
PFI and PDP based on conditional subgroups often outperform methods such as
conditional PFI based on knockoffs, or accumulated local effect plots.
Furthermore, our approach allows for a more fine-grained interpretation of
feature effects and importance within the conditional subgroups.
