Sequential Permutation Testing of Random Forest Variable Importance Measures
Hypothesis testing of random forest (RF) variable importance measures (VIMP)
remains the subject of ongoing research. Among recent developments, heuristic
approaches to parametric testing have been proposed whose distributional
assumptions are based on empirical evidence. Other formal tests under
regularity conditions were derived analytically. However, these approaches can
be computationally expensive or even practically infeasible. This problem also
occurs with non-parametric permutation tests, which are, however,
distribution-free and can generically be applied to any type of RF and VIMP.
Embracing this advantage, it is proposed here to use sequential permutation
tests and sequential p-value estimation to reduce the high computational costs
associated with conventional permutation tests. The popular and widely used
permutation VIMP serves as a practical and relevant application example. The
results of simulation studies confirm that the theoretical properties of the
sequential tests apply, that is, the type-I error probability is controlled at
a nominal level and a high power is maintained with considerably fewer
permutations needed in comparison to conventional permutation testing. The
numerical stability of the methods is investigated in two additional
application studies. In summary, theoretically sound sequential permutation
testing of VIMP is possible at greatly reduced computational costs.
Recommendations for application are given. A respective implementation is
provided through the accompanying R package $rfvimptest$. The approach can also
be easily applied to any kind of prediction model.