MDA for random forests: inconsistency, and a practical solution via the
  Sobol-MDA
  Variable importance measures are the main tools to analyze the black-box
mechanisms of random forests. Although the mean decrease accuracy (MDA) is
widely accepted as the most efficient variable importance measure for random
forests, little is known about its statistical properties. In fact, the
definition of MDA varies across the main random forest software. In this
article, our objective is to rigorously analyze the behavior of the main MDA
implementations. Consequently, we mathematically formalize the various
implemented MDA algorithms, and then establish their limits when the sample
size increases. This asymptotic analysis reveals that these MDA versions differ
as importance measures, since they converge towards different quantities. More
importantly, we break down these limits into three components: the first two
terms are related to Sobol indices, which are well-defined measures of a
covariate contribution to the response variance, widely used in the sensitivity
analysis field, as opposed to the third term, whose value increases with
dependence within covariates. Thus, we theoretically demonstrate that the MDA
does not target the right quantity to detect influential covariates in a
dependent setting, a fact that has already been noticed experimentally. To
address this issue, we define a new importance measure for random forests, the
Sobol-MDA, which fixes the flaws of the original MDA, and consistently
estimates the accuracy decrease of the forest retrained without a given
covariate, but with an efficient computational cost. The Sobol-MDA empirically
outperforms its competitors on both simulated and real data for variable
selection. An open source implementation in R and C++ is available online.
