Variable importance scores
  Scoring of variables for importance in predicting a response is an
ill-defined concept. Several methods have been proposed but little is known of
their performance. This paper fills the gap with a comparative evaluation of
eleven methods and an updated one based on the GUIDE algorithm. For data
without missing values, eight of the methods are shown to be biased in that
they give higher or lower scores to different types of variables, even when all
are independent of the response. Of the remaining four methods, only two are
applicable to data with missing values, with GUIDE the only unbiased one. GUIDE
achieves unbiasedness by using a self-calibrating step that is applicable to
other methods for score de-biasing. GUIDE also yields a threshold for
distinguishing important from unimportant variables at 95 and 99 percent
confidence levels; the technique is applicable to other methods as well.
Finally, the paper studies the relationship of the scores to predictive power
in three data sets. It is found that the scores of many methods are more
consistent with marginal predictive power than conditional predictive power.
