Consistent feature attribution for tree ensembles
  Note that a newer expanded version of this paper is now available at:
arXiv:1802.03888
  It is critical in many applications to understand what features are important
for a model, and why individual predictions were made. For tree ensemble
methods these questions are usually answered by attributing importance values
to input features, either globally or for a single prediction. Here we show
that current feature attribution methods are inconsistent, which means changing
the model to rely more on a given feature can actually decrease the importance
assigned to that feature. To address this problem we develop fast exact
solutions for SHAP (SHapley Additive exPlanation) values, which were recently
shown to be the unique additive feature attribution method based on conditional
expectations that is both consistent and locally accurate. We integrate these
improvements into the latest version of XGBoost, demonstrate the
inconsistencies of current methods, and show how using SHAP values results in
significantly improved supervised clustering performance. Feature importance
values are a key part of understanding widely used models such as gradient
boosting trees and random forests, so improvements to them have broad practical
implications.
