Hierarchical Shrinkage: improving the accuracy and interpretability of
  tree-based methods
  Tree-based models such as decision trees and random forests (RF) are a
cornerstone of modern machine-learning practice. To mitigate overfitting, trees
are typically regularized by a variety of techniques that modify their
structure (e.g. pruning). We introduce Hierarchical Shrinkage (HS), a post-hoc
algorithm that does not modify the tree structure, and instead regularizes the
tree by shrinking the prediction over each node towards the sample means of its
ancestors. The amount of shrinkage is controlled by a single regularization
parameter and the number of data points in each ancestor. Since HS is a
post-hoc method, it is extremely fast, compatible with any tree growing
algorithm, and can be used synergistically with other regularization
techniques. Extensive experiments over a wide variety of real-world datasets
show that HS substantially increases the predictive performance of decision
trees, even when used in conjunction with other regularization techniques.
Moreover, we find that applying HS to each tree in an RF often improves
accuracy, as well as its interpretability by simplifying and stabilizing its
decision boundaries and SHAP values. We further explain the success of HS in
improving prediction performance by showing its equivalence to ridge regression
on a (supervised) basis constructed of decision stumps associated with the
internal nodes of a tree. All code and models are released in a full-fledged
package available on Github (github.com/csinva/imodels)
