Trees, Forests, Chickens, and Eggs: When and Why to Prune Trees in a
  Random Forest
  Due to their long-standing reputation as excellent off-the-shelf predictors,
random forests continue remain a go-to model of choice for applied
statisticians and data scientists. Despite their widespread use, however, until
recently, little was known about their inner-workings and about which aspects
of the procedure were driving their success. Very recently, two competing
hypotheses have emerged -- one based on interpolation and the other based on
regularization. This work argues in favor of the latter by utilizing the
regularization framework to reexamine the decades-old question of whether
individual trees in an ensemble ought to be pruned. Despite the fact that
default constructions of random forests use near full depth trees in most
popular software packages, here we provide strong evidence that tree depth
should be seen as a natural form of regularization across the entire procedure.
In particular, our work suggests that random forests with shallow trees are
advantageous when the signal-to-noise ratio in the data is low. In building up
this argument, we also critique the newly popular notion of "double descent" in
random forests by drawing parallels to U-statistics and arguing that the
noticeable jumps in random forest accuracy are the result of simple averaging
rather than interpolation.
