To Bag is to Prune
  It is notoriously difficult to build a bad Random Forest (RF). Concurrently,
RF blatantly overfits in-sample without any apparent consequence out-of-sample.
Standard arguments, like the classic bias-variance trade-off or double descent,
cannot rationalize this paradox. I propose a new explanation: bootstrap
aggregation and model perturbation as implemented by RF automatically prune a
latent "true" tree. More generally, randomized ensembles of greedily optimized
learners implicitly perform optimal early stopping out-of-sample. So there is
no need to tune the stopping point. By construction, novel variants of Boosting
and MARS are also eligible for automatic tuning. I empirically demonstrate the
property, with simulated and real data, by reporting that these new completely
overfitting ensembles perform similarly to their tuned counterparts -- or
better.
