BoostTree and BoostForest for Ensemble Learning
  Bootstrap aggregating (Bagging) and boosting are two popular ensemble
learning approaches, which combine multiple base learners to generate a
composite model for more accurate and more reliable performance. They have been
widely used in biology, engineering, healthcare, etc. This article proposes
BoostForest, which is an ensemble learning approach using BoostTree as base
learners and can be used for both classification and regression. BoostTree
constructs a tree model by gradient boosting. It achieves high randomness
(diversity) by sampling its parameters randomly from a parameter pool, and
selecting a subset of features randomly at node splitting. BoostForest further
increases the randomness by bootstrapping the training data in constructing
different BoostTrees. BoostForest outperformed four classical ensemble learning
approaches (Random Forest, Extra-Trees, XGBoost and LightGBM) on 34
classification and regression datasets. Remarkably, BoostForest has only one
hyper-parameter (the number of BoostTrees), which can be easily specified. Our
code is publicly available, and the proposed ensemble learning framework can
also be used to combine many other base learners.
