Optimizing Ensemble Weights and Hyperparameters of Machine Learning
  Models for Regression Problems
  Aggregating multiple learners through an ensemble of models aim to make
better predictions by capturing the underlying distribution of the data more
accurately. Different ensembling methods, such as bagging, boosting, and
stacking/blending, have been studied and adopted extensively in research and
practice. While bagging and boosting focus more on reducing variance and bias,
respectively, stacking approaches target both by finding the optimal way to
combine base learners. In stacking with the weighted average, ensembles are
created from weighted averages of multiple base learners. It is known that
tuning hyperparameters of each base learner inside the ensemble weight
optimization process can produce better performing ensembles. To this end, an
optimization-based nested algorithm that considers tuning hyperparameters as
well as finding the optimal weights to combine ensembles (Generalized Weighted
Ensemble with Internally Tuned Hyperparameters (GEM-ITH)) is designed. Besides,
Bayesian search was used to speed-up the optimizing process, and a heuristic
was implemented to generate diverse and well-performing base learners. The
algorithm is shown to be generalizable to real data sets through analyses with
ten publicly available data sets.
