Neural Network Ensembles: Theory, Training, and the Importance of
  Explicit Diversity
  Ensemble learning is a process by which multiple base learners are
strategically generated and combined into one composite learner. There are two
features that are essential to an ensemble's performance, the individual
accuracies of the component learners and the overall diversity in the ensemble.
The right balance of learner accuracy and ensemble diversity can improve the
performance of machine learning tasks on benchmark and real-world data sets,
and recent theoretical and practical work has demonstrated the subtle trade-off
between accuracy and diversity in an ensemble. In this paper, we extend the
extant literature by providing a deeper theoretical understanding for assessing
and improving the optimality of any given ensemble, including random forests
and deep neural network ensembles. We also propose a training algorithm for
neural network ensembles and demonstrate that our approach provides improved
performance when compared to both state-of-the-art individual learners and
ensembles of state-of-the-art learners trained using standard loss functions.
Our key insight is that it is better to explicitly encourage diversity in an
ensemble, rather than merely allowing diversity to occur by happenstance, and
that rigorous theoretical bounds on the trade-off between diversity and learner
accuracy allow one to know when an optimal arrangement has been achieved.
