Diversity and Generalization in Neural Network Ensembles
  Ensembles are widely used in machine learning and, usually, provide
state-of-the-art performance in many prediction tasks. From the very beginning,
the diversity of an ensemble has been identified as a key factor for the
superior performance of these models. But the exact role that diversity plays
in ensemble models is poorly understood, specially in the context of neural
networks. In this work, we combine and expand previously published results in a
theoretically sound framework that describes the relationship between diversity
and ensemble performance for a wide range of ensemble methods. More precisely,
we provide sound answers to the following questions: how to measure diversity,
how diversity relates to the generalization error of an ensemble, and how
diversity is promoted by neural network ensemble algorithms. This analysis
covers three widely used loss functions, namely, the squared loss, the
cross-entropy loss, and the 0-1 loss; and two widely used model combination
strategies, namely, model averaging and weighted majority vote. We empirically
validate this theoretical analysis with neural network ensembles.
