Efficient Diversity-Driven Ensemble for Deep Neural Networks
  The ensemble of deep neural networks has been shown, both theoretically and
empirically, to improve generalization accuracy on the unseen test set.
However, the high training cost hinders its efficiency since we need a
sufficient number of base models and each one in the ensemble has to be
separately trained. Lots of methods are proposed to tackle this problem, and
most of them are based on the feature that a pre-trained network can transfer
its knowledge to the next base model and then accelerate the training process.
However, these methods suffer a severe problem that all of them transfer
knowledge without selection and thus lead to low diversity. As the effect of
ensemble learning is more pronounced if ensemble members are accurate and
diverse, we propose a method named Efficient Diversity-Driven Ensemble (EDDE)
to address both the diversity and the efficiency of an ensemble. To accelerate
the training process, we propose a novel knowledge transfer method which can
selectively transfer the previous generic knowledge. To enhance diversity, we
first propose a new diversity measure, then use it to define a diversity-driven
loss function for optimization. At last, we adopt a Boosting-based framework to
combine the above operations, such a method can also further improve diversity.
We evaluate EDDE on Computer Vision (CV) and Natural Language Processing (NLP)
tasks. Compared with other well-known ensemble methods, EDDE can get highest
ensemble accuracy with the lowest training cost, which means it is efficient in
the ensemble of neural networks.
