Deep Neural Network Ensembles
  Current deep neural networks suffer from two problems; first, they are hard
to interpret, and second, they suffer from overfitting. There have been many
attempts to define interpretability in neural networks, but they typically lack
causality or generality. A myriad of regularization techniques have been
developed to prevent overfitting, and this has driven deep learning to become
the hot topic it is today; however, while most regularization techniques are
justified empirically and even intuitively, there is not much underlying
theory. This paper argues that to extract the features used in neural networks
to make decisions, it's important to look at the paths between clusters
existing in the hidden spaces of neural networks. These features are of
particular interest because they reflect the true decision making process of
the neural network. This analysis is then furthered to present an ensemble
algorithm for arbitrary neural networks which has guarantees for test accuracy.
Finally, a discussion detailing the aforementioned guarantees is introduced and
the implications to neural networks, including an intuitive explanation for all
current regularization methods, are presented. The ensemble algorithm has
generated state-of-the-art results for Wide-ResNets on CIFAR-10 (top 5 for all
models) and has improved test accuracy for all models it has been applied to.
