Inner Ensemble Networks: Average Ensemble as an Effective Regularizer
  We introduce Inner Ensemble Networks (IENs) which reduce the variance within
the neural network itself without an increase in the model complexity. IENs
utilize ensemble parameters during the training phase to reduce the network
variance. While in the testing phase, these parameters are removed without a
change in the enhanced performance. IENs reduce the variance of an ordinary
deep model by a factor of $1/m^{L-1}$, where $m$ is the number of inner
ensembles and $L$ is the depth of the model. Also, we show empirically and
theoretically that IENs lead to a greater variance reduction in comparison with
other similar approaches such as dropout and maxout. Our results show a
decrease of error rates between 1.7\% and 17.3\% in comparison with an ordinary
deep model. We also show that IEN was preferred by Neural Architecture Search
(NAS) methods over prior approaches. Code is available at
https://github.com/abduallahmohamed/inner_ensemble_nets.
