Multi-Sample Dropout for Accelerated Training and Better Generalization
  Dropout is a simple but efficient regularization technique for achieving
better generalization of deep neural networks (DNNs); hence it is widely used
in tasks based on DNNs. During training, dropout randomly discards a portion of
the neurons to avoid overfitting. This paper presents an enhanced dropout
technique, which we call multi-sample dropout, for both accelerating training
and improving generalization over the original dropout. The original dropout
creates a randomly selected subset (called a dropout sample) from the input in
each training iteration while the multi-sample dropout creates multiple dropout
samples. The loss is calculated for each sample, and then the sample losses are
averaged to obtain the final loss. This technique can be easily implemented by
duplicating a part of the network after the dropout layer while sharing the
weights among the duplicated fully connected layers. Experimental results using
image classification tasks including ImageNet, CIFAR-10, and CIFAR-100 showed
that multi-sample dropout accelerates training. Moreover, the networks trained
using multi-sample dropout achieved lower error rates compared to networks
trained with the original dropout. The additional computation cost due to the
duplicated operations is not significant for deep convolutional networks
because most of the computation time is consumed in the convolution layers
before the dropout layer, which are not duplicated.
