Effective and Efficient Dropout for Deep Convolutional Neural Networks
  Convolutional Neural networks (CNNs) based applications have become
ubiquitous, where proper regularization is greatly needed. To prevent large
neural network models from overfitting, dropout has been widely used as an
efficient regularization technique in practice. However, many recent works show
that the standard dropout is ineffective or even detrimental to the training of
CNNs. In this paper, we revisit this issue and examine various dropout variants
in an attempt to improve existing dropout-based regularization techniques for
CNNs. We attribute the failure of standard dropout to the conflict between the
stochasticity of dropout and its following Batch Normalization (BN), and
propose to reduce the conflict by placing dropout operations right before the
convolutional operation instead of BN, or totally address this issue by
replacing BN with Group Normalization (GN). We further introduce a structurally
more suited dropout variant Drop-Conv2d, which provides more efficient and
effective regularization for deep CNNs. These dropout variants can be readily
integrated into the building blocks of CNNs and implemented in existing deep
learning platforms. Extensive experiments on benchmark datasets including
CIFAR, SVHN and ImageNet are conducted to compare the existing building blocks
and the proposed ones with dropout training. Results show that our building
blocks improve over state-of-the-art CNNs significantly, which is mainly due to
the better regularization and implicit model ensemble effect.
