Rethinking the Usage of Batch Normalization and Dropout in the Training
  of Deep Neural Networks
  In this work, we propose a novel technique to boost training efficiency of a
neural network. Our work is based on an excellent idea that whitening the
inputs of neural networks can achieve a fast convergence speed. Given the
well-known fact that independent components must be whitened, we introduce a
novel Independent-Component (IC) layer before each weight layer, whose inputs
would be made more independent. However, determining independent components is
a computationally intensive task. To overcome this challenge, we propose to
implement an IC layer by combining two popular techniques, Batch Normalization
and Dropout, in a new manner that we can rigorously prove that Dropout can
quadratically reduce the mutual information and linearly reduce the correlation
between any pair of neurons with respect to the dropout layer parameter $p$. As
demonstrated experimentally, the IC layer consistently outperforms the baseline
approaches with more stable training process, faster convergence speed and
better convergence limit on CIFAR10/100 and ILSVRC2012 datasets. The
implementation of our IC layer makes us rethink the common practices in the
design of neural networks. For example, we should not place Batch Normalization
before ReLU since the non-negative responses of ReLU will make the weight layer
updated in a suboptimal way, and we can achieve better performance by combining
Batch Normalization and Dropout together as an IC layer.
