Mean field theory for deep dropout networks: digging up gradient
  backpropagation deeply
  In recent years, the mean field theory has been applied to the study of
neural networks and has achieved a great deal of success. The theory has been
applied to various neural network structures, including CNNs, RNNs, Residual
networks, and Batch normalization. Inevitably, recent work has also covered the
use of dropout. The mean field theory shows that the existence of depth scales
that limit the maximum depth of signal propagation and gradient
backpropagation. However, the gradient backpropagation is derived under the
gradient independence assumption that weights used during feed forward are
drawn independently from the ones used in backpropagation. This is not how
neural networks are trained in a real setting. Instead, the same weights used
in a feed-forward step needs to be carried over to its corresponding
backpropagation. Using this realistic condition, we perform theoretical
computation on linear dropout networks and a series of experiments on dropout
networks. Our empirical results show an interesting phenomenon that the length
gradients can backpropagate for a single input and a pair of inputs are
governed by the same depth scale. Besides, we study the relationship between
variance and mean of statistical metrics of the gradient and shown an emergence
of universality. Finally, we investigate the maximum trainable length for deep
dropout networks through a series of experiments using MNIST and CIFAR10 and
provide a more precise empirical formula that describes the trainable length
than original work.
