Dropout in Training Neural Networks: Flatness of Solution and Noise
  Structure
  It is important to understand how the popular regularization method dropout
helps the neural network training find a good generalization solution. In this
work, we show that the training with dropout finds the neural network with a
flatter minimum compared with standard gradient descent training. We further
find that the variance of a noise induced by the dropout is larger at the
sharper direction of the loss landscape and the Hessian of the loss landscape
at the found minima aligns with the noise covariance matrix by experiments on
various datasets, i.e., MNIST, CIFAR-10, CIFAR-100 and Multi30k, and various
structures, i.e., fully-connected networks, large residual convolutional
networks and transformer. For networks with piece-wise linear activation
function and the dropout is only at the last hidden layer, we then
theoretically derive the Hessian and the covariance of dropout randomness,
where these two quantities are very similar. This similarity may be a key
reason accounting for the goodness of dropout.
