{'fraction_non_alphanumeric': 0.039883496969981676, 'fraction_numerical': 0.014421947667590548, 'mean_word_length': 4.323871451794423, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Implicit regularization of dropout

It is important to understand how the popular regularization method dropout helps the neural network training find a good generalization solution. In this work, we theoretically derive the implicit regularization of dropout and study the relation between the Hessian matrix of the loss function and the covariance matrix of the dropout noise, supported by a series of experiments. We then numerically study two implications of the implicit regularization of dropout, which intuitively rationalize why dropout helps generalization. First, we find that the training with dropout finds the neural network with a flatter minimum compared with standard gradient descent training in experiments, and the implicit regularization is the key for finding flat solutions. Second, trained with dropout, input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) would tend to condense on isolated orientations. Condensation is a feature in non-linear learning process, which makes the neural network low complexity. Although our theory mainly focuses on the dropout used in the last hidden layer, our experiments apply for general dropout in training neural networks. This work points out the distinct characteristics of dropout compared with stochastic gradient descent and serves as an important basis for fully understanding dropout.


Introduction
Dropout is used with gradient-descent-based algorithms for training deep neural networks (DNNs) (Hinton et al., 2012;Srivastava et al., 2014), which drives the state-of-the-art test performance in deep learning (Tan and Le, 2019;Helmbold and Long, 2015). During training, the output of each neuron is multiplied with a random variable with probability p as 1/p and 1 − p as zero. Note that p is called dropout rate, and every time for computing concerned quantity, the variable is randomly sampled at each feedforward operation. Dropout has been an indispensable trick in the training of DNNs.
An approach to understanding the effect of dropout is to explicitly obtain the implicit regularization of dropout during the training. In this work, we study the training of a neural network, where the last hidden layer (closest to the output layer) is equipped with a dropout. We theoretically derive the modified gradient flow (Hairer et al., 2003) of the dropout training, where the modified gradient flow is close to the discrete iterates of the fullbatch gradient descent with dropout randomness up to the second-order of learning rate. We also derive two important quantities in the expectation with respect to the dropout randomness. The first one is the covariance matrix of the dropout randomness and the second one is the Hessian of the dropout loss landscape, which have very similar forms.
There are two terms in the implicit regularization of dropout. One originates from the random dropout of neurons, denoted as R 1 (θ), and the other one originates from the discretization, denoted as R 2 (θ). R 1 (θ) is independent with the learning rate. Therefore, as the learning rate approaches the limit of zero, only R 2 (θ) vanishes and the effect of R 1 (θ) still plays an important role in the training process. This is starkly different from gradient descent (GD) or stochastic gradient descent (SGD) Smith et al. (2020), where the implicit regularization completely vanishes in the limit of zero learning rate. We use a series of experiments to validate the effect of the two implicit regularization terms and the similarity between covariance matrix and Hessian. We then numerically study two implications of the implicit regularization of dropout.
First, we study the flatness of the solution. Previous study conjectured on SGD that the close relation between the covariance matrix of noise and the Hessian matrix of the loss landscape may lead to a flat minimum (Zhu et al., 2018;Feng and Tu, 2021). Inspired by this, we show that compared with the standard gradient descent (GD), the GD with dropout selects flatter minima. As suggested by many existing works (Keskar et al., 2016;Neyshabur et al., 2017;Zhu et al., 2018), flatter minima are more likely to have better generalization and stability. By using the modified gradient flow to train neural networks, we found R 1 (θ) makes a major contribution of finding a flat minimum, which indicates that the main mechanism of finding flat solution in dropout could be different from SGD.
Second, we study the weight condensation in dropout training. Previous work  found that in the nonlinear training regime, input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) would condense on isolated orientations under gradient flow training. In the linear regime, such as neural tangent kernel initialization (Jacot et al., 2018), the weights are close to the random initial values and condensation can not happen in gradient descent training. As the implicit regularization R 1 (θ) penalizes the L 2 norm of the output of all neurons, we conjecture that the dropout facilitates the condensation. We then find that even in the linear regime, with dropout, weights show a clear condensation in experiments. Note that SGD training does not show any help on condensation. As the condensation makes the neural network low complexity, dropout may help the generalization by constraining the model complexity.
This work comprehensively studies the implicit regularization of dropout and its implications. Although our theory mainly focuses on the dropout used in the last hidden layer, our experiments apply for general dropout in training neural networks. Our results show that dropout has a distinct implicit regularization for finding flat minima and facilitating weight condensation, which may together lead to improve generalization of neural networks.

Related Works
Dropout is proposed as a simple way to prevent neural networks from overfitting, and thus improving the generalization of the network (Hinton et al., 2012;Srivastava et al., 2014). Many works aim to find an explicit form of dropout. McAllester (2013) presents PAC-Bayesian bounds, and Wan et al. (2013), Mou et al. (2018) derive Rademacher generalization bounds. These results show that the reduction of complexity brought by dropout is O(p), where p is the probability of keeping an element in dropout. Mianjy and Arora (2020) show that dropout training with logistic loss achieves -suboptimality in test error in O(1/ ) iterations. All of the above works need specific settings, such as norm assumptions and logistic loss, and they only give a rough estimate of the generalization error bound, which usually consider the worst case. However, it is not clear what is the characteristic of the dropout training process and how to bridge the training with the generalization. In this work, we show that the implicit regularization and the noise structure of dropout, which may be a key reason why dropout finds solutions with better generalization.
Some works attribute the improvement in flatness to the similarity between the covariance matrix and the Hessian matrix of the loss function of SGD (Papyan, 2018(Papyan, , 2019. For example, Feng and Tu (2021) investigate the inverse variance-flatness relation for SGD and Zhu et al. (2018) study the Hessian-variance alignment for SGD. In this work, we find the noise structure of dropout is highly correlated with the loss landscape. In addition, we also find that the implicit regularization of dropout is also a key for finding a flat minimum.
The modified gradient flow is defined as the gradient flow which is close to discrete iterates of the original training path up to some high-order learning rate term (Hairer et al., 2003).  derive the modified gradient flow of discrete fullbatch gradient descent training asR S, is the i-th batch loss and the last term is also called "non-uniform" term . Our work shows that there exist several distinct features between dropout and SGD. In the limit of vanishing learning rate, the modified gradient flow of dropout still has an extra implicit regularization term while that of SGD converges to the full-batch gradient flow (Yaida, 2018).  identify the condensed regime for two-layer wide ReLU NNs that weights condense in isolated directions during the training. Zhang et al. (2021b show that neural networks of different widths often exhibit similar condensation behavior, e.g., stagnating at similar loss with almost the same output function. Zhang et al. (2021b propose an embedding princple that the loss landscape of an NN contains all critical points of all the narrower NNs. The embedding principle provides a basis why the condensation can happen from the perspective of loss landscape. Several works study the mechanism of condensation at an initial training stage, such as for ReLU network (Maennel et al., 2018;Pellegrini and Biroli, 2020) and network with continuously differentiable activation functions (Zhou et al., 2021).

Deep Neural Networks
Consider L-layer (L ≥ 2) fully-connected DNNs. We regard the input as the 0th layer and the output as the Lth layer. Let m l be the number of neurons in the lth layer. In particular, m 0 = d and m L = d . For any i, k ∈ N and i < k, we denote [i : k] = {i, i + 1, . . . , k}. In particular, we denote [k] := {1, 2, . . . , k}. Given weights W [l] ∈ R m l ×m l−1 and bias b [l] ∈ R m l for l ∈ [L], we define the collection of parameters θ as a 2L-tuple (an ordered list of 2L elements) whose elements are matrices or vectors where the lth layer parameters of θ is the ordered pair θ| l = W [l] , b [l] , l ∈ [L]. We may misuse of notation and identify θ with its vectorization vec(θ) ∈ R M with M = L−1 l=0 (m l + 1)m l+1 . Given θ ∈ R M , the neural network function f θ (·) is defined recursively. First, we write f For notational simplicity, we denote

Loss function
The training data set is denoted as For simplicity, here we assume an unknown function y satisfying y(x i ) = y i for i ∈ [n]. The empirical risk reads as where the expectation E S h(x) := 1 n n i=1 h(x i ) for any function h : R d → R and the loss function (·, ·) is differentiable and the derivative of with respect to its first argument is denoted by ∇ (y, y * ). The error with respect to data sample (x i , y i ) reads as For notation simplicity, we denote (f θ (x i ), y i ) = θ,i .

Dropout
For f [l] θ (x) ∈ R m l , we sample a scaling vector η ∈ R m l with independent random coordinates, θ (x). Note that η is a zero-mean random variable. We then apply dropout by computing θ (x), and using f [l] θ,η (x) instead of f [l] θ (x). Here we use for the Hadamard product of two matrices of the same dimension. With slight abuse of notation, we let η denote the collection of such vectors over all layers. f drop θ,η (x) denotes the output of model f θ (x) on input x using dropout noise η. R drop S (θ, η) denotes the empirical risk with respect to network with dropout layer f drop θ,η , i.e., 3.4 Random data collected from dropout training

Random trajectory data
The training process of neural networks is usually divided into two phases, fast convergence and exploration phase (Shwartz-Ziv and Tishby, 2017). In this work, we follow the experimental scheme in Feng and Tu (2021) to show the similarity between dropout and SGD. This can be understood by frequency principle (Xu et al., 2019(Xu et al., , 2020Zhang et al., 2021a), which states that DNNs fast learn low-frequency components but slowly learn high-frequency ones. We collect parameter sets S para = {θ i } N i=1 from N consecutive training steps in the exploration phase, where θ i is the network parameter set at ith sample point.

Random gradient data
We often need larger time interval for enough sampling to estimate the covariance accurately. Although the network loss is small, compared with the initial sampling parameters, the network parameters could have large changes during the long-time sampling. Therefore, much extra noise may be induced. Meanwhile, for dropout, it is difficult to get a small loss value on large networks and datasets, therefore, model parameters often have large fluctuations during the sampling. To overcome this problem, we propose a more appropriate sampling method to avoid additional noise caused by sampling parameters in a large time interval.
We train the network until the loss is small enough and then freeze the training. We sample N gradients of the loss function w.r.t. the parameters with different dropout variables, i.e., S grad = {g i } N i=1 . In each sample, the dropout rate is fixed. In this way, we can get the noise structure of dropout without being affected by parameter changes caused by long-term training.

Hessian-variance alignment
Similar to Zhu et al. (2018), we quantify the alignment between the noise structure and the curvature of loss landscape by where Σ i is the ith-step covariance matrix of dropout layers and H i is the Hessian matrix of the loss landscape at network parameters of the ith-step.

Inverse variance-flatness relation
We study the inverse variance flatness relation for both random trajectory data and random gradient data. For convenience, we denote data as S and its covariance as Σ.

Variance vs. interval flatness
The definitions of variance and interval flatness are as follows: Definition 1 (Variance of data at an eigen direction). For data S and its covariance Σ, by denoting λ i (Σ) as the ith eigenvalue of Σ, we write λ i (Σ) as the variance of the data at the corresponding eigen direction.
Definition 2 (Interval flatness * ). For a specific solution θ * 0 , the loss function profile L v along the direction v is: L v (δθ) ≡ L(θ * 0 + δθv), where δθ represents the distance moved in the v direction. The interval flatness F v is defined as the width of the region within which L v (δθ) ≤ 2L v (0). We determine F v by finding two closest points θ l v < 0 and θ r v > 0 on each side of the minimum that satisfy The interval flatness is defined as: Remark 3. The experiments show that the result is not sensitive to the selection of the pre-factor 2. A larger value of F v means a flatter landscape in the direction v.
Denote λ i (Σ) as the ith eigenvalue of Σ, and denote its corresponding eigen-vector as v i (Σ). The interval flatness of the loss landscape in the direction v i (Σ) is denoted as F v i (Σ) . We then experimentally explore the relation of

Projected variance vs. Hessian flatness
The definitions of projected variance and Hessian flatness are as follows: Definition 4 (Projected variance). For a given direction v ∈ R d θ and a parameter set the inner product of v and θ i is denoted by Proj v (θ i ) := θ T i v, then we can define the projected variance at direction v with respect to the sample set S as follows, Definition 5 (Hessian flatness). For Hessian matrix H, by denoting λ i (H) as the ith eigenvalue of H corresponding to the eigenvector v i , we write λ i (H) as the Hessian flatness of direction v i .
To obtain the variance induced by the dropout at a fixed parameter position θ, we propose another way to characterize the inverse variance-flatness relation. For given data S and Hessian matrix H, we experimentally explore the relation of {Var( is the ith eigenvalue and eigenvector of H, respectively. * . This definition is also used in Feng and Tu (2021)

Experimental setups
For Fig. 1, Fig. 8, Fig. 9, we use the FNN with size 784 − 1000 − 10. We add dropout layers behind the hidden layer with different dropout rates. We train the network using GD with a learning rate of 5e − 3. For Fig. 2, Fig. 3(b), we use vgg-9 (Simonyan and Zisserman, 2014) w/o dropout layers to classify CIFAR-10 using GD or SGD. For experiments with dropout layers, we add dropout layers after the pooling layers, the dropout rates of dropout layers are 0.5. We only use the first 1000 examples for training to compromise with the computational burden.
For Fig. 3(a), we classify the Fashion-MNIST dataset by training four-layer NNs of width 4096 with a training batch size of 64. We add dropout layers behind the hidden layers with different dropout rates p. The learning rate is 5e − 3.
For Fig. 7(a), we use the FNN with size 784 − 1024 − 1024 − 10. We add dropout layers behind the first and the second layers with dropout rate of 0.8 and 0.5, respectively. We train the network using default Adam optimizer (Kingma and Ba, 2015) with a learning rate of 0.0001.
For Fig. 7(b), we use vgg-9 to compare the loss landscape flatness w/o dropout layers. For experiment with dropout layers, we add dropout layers after the pooling layers, the dropout rates of dropout layers are 0.8. Models are trained using GD with Nesterov momentum, training-size 2048 for 300 epochs. The learning rate is initialized at 0.1, and divided by a factor of 10 at epochs 150, 225 and 275. We only use the first 2048 examples for training to compromise with the computational burden.
For Fig. 4, Fig. 5(a, d), Fig. 6(a, d), we use the FNN with size 784 − 50 − 50 − 10. We train the network using GD with the first 10,000 training data as the training set. We add a dropout layer behind the second layer. The dropout rate and learning rate are specified and unchanged in each experiment. We only consider the parameter matrix corresponding to the weight and the bias of the fully-connected layer between two hidden layers. For Fig. 5(b, e), Fig. 6 Fig. 7(c), we use ResNet-20 (He et al., 2016) to compare the loss landscape flatness w/o dropout layers. For experiment with dropout layers, we add dropout layers after the convolutional layers, the dropout rates of dropout layers are 0.8. We only consider the parameter matrix corresponding to the weight of the first convolutional layer of the first block of the ResNet-20. Models are trained using GD, training-size 50000 for 1200 epochs. The learning rate is initialized at 0.01. Since the Hessian calculation of ResNet takes much time, for the ResNet experiment, we only perform it at a specific dropout rate and learning rate.
For Fig. 5(c, f), Fig. 6(c, f), Fig. 7(d), we use transformer (Vaswani et al., 2017) , the meaning of the parameters is consistent with the original paper. In order to calculate the Hessian matrix and eigendecomposition more accurately and quickly, we reasonably reduce the number of network parameters. We only consider the parameter matrix corresponding to the weight of the fully-connected layer whose output is queries in the Multi-Head Attention layer of the first block of the decoder. For experiment with dropout layers, we apply dropout to the output of each sub-layer, before it is added to the sub-layer input and normalized. In addition, we apply dropout to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks. The dropout rates of dropout layers are 0.9. For the English-German translation problem, we use the cross-entropy loss with label smoothing trained by full-batch Adam based on the Multi30k dataset. The learning rate strategy is the same as that in the article. The warm up step is 4000 epochs, the training step is 10000 epochs. We only use the first 2048 examples for training to compromise with the computational burden.
For Fig. 10, Fig. 11, we use the FNN with size 1 − 1000 − 1 to fit the target function as follows, where σ(x) = ReLU(x). For Fig. 10, we add dropout layers behind the hidden layer with dropout rate p = 0.8. We train the network using GD with a learning rate of 1e − 3. For Fig. 11, we train the network using SGD with a learning rate of 1e − 3

Main theoretical results
In this section, we first derive the implicit regularization of dropout in the expectation sense. Then, we obtain the Hessian matrix of the loss function in the expectation sense and the covariance matrix of the dropout noise. We first summarize assumptions and provide definitions needed for all our results below.
Setting 1 (Dropout structure). Consider an L-layer (L ≥ 2) fully-connected DNN has only one dropout layer after the (L − 1)th layer of the network, Setting 2 (Loss function). Take the mean squared error (MSE) as our loss function, Setting 3 (Network structure). Take the piece-wise linear function as our activation function. For convenience, we further set that the model output is a one-dimensional vector, i.e. m L = 1.
Assumption 1. We examine the loss landscape after training reaches a stable stage, so we assume that the gradient of the loss function in the average sense is small enough, i.e.

Modified gradient flow
Based on the above assumptions, we obtain a modified equation based on dropout gradient flow.
Lemma 6 (The expectation of dropout loss). Given an L-layer fully-connected neural network with dropout f drop θ,η (x), based on Setting 1-3, we have the expectation of dropout MSE: Proof With the definition of MSE, we have, With the definition of η, we have E(η) = 0, thus, Based on the above lemma, we further consider the discrete iterate training of gradient descent with dropout and obtain the modified gradient flow of dropout.
Theorem 7 (Modified gradient flow of dropout). Based on Setting 1-3, the mean iterate of f drop θ,η (x) with a small but finite learning rate , stays close to the path of gradient flow on a modified lossθ = −∇ θR drop S (θ), where the modified lossR drop S (θ) satisfies: where

Proof
We assume the ODE of the GD with dropout has the formθ = f (θ), and introduce a modified gradient flowθ =f (θ), where, Thus by Taylor expansion, for small but finite , we have, For GD, we have, Combining θ(t + ) = θ t+1 , θ(t) = θ t , we have, Thus, with random variable η, In the average sense and combining Lemma 6 , we have, We have,

Hessian and covariance
With similar computation as above, we can derive the Hessian of the loss landscape in the expectation sense with respect to the dropout noise η and the covariance matrix of dropout noise.
Theorem 8 (Hessian matrix with dropout regularization). Based on the Setting 1-3 and Assumption 1, the Hessian matrix of the loss function with respect to f drop θ,η (x) can be written in the mean sense as:

Proof
We first compute the Hessian matrix after taking expectation with respect to the dropout variable, The first and second terms on the RHS of the Eq. (3) are as follows, Note that for linear activate function,

Thus the Eq. (3) can be rewritten as
Theorem 9 (Covariance matrix with dropout regularization). Based on the Setting 1-3 and Assumption 1, the covariance matrix of the loss function under the randomness of dropout variable η and data x can be written as: Proof For simplicity, we approximate the loss function through Taylor expansion, which is also used in Wei et al. (2020), The covariance matrix obtained using SGD under dropout regularization is Combining the properties of the dropout variable η, we have, We calculate the two terms on the RHS of the Eq. (4) separately:

Under the assumption that
Thus the Eq. (4) can be rewritten as Note that Proposition 1. Based on the Setting 1-3 and Assumption 1, we can further restrict the problem to a binary classification problem, i.e. y i ∈ {0, 1}, ∀i ∈ [n], and assume the model output f θ (x i ) ∈ [δ, 1 − δ] (for example, we can limit the network output using a threshold activation function), where δ is a small positive constant, then we have: Proof The properties (i)-(ii) are direct consequences of Theorem 8, 9.

Numerical verification of implicit regularization terms
In this section, we numerically verify the validity of two implicit regularization terms. We also show the Hessian and the covariance are highly correlated by a series of experiments. As R 1 (θ) is independent with the learning rate and R 2 (θ) vanishes in the limit of zero learning rate, we select small learning rate to verify the validity of R 1 (θ). From the form †. A B means that (B − A) is semi-positive definite.
of Eq. 2, the modified equation of dropout can be approximated by R S (θ) + R 1 (θ) when the learning rate is small enough. Therefore, we examine the validity of R 1 (θ) through the similarity of the neural network trained by the two loss functions, i.e., R drop S (θ) and R S (θ) + R 1 (θ) under a small learning rate. As shown in Fig. 1, we train two-layer NNs of width 1000 to classify the first 1000 images of MNIST using two loss forms with learning rate = 5 × 10 −3 . In Fig. 1 (a), we show the test accuracy of two losses trained under different dropout rates. For R S (θ) + R 1 (θ), the dropout rate has an effect on R 1 (θ), although the network does not have a dropout layer. It can be seen that for different dropout rates, the networks obtained by the two losses have obtained very similar test accuracy. It is worth mentioning that for the network trained with R S (θ), only an accuracy of 79% can be obtained. In Fig. 1 (b), we show the values of R S (θ) and R 1 (θ) for the two networks at different dropout rates. For the network obtained by R drop S (θ) training, we can still calculate the two terms through its parameters. It can be seen that for different dropout rates, the values of R S (θ) and R 1 (θ) of the two networks are almost indistinguishable.
value of the resulting model in (a) under different learning rates or regularization coefficients λ.

Validation of R
To validate R 2 (θ) in the training with learning rate , we can train another network by adding a penalty term (λ/4) ∇ θ R drop S (θ, η) 2 to the dropout loss R drop S (θ, η) with a learning rate much smaller than . The exact form of R 2 (θ) has an expectation with respect to η, but in experiments, we ignore this expectation. Therefore, we can only qualitatively validate the effect of R 2 (θ).
As shown in Fig. 2, we train the NNs with different learning rates (blue) using loss function R drop S (θ, η), and a fixed small learning rate (orange) using loss function R drop Fig. 2 (a), the learning rate and the regularization coefficient λ are similar when they reach the maximum test accuracy (red point).
In addition, as shown in Fig. 2 (b), we examine the value of under different learning rates (blue dots) and different regularization coefficients (orange dots). In practical experiments, we take 3000 different η's to approximate the expectation. We can see that increasing either the learning rate or the regularization coefficient λ ) at the end of training, and their values are very similar.
Due to the limitation of the training speed of full-batch GD, we only use a few training samples in the above experiments. We conduct similar experiments with dropout using SGD. As shown in Fig. 3, the learning rate and the regularization coefficient λ are similar when they reach the maximum test accuracy (red point).

Hessian-variance alignment
Similar to SGD, the effect of dropout can be equivalent to imposing a specific noise on the gradient. A random noise, such as isotropic noise, can help the training escape local minima, but can not robustly improve generalization (An, 1996;Zhu et al., 2018). The noise induced by the dropout should have certain properties that can lead the training to good minima. Our derivation show that the Hessian of the loss landscape and the covariance of dropout noise is highly correlated. In the following, we would show the high correlation between these two quantities in experiments.
In this subsection, we study the alignment between the Hessian and the random gradient covariance at each training step, i.e., Hessian-variance alignment. Note that the training is performed by GD without dropout. At step i, we sample the gradients of parameters {g j i } N j=1 by tentatively adding a dropout layer between the hidden layers. For each step i, we the compute Tr(H i Σ i ), where H i is the Hessian of the loss at the parameter set at step i and Σ i is the covariance of {g j i } N j=1 . In order to show the anisotropic structure, we construct the isotropic noise for comparison, i.e.,Σ i = Tr Σ i D I of the covariance matrix Σ i , where D is the number of parameters. In our experiments, D = 2500. As shown in Fig. 4, in the whole training process under different learning rates and dropout rates, Tr(H i Σ i ) is much larger than Tr(H iΣi ), indicating the anisotropic structure of dropout noise and its high alignment with the Hessian matrix.

Inverse variance-flatness relation
The alignment also implies that the noise induced by the dropout satisfies the inverse variance-flatness relation, that is, the noise variance is larger along the sharper direction of the loss landscape at a minimum. In this subsection, we will verify this relation by two sets of experiments.

Variance vs. interval flatness
We use the principal component analysis (PCA) to study the weight variations when the accuracy is nearly 100%. For FNNs, networks are trained on MNIST with the first 10000 examples as the training set for computational efficiency. For ResNets, networks are trained on CIFAR-100 with 50000 examples as the training set. For the transformer structure, the network is trained by Multi30k (Vaswani et al., 2017). The networks are trained with full batch for different learning rates and dropout rates under the same random seed (that is, with the same initialization parameters). When the loss is small enough, we sample the parameters or gradients of parameters N times (N = 3000 in this experiment) and use the method introduced in Section 3.4 to construct covariance matrix Σ by the weights S para or gradients S grad of specific network parameters. The PCA is done for the covariance matrix Σ. We then compute the interval flatness of the loss function landscape at eigen-directions, i.e., indicate the variance of weights S para or gradients S grad at corresponding eigen-directions. As shown in Fig. 5, for different learning rates and dropout rates, there is an inverse relationship between the interval flatness of the loss function landscape {F v i (Σ) } N i=1 and the dropout variance, i.e., the PCA spectrum We can approximately see a powerlaw relationship between More detailed, for the small flatness part, the variance of noise induced by dropout is generally large, which indicates that the noise induced by dropout has larger variance in sharp directions; for the large flatness part, as the loss landscape gets flatter, the linear relationship is more obvious, and we can see a clearer asymptotic behavior in the results. Overall, we can observe the negative correlation between the variance and flatness in Fig. 5.

Projected variance vs. Hessian flatness
The eigenvalues of the Hessian of the loss at a minimum are also often used to indicate the flatness. A large eigenvalue corresponds to a sharper direction. In this section, we study the relationship between eigenvalues of Hessian H of loss landscape at the end point of training and the variances of dropout at corresponding eigen-directions. As mentioned in the Preliminary, we sample the parameters or gradients of parameters 1000 times, that is N = 1000. For each eigen-direction v i of Hessian H, we project the sampled parameters or the gradients of sampled parameter to direction v i by inner product, denoted by Proj v i (S). Then, we compute the variance of the projected data, i.e., Var(Proj v i (S)).
As shown in Fig. 6, we find that there is also a power-law relationship between for different dropout rates and learning rates, no matter S is sampled from parameters or gradients of parameters. The positive correlation between the eigenvalue and the projection variance show the structure of the dropout noise, which helps the network escape the bad minima. At the same time, as shown in Figs. 5 for different choices of dropout rate p and learning rate lr with different network structures. The PCA is done for different datasets S sampled from parameters for the top line and sampled from gradients of parameters for the bottom line. The dash lines give the approximate slope of the scatter. and 6, we can see that gradient sampling has a more clear linear structure than that of parameter sampling.

Implicit regularization of dropout on the flatness of solution
Dropout is almost ubiquitous in training deep networks. It is interesting and important to understand what makes dropout improve the generalization of training neural networks. Inspired by the study of SGD (Keskar et al., 2016) and the similarity between Hessian and covariance, we explore the flatness of the minima found by dropout. We also study the contribution of R 1 (θ) and R 2 (θ) to the flatness of loss landscape and network generalization.

Dropout finds flatter minima
We adopt the method of Li et al. (2017) in this work as follows. To obtain a direction for a network with parameters θ, we begin by producing a random Gaussian direction vector d with dimensions compatible with θ. Then, we normalize each filter in d to have the same norm of the corresponding filter in θ. For FNN, each layer can be regarded as a filter. The normalization process is equivalent to normalizing the layer. For CNN, each convolution kernel may have multiple filters. Each filter is normalized individually. In other words, we make the replacement d i,j ← d i,j d i,j θ i,j , where d i,j , θ i,j represent the jth filter of the ith layer of the random direction d and the network parameters θ, and · denotes the Frobenius norm. It should be noted that j is not the index of the weight, but the filter. We use f (α) = L (θ + αd) to characterize the loss landscape around the minima obtained with dropout layers θ * d and without dropout layer θ * . For all network structures shown in Fig. 7, dropout can improve the generalization of the network and find a flatter minimum. In Fig. 7(a, b), for both networks trained with and without dropout layers, the training loss values are all closed to zero, but their flatness and generalization are still different. In Fig. 7(c, d), due to the complexity of the dataset, i.e., CIFAR-100 and Multi30k, and networks, i.e., ResNet-20 and transformer, networks with dropout layers does not achieve very small training error but the ones with dropout find flatter minima with much better generalization.

Effect of two implicit regularization terms on generalization and flatness
The modified gradient flow has no noise in the training. However, we find that it can also find a flat mininum that generalizes well. In this subsection, we study the effect of each regularization term through training networks by the following loss functions: For convenience, we denote ( /4) ∇ θ R drop S (θ, η) 2 asR 2 (θ, η). It is easy to find that E ηR2 (θ, η) = R 2 (θ). In L i (θ), i ∈ [4], we explicitly add or subtract penalty terms of either R 1 (θ) or R 2 (θ) to examine their contribution to dropout regularization.
First, we study the effect on generalization. In Fig. 8, we compare the test accuracy obtained by training with the above four losses and also use the results of R S (θ) and R drop S (θ, η) as reference results under different dropout rates with two different learning rates (solid: = 0.05, dashed: = 0.005). There are two ways to consider the effect of R 1 (θ), one is to add R 1 (θ) to R S (θ) and the other way is to subtract R 2 (θ) from R drop S (θ, η), as shown in Fig. 8(a). Clearly, both approaches show that with R 1 (θ), the training without dropout finds a solution that almost has the same test accuracy as the training with dropout. For R 2 (θ), either by addition or subtraction, as shown in Fig. 8(b), compared with R 1 (θ), the effect of R 2 (θ) only slightly improves the generalization from full-batch gradient descent training.  Second, we study the effect on flatness. To explore the flatness, we show the a onedimensional cross-section by the interpolation between two minima found by the training of two different loss functions. For either R 1 (θ) or R 2 (θ), we use addition or subtraction to study its effect. As shown in Fig. 9(a), for R 1 (θ), the loss value of the interpolation between the minima found by the addition approach (L 1 ) and the subtraction approach (L 3 ) stays very small, which is similar for R 1 (θ) in Fig. 9(b). We then compare the flatness of minima found by the training with R 1 (θ) and R 2 (θ). As shown in Fig. 9(c-f), the flatness of the minima found by the training with R 1 (θ) is consistently flatter than that of R 2 (θ).
The experiments in this section show that the unique implicit regularization of dropout, R 1 (θ), is significantly important to improve the generalization and find flat minima. Although many works have also pointed out the co-existence of good generalization and flat minima, it is still unclear how flat minima lead to better generalization. To reveal this connection, we probe into the evolution of weights.

Dropout facilitates condensation
As can be seen from the implicit regularization term R 1 , dropout regularization imposes an additional L 2 constraint on the output of each neuron. To understand the effect of R 1 , we take an example of two neurons. (f) L3&L4 Figure 9: Interpolations between models with α interpolation coefficient. For L i &L j , at α = 0 there is one trained model (trained by loss function L i ), at α = 1 there is another (trained by loss function L j ). Different curves represent different dropout rates used for training.
We use the following two-layer ReLU network to fit a one-dimensional function: is the jth column of W [2] . For example, consider m = 2, σ(x) = ReLU(x), and suppose the network can perfectly fit a training data set of two data points generated by a target function of σ(W * x), denoted as o * . Denote the output of the j-th neuron over samples as R 1 can then be written as 2 //W * , which is the condensation phenomenon. A network with condensed parameters is equivalent to another network with smaller width (Zhou et al., 2021;. Therefore, the effective complexity of the network is smaller than what it looks superficially. Such low effective complexity may be an underlying reason for good generalization. In addition, the embedding principle (Zhang et al., 2021bBai et al., 2022) shows that although the condensed network is equivalent to a small one in the sense of approximation, it has more degeneracy and more descent directions that may lead to an easier training process. Future work can study how condensation affects the flatness of minima. We then study the effect of dropout on the neural network output and the weight condensation.
We train a network with 1000 hidden neurons to fit the data shown in Fig. 10 with mean squared error. The training with or without dropout under the same initialization can well fit the training data. In order to clearly see the condensation effect of dropout, we take the initial parameter distribution in the linear regime , θ ∼ N 0, 1 m 0.2 , where m = 1000 is the network width. When we look into the detail of the fitting, we find that the training without dropout in Fig. 10(a, e) has much more oscillation than the training with dropout in Fig. 10(b, f). To better understand the underlying effect of dropout, we look into the weight condensation.
Due to the homogeneity of the ReLU activation, the parameter pair (W is two-dimensional due to the incorporation of bias. Therefore, we use the angle to the x-axis in [−π, π) to indicate the orientation of eachŴ j )} m j=1 is shown in Fig. 10(c, d). Compared with the initial weight distribution (blue dots), the weight trained without dropout is close to its initial value. However, for the training with dropout, the weight after training is significantly different from the initialization. In addition, we find that the weights, which are non-zero after training, tend to condense on two orientations. For generality, we also study the effect of dropout on condensation on three-layer networks, where the dropout layer is between the two hidden layers of the networks. We study the featuers of the first layer which amplitudes are given by A j = W j is the same as above. As shown in the second row of Fig. 10, we can still see that the training with dropout shows a tendency of condensation.
Compared with the linear regime, NNs in the nonlinear regime have the characteristics of feature learning, so they tend to have better generalization. However, during the training of networks in the nonlinear regime, the training speed is often slow due to the fact that the training trajectory is close to the stationary point. This makes networks in the nonlinear regime less used due to speed constraints. Compared with the network in the nonlinear regime, the network trained with dropout regularization in the linear regime also tends to have condensation, and at the same time, it rarely encounters stationary points, which could circumvent the slow training speed.
One may wonder if any noise injected into the training process could lead to the condensation. We also perform similar experiments for SGD. As shown in Fig. 11, no clear condensation emerges even the training has also noise. Therefore, the exeperiments in this section underlies the special characteristics of dropout that can facilitate the condensation.

Conclusion
In this work, we study the implicit regularization of dropout when it is used in the last hidden layer of a neural network. Theoretically, we derive two implicit regularization terms In (c,d,g,h), blue dots and orange dot are for the weight feature distribution at initial and final training stages, respectively. The above line is the result of two-layer networks, with the dropout layer after the hidden layer. The bottom row is the result of three-layer networks, with the dropout layer between the two hidden layers.  and study the relation between the Hessian of loss landscape and the covariance of dropout noise. Numerical experiments are performed to validate the implicit regularization. Due to the close relation between Hessian and covariance, we study the flatness of minima found by dropout training and find that dropout can find flat minima in many practical datasets.
The unique implicit regularization term R 1 in dropout, which is different from SGD, is found to be a key source to improve the generalization and the flatness of solution. We also find that R 1 can facilitate the formation of condensation during the training, which may establish a connection between the flatness and generalization for further study. This work reveals the rich and unique properties of dropout, which is a basis for comprehensive understanding dropout.