{'fraction_non_alphanumeric': 0.03517751616955935, 'fraction_numerical': 0.011493183192964497, 'mean_word_length': 4.449290060851927, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Guidelines for the Regularization of Gammas in Batch Normalization for Deep Residual Networks

L2 regularization for weights in neural networks is widely used as a standard training trick. However, L2 regularization for gamma, a trainable parameter of batch normalization, remains an undiscussed mystery and is applied in different ways depending on the library and practitioner. In this paper, we study whether L2 regularization for gamma is valid. To explore this issue, we consider two approaches: 1) variance control to make the residual network behave like identity mapping and 2) stable optimization through the improvement of effective learning rate. Through two analyses, we specify the desirable and undesirable gamma to apply L2 regularization and propose four guidelines for managing them. In several experiments, we observed the increase and decrease in performance caused by applying L2 regularization to gamma of four categories, which is consistent with our four guidelines. Our proposed guidelines were validated through various tasks and architectures, including variants of residual networks and transformers.


Introduction
Deep neural networks have exhibited remarkable performance in various fields. Previously, large neural networks with deep and wide architectures were considered difficult to train. However, various regularization techniques such as L 2 regularization [26], batch normalization (BN) [12], and residual learning [8] have made it possible to train large neural networks, leading to successful performance.
Since the classic machine learning era, L 2 regularization has been applied as a restriction on the weights W of neural networks and has not been applied to bias b. However, trainable parameters in modern neural networks are not limited to weights W and bias b. BN outputs γ ixi + β i from normalized featurex, where γ and β are also trainable parameters. Because β plays a similar role to bias b, we ignore L 2 regularization of β. However, γ controls the scale of the feature map and can be viewed as a special case of weights (Section 3). Despite the similar roles of γ and weights, L 2 regularization of the γ parameters of BN remains an undiscussed mystery. Moreover, each deep learning library and practice provides a different method: TensorFlow and Keras To implement L 2 regularization, a regularizer option needs to be set for each layer. For example, kernel_regularizer can be applied to the convolution and fully connected layer to implement L 2 regularization on weight W . L 2 regularization of γ is functionally supported through gamma_regularizer, which may be set to the BN layer. In practice, however, gamma_regularizer is rarely used. To our knowledge, in the TensorFlow and Keras official tutorials and code, practical use of gamma_regularizer does not exist. In other words, L 2 regularization in TensorFlow and Keras generally means using kernel_regularizer on weights, and L 2 regularization has not been applied to γ.
PyTorch L 2 regularization is implemented by applying a weight decay at the optimizer level. However, the weight decay of PyTorch is applied to all trainable parameters, including all W , b, γ, and β. Although the validity of weight decay for b and β is also arguable, we focus on whether PyTorch's practice of applying weight decay to γ is valid.
Practice and empirical observation In several work [10,13,6], L 2 regularization was not applied γ and β. However, [22,24] used L 2 regularization on γ and β, while [22] turned off it during fine-tuning. Reference [18] empirically observed that L 2 regularization on γ and β often improves performance depending on the architecture. Because these practices lack theoretical analysis, in this paper, we explore whether it is desirable to apply L 2 regularization to the γ parameter of BN. First, we claim that γ plays a role in controlling variance in residual networks. We show that the variance of the features in the residual block is either accumulated or reset according to the layer arrangement. For better behavior in residual networks, we propose a strategy of making the accumulated variance small and the reset variance large (Section 2). Second, we present an analysis of the effective learning rate from optimization perspective of γ (Section 3). Through our theoretical analysis, we present four guidelines for managing γ ( Table 1).
The validity of the four guidelines is confirmed through several experiments (Section 4). We demonstrate a performance decrease due to incorrect L 2 regularization on γ and a performance increase due to correct L 2 regularization on γ. We observed this phenomenon in various residual networks and transformers.

Variance analysis in residual networks
In this paper, we target residual networks because they are widely used as a standard architecture and involve many variants. In residual networks, an input image is first passed through the early stage. Then, four stages are applied, and each stage is composed of several residual blocks. Each residual block consists of a skip connection and a residual branch, in which several weight, BN, and ReLU layers are placed in the residual branch.
Our assumption is that the condition when the skip connection dominates over the residual branch, which makes the residual block behave more like an identity mapping, is advantageous for residual network training. In fact, in the original ResNet paper, residual learning was introduced with the intention of modelling identity mappings [8]. Reference [6] reported that initializing γ = 0 in the last BN causes the residual block to behave like an identity function and eases optimization. Reference [3] found that a residual block behaves like an identity mapping because the variance of the skip connection is larger than the variance of the residual branch. In their paper, the γ parameter of BN was assumed to be 1, but we show here that variance in the residual block varies according to γ. We consider the bias parameter b in the weight layers and β in the BN layers to be 0. L 2 regularization of b or β is ignored in this paper. In BN, we assume that the mini-batch size is sufficiently large, and each feature map is normalized to N (0, 1) and then rescaled using γ. In other words, the i-th The term "weight layers" used below indicates convolution layers in ResNet, but will be described by generalizing to fully connected layers. We assume that each element of the weights comes from He initialization N (0, 2/f an in ) [7].

Case 1. Basic block in PreActResNet
PreActResNet [9] is a modified version of the original ResNet [8], and is also called ResNetV2. The residual branch of PreActResNet consists of two or three [BN-ReLU-Weight] blocks (Figure 1, Left). The l-th residual block outputs x l+1 , which is the sum of the residual branch f l (x l ) and skip connection x l , i.e., x l+1 = x l + f l (x l ). Because the covariance between the residual branch and the skip connection is zero [3], we have V ar[ Here, we examine the variance of the residual branch V ar[f l (x l )]. Consider the residual branch with two [BN-ReLU-Weight] blocks. Because of the normalization in the second BN, the γ of the first BN and W 1 do not affect the variance of the residual branch. Therefore, the variance of the residual branch is determined by the second [BN-ReLU-Weight] block. By the mean and variance of W from He initialization and BN, the variance of the i-th element of the residual branch is (1) The third equality holds because Starting from the s-th residual block, imagine that the variance of the residual branch is accumulated in a row. The variance of the l-th residual block is where x s denotes the first input feature map for each stage of ResNet. As such, in the residual block, the variance of the residual branch is accumulated. Here, we want to make the residual block satisfy V ar[x l,i ] > V ar[f l,i (x l )] so that the skip connection dominates over the residual branch. From Eq. 4 and Eq. 6, the inequality V ar[x s,i ] + l−1 m=s γ 2 2,m,i > γ 2 2,l,i requires two conditions: The variance at the stage starting point in the second condition is discussed later in Cases 3 and 4. According to the first condition, γ in the second BN in all residual blocks should be small to reduce the variance of the residual branch. Similarly, when the residual branch consists of three [BN-ReLU-Weight] blocks, the γ in the third BN should be small. Thus, in order for the skip connection to dominate over the residual branch, we should control γ in the last BN in the residual branch. To ensure that those called γ last are small during training, we recommend applying L 2 regularization on γ last . In summary, we present the following guideline: Guideline 1. Because residual block plays the role of variance accumulation, we should decay γ last .

Case 2. Basic block in the original ResNet
We check whether Guideline 1 is also applicable to the original ResNet. The original ResNet, also called ResNetV1, constitutes a residual branch with two or three [Weight-BN-ReLU] blocks, but the last ReLU is deployed after addition (Figure 1, Right). First, we examine the variance of residual branch V ar[f l (x l )]. Because the residual branch ends with BN, the variance of the residual branch is equal to the variance of the last BN, i.e., V ar[f l,i (x l )] = γ 2 last,l,i . Contrary to PreActResNet, here, because ReLU is applied after addition, This ReLU after addition makes a different variance accumulation.
Thus, the added variance is halved. Here, we introduce the following substitutions: a l = 2 l V ar[x l,i ] and b l = 2 l γ 2 last,l,i . Rewriting Eq. 10, we have a l+1 = a l + b l , and thus a l = a s + As such, the variance of the residual branch is accumulated here as well, but because the variance of the last ReLU is halved, the variance of the previous block decays. We call this half variance accumulation. For the skip connection to dominate, the condition V ar[x l,i ] > V ar[f l,i (x l )] requires that 1) γ 2 last,l,i should be small and 2) V ar[x s,i ] should be large. Therefore, it is desirable to decay γ last , and thus, Guideline 1 is valid for the original ResNet as well.

Case 3. Downsampling block in ResNet
Convolutional neural networks downsample high-dimensional image features to low-dimensional ones using pooling layers [14]. For ResNet, downsampling is performed using strided convolution [17] as well as pooling. For the original ResNet, at the end of each stage, a downsampling block (Figure 2, Right) is applied, where the strided convolution is used in both the first weight layers in the residual branch and the skip path. Here, the skip path of the downsampling block is composed of strided convolution and BN rather than identity. We denote the skip path output as g l (x l ) = B down,l (W down,l x l ). Because each branch ends with BN, the variance of each branch is determined by the last BN as follows: V ar[f l,i (x l )] = γ 2 last,l,i .
Note that the output variance of the downsampling block is not affected by the input variance and is newly determined by γ down and γ last . We call this variance reset.
Again, it is desirable to decay γ last so that the residual branch does not dominate. However, it is not desirable to decay γ down . There are two reasons for this. First, in order for the skip path in the downsampling block to dominate, we should decay γ last and should not decay γ down . Second, it is desirable to have a large reset variance. The output variance of the downsampling block becomes the input variance of the successive (l + 1)-th residual block, which is the variance at the stage starting point. In Cases 1 and 2, we confirmed that a large V ar[x s,i ] is favorable. Thus, a large output variance of the downsampling block is desirable. In other words, if we decay both γ down and γ last , the reset variance decreases, which is undesirable. Therefore, we claim that γ down should not be subjected to L 2 regularization to ensure that the reset variance appears dominant afterward. In this regard, we present the following guideline: Guideline 2. Because the downsampling block plays the role of variance reset, we should not decay γ down .
Thus, the downsampling block of PreActResNet plays the role of variance reset as well. To ensure that the reset variance is dominant afterward, we should not decay γ 1 at the first BN. For convenience, we also call this parameter γ down .

Case 4. Early stage in ResNet
The input variance of the first residual block is determined by the early stage of ResNet before the residual block starts. The early stage of ResNet, which is also called the stem, consists of convolution, BN, ReLU, and pooling layers without a skip connection in the original ResNet (Figure 4, Right). Ignoring pooling, from the output of early stage Therefore, γ 0 sets the output variance of the early stage. The output of the early stage becomes the input of the residual block of the first stage. Because a large variance at the stage starting point is favorable, γ 0 should be large. In this regard, we present the following guideline: Guideline 3. Because the early stage plays the role of variance set, we should not decay γ 0 .
In case of PreActResNet, BN is not applied in the early stage (Figure 4, Left). Thus, we do not have to consider γ 0 .
Finally, in the residual branch, additional γ parameters exist in the BN before the last BN for both PreActResNet and the original ResNet. We call those parameters γ others . Because they do not affect the variance of the residual branches, our variance analysis cannot explain their role. The γ others parameters are discussed again in Section 3; here, we simply present the conclusion: Guideline 4. Though other BNs do not determine the variance, L 2 regularization on γ others helps optimization by improving the effective learning rate. Thus, we should decay γ others . Further, using ResNet-50, we measured the variance of each feature map ( Figure 5). First, the variance is (re)set at layer 1-0, and then the variance decreases at layer 1-1, which is expected from the half variance accumulation (Eqs. 10 and 11). In layer 2-0, where the subsequent stage begins, the variance is reset to a new larger value and decreases again from layer 2-1 to layer 2-3. As such, both the variance reset to a larger value and half variance accumulation with decreasing value appear in each residual block, which agrees with our claim.

Case 5. Transformer block
The transformer is attracting attention for its high performance in various tasks, including computer vision [5] and natural language processing [20,4]. The transformer uses a series of transformer blocks, which consists of residual branches with self-attention and multilayer perceptron blocks. In contrast to ResNet, the transformer uses LayerNorm (LN) [1]. However, LN also rescales normalized features using γ and β. Therefore, our variance analysis can explain how the γ parameter of LN affects the variance within the transformer block. See the Appendix for the mathematical details regarding the roles of γ 1 and γ 2 in the first and second LNs. Here, we simply present the following guideline for the transformer block: Guideline 1T. Because a transformer block accumulates the variance, we should decay γ 1 and γ 2 .
Note that the transformer has no downsampling block (and hence there is no Guideline 2T). Further, some transformers such as bidirectional encoder representations from transformers (BERT) [4] have LN in the early stage, whose γ 0 should not be subjected to the L 2 regularization (Guideline 3T).
Finally, some transformers [20] have another LN between the final transformer block and the head layer. The role of the γ other parameters of LN cannot be explained by our variance analysis, but in the same context as Guideline 4, L 2 regularization on γ other helps optimization (Guideline 4T).

Effective learning rate analysis
In this section, we explore why γ others parameters should be decayed even though their scale does not affect the BN output. Reference [19,11,26] argued that in a neural network with BN, the weight scale does not affect the BN output. They introduced the effective learning rate to claim that a smaller weight scale is advantageous for optimization, which can be achieved through L 2 regularization on weights. In our paper, we note that γ is a trainable parameter in a neural network similar to weights. Extending their claim, we investigate the effective learning rate for the optimization of γ.
BN is composed of a normalization step N (·) and linear scaling step L(·). Here, we decompose the diagonal matrix into its norm and direction, i.e., Γ l = Γ l Γ l . Then, Thus, the scale of γ in BN does not affect the intermediate block output. If we investigate the gradient of y(x l ; W l , Γ l ) with respect to Γ l , we obtain This scale-variant gradient property was used in [19,11] to claim that L 2 regularization on weights helps optimization even in neural networks with BN. Thus, effective learning rate analysis is expected to be applicable to γ in BN. From Eq. 22, we have ∇ Γ l L(Γ l ) = 1 Γ l ∇Γ l L(Γ l ) for loss function L. First, gradient descent with respect to Γ l is in the form of Γ l,t+1 = Γ l,t − η∇ Γ l,t L(Γ l,t ).
If we investigate gradient descent with respect toΓ l , we have, Figure 6: Norm of the first update by the initial value of γ.
Here, η Γ l,t −2 is called the effective learning rate [19,11]. If the scale of γ in BN decreases, the effective learning rate increases, which prevents the optimization from saturating due to the large weight norm. Although we followed the derivation in [19], the derivation in [11] shows similar results. Thus, although L 2 regularization on γ others does not affect the variance, it improves the effective learning rate and optimization. In this regard, we advocate applying L 2 regularization to γ others . Now, we empirically validate Eq. 24. Using ResNet-18, we varied the initial value of γ and measured the norm of the first update, Γ l,1 −Γ l,0 ( Figure 6). From their log values, a regression fit yielded a coefficient near −2, which agrees with Γ l,t+1 −Γ l,t ∝ Γ l,t −2 .
Note that this effective learning rate analysis is applicable to the γ parameters of all four categories. Although applying L 2 regularization on γ down and γ 0 is harmful to variance control, it could be advantageous for improving optimization with respect to the effective learning rate.

Image classification tasks
We tested whether Guidelines 1-4 are valid in practical tasks. First, in the training task of ResNet and its variants [8,9,25,23], we measured the performance of applying L 2 regularization to the weights and γ of each group with 1 2 λ( W   Table 2 shows image classification accuracy on the Oxford-IIIT PET dataset [16] for different L 2 regularization setups. First, when the γ last parameters were subjected to L 2 regularization, the test accuracy increased approximately 1%-4% compared to the case where L 2 regularization was not applied to them. This is consistent with Guideline 1, which states that we should decay the γ last . However, when L 2 regularization was applied to γ down , the test accuracy decreased approximately 1%-3%. This is consistent with Guideline 2, which states that L 2 regularization for γ down is undesirable. Similarly, for γ 0 and γ others , the experimental results agree with Guidelines 3 and 4.
Limitations However, for γ 0 , an increase in performance was observed in some models. When the effect of improving the effective learning rate is more dominant than the variance control, it can result in performance improvement. Because the first stage is relatively shallow compared to other stages, the variance from γ 0 would have a minor effect, and improvement of the effective learning rate may become dominant. The coexistence of variance control and improved effective learning rate requires further research. See the Appendix for more results on other datasets, including NABirds, Food-101, and CIFAR-10.

Natural language processing tasks
We also tested whether our proposed guidelines are valid for transformers on natural language processing tasks. First, we visited a machine translation task using a transformer. The BLEU score [15] was measured for the German to English translation task using the IWSLT-14 dataset [2] (Table  3). We measured the performance before and after applying L 2 regularization to γ 1 and γ 2 . For both cases, an increase in the BLEU score was observed. This result is consistent with Guideline 1T, which states that small γ is desirable for transformer blocks because they accumulate variance. Our claim was also verified in the text classification task using BERT for the SST-2 task from GLUE [21]. Following Guideline 1T improved test accuracy here as well (Table 4).

Conclusion
In this paper, we studied L 2 regularization for the γ parameters of BN. We theoretically showed that this γ should be controlled so that the residual blocks behave similarly to identity mapping. Specifically, we discussed the cases in which it is desirable to decay the γ parameters of BN and the cases in which it is not, presenting four guidelines for their management. The proposed guidelines were verified in experiments on several variants of residual networks and transformer models. We confirmed that the performance increases when the guidelines are followed and decreases when the guidelines are violated.