{'fraction_non_alphanumeric': 0.03315994798439532, 'fraction_numerical': 0.01470441132339702, 'mean_word_length': 4.747341189997125, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Normalized Activation Function: Toward Better Convergence

Activation functions are essential for neural networks to introduce non-linearity. A great number of empirical experiments have validated various activation functions, yet theoretical research on activation functions are insufficient. In this work, we study the impact of activation functions on the variance of gradients and propose an approach to normalize activation functions to keep the variance of the gradient same for all layers so that the neural network can achieve better convergence. First, we complement the previous work on the analysis of the variance of gradients where the impact of activation functions are just considered in an idealized initial state which almost cannot be preserved during training and obtained a property that good activation functions should satisfy as possible. Second, we offer an approach to normalize activation functions and testify its effectiveness on prevalent activation functions empirically. And by observing experiments, we discover that the speed of convergence is roughly related to the property we derived in the former part. We run experiments of our normalized activation functions against common activation functions. And the result shows our approach consistently outperforms their unnormalized counterparts. For example, normalized Swish outperforms vanilla Swish by 1.2% on ResNet50 with CIFAR-100 in terms of top-1 accuracy. Our method improves the performance by simply replacing activation functions with their normalized ones in both fully-connected networks and residual networks.


INTRODUCTION
Deep neural networks (Krizhevsky et al., 2012;He et al., 2016;Vaswani et al., 2017;Devlin et al., 2019) have attained great empirical success across computer vision, natural language processing, and speech tasks. It should be partly attributed to the decades of research to understand the difficulty of training a deep neural network and proposed solutions.
Various initialization (Glorot & Bengio, 2010;Saxe et al., 2014;He et al., 2015;Krähenbühl et al., 2016) methods are proposed to help with the convergence of deep models. Xavier initialization (Glorot & Bengio, 2010) was purposed to keep the variance of the gradient for all weight matrices in order to mitigate the problem of vanishing and exploding gradients. Since Xavier initialization is designed for symmetric activation function, its derivation considers the activation function has unit derivative in an idealized initial state. He et al. (2015) has taken account of ReLU in an idealized initial state and gives the specified initialization strategy for neural networks using ReLU and ReLU-like activation functions. However, We would prefer to have a unified method instead of deriving different initializations for different activation functions. More importantly, these works solely consider the initial state of a network and, therefore, the effectiveness may shrink rapidly during training.
Additionally, GPN (Gaussian-Poincaré normalized) function (Lu et al., 2020) is a related work proposed to normalize the norm of the output and the derivative of activation function with the goal of preventing vanishing and exploding gradients. The purpose of their approach is different from ours, to keep the norm of the forward output and the backward pseudo-output same as the norm of the forward input and backward pseudo-input, which is more similar to Xavier initialization. And our approach is applicable in residual networks and can work without the constraint on the variance of the input.
In this work, we analyze the impact of activation functions on gradients and introduce a theoretically sound approach to normalize activation functions. A motivation for our approach comes from dropout (Srivastava et al., 2014) loosely. During the back-propagation, ReLU performs like dropout. Nonetheless, in contrast to ReLU, the outputs of dropout are scaled by a factor to recover the variance, which inspired us. Our contributions can be listed as: • We introduce a unified method to normalize different activation functions without derivation for different initialization. • Our method works well relatively regardless of the change from the initial state due to the normalization factor which is updated dynamically during training. • We investigate its compatibility with BN (Batch Normalization) (Ioffe & Szegedy, 2015) and residual networks (He et al., 2016) and find normalized ReLU and normalized Swish can improve the performances for kinds of networks.

APPROACH
First, We analyze the impact of activation functions on convergence. Then, we demonstrate our approach to normalize activation functions derived from the former part.

THE IMPACT OF ACTIVATION FUNCTIONS ON GRADIENTS
Consider a N -layer network with weight matrices W n , bias vectors b n , activation function δ n , preactivations h n and postactivations x n . Assume x 0 is the input of the network and d n is the input size of layer n. We can say that, h n = W T n x n−1 + b n (2) where the x n−1 is the input of n-th layer and x n is the output of n-th layer.
According to these definitions, we can obtain the equation below in a linear regime, which is similar to the formula in Glorot & Bengio (2010): Then following Xavier initialization we use to constrain weight matrices, we can derive that: Now, let us take the activation function into consideration. Define ρ i , ρ i as: Note that we assume all h n are approximately zero-mean Gaussian. By combining them into 3, we have For the reason that we use Xavier initialization that normalizes the weight matrices, we can loosely simplify 7 into: In order to make the variance of the gradient on each layer approximately same to achieve better convergence, we would like the activation function to satisfy an interesting property as feasible: In fact, this indicates two properties implicitly: is inherent to an activation function, whereas we can make it more satisfy property (i) by normalizing ρ i and ρ i .

APPROACH
In order to normalize ρ i and ρ i , let us apply a normalization factor λ i to the post-activation of the activation function δ i . From equation 8, we would have: From the perspective of forward-propagation, it is expected that: From the perspective of backward-propagation, we would expect to have: Since ρ i and ρ i given an activation function δ i can be calculated based on its input of current batch in the period of forward-propagation, we take the reciprocal of their harmonic mean for λ 2 i as a compromise, of which the strategy is similar to yet slightly different from Xavier Initialization, between preceding two contraints: As we know, the output of each layer can easily keep zero-mean in a linear regime. However, assymmetric activation functions distort the distribution of output from zero mean and the normalization factor further deteriorates the distortion. Additionally, the equation 7 which underlies all of our derivations rests on a fundamental assumption that the weight matrices are zero-mean, that can not be enssured during training due to internal covariate shift (ICS) (Ioffe & Szegedy, 2015). In order to inhibit the distortion, we shift x i−1 , the post-activation of δ i−1 , to zero-mean in order to obtain zero-mean gradient on the weight matrix. And at the same time, the post-activation fixed to zero-mean further stablizes the condition of equation 7 for Of course, it requires the assumption that x i−1 is indenpent of ∂Cost ∂hi . At present, we can give the formula of our approach to normalize ρ and ρ for given activation function δ:δ where µ is the expectation of δ (x); λ denotes the normalization factor; f is a bounded function to adjust λ; α is a learnable parameter. We callδ (x) the normalized activation function. In this paper, we use f (α) = 0.3Tanh (α) and α = 0 as initialization in all experiments. In order to controll the noise, ρ, ρ and µ are updated by a momentum parameter m based on history and current mini-batch. And they are also filtered out abnormal values out of bounds with two hyperparameters L and U . The updating calculation can be described as the following: where t denotes the number of batches (or iterations).
Note that ρ, ρ and µ are obtained without gradient calculation. It means they are treated as three constants during backward-propagation. Therefore, this setting bypasses the problem of blocking the first and second derivatives of the loss that batch normalization operation suffers from (Zhou et al., 2022). The R score of Tanh and Swish is desirably near 0 when the variance of input is small. However, they suffer from pseudo-linearity (Philipp et al., 2018) at same time. The R score of ReLU and LeakyReLU are fixed at -0.383. Data are calculated with Mathematica.
As a result that x can be roughly guarenteed zero-mean as the assumption, we evaluated several prevalent activation functions by analytically calculating R score defined below.
In a normalized activation function, ρ and ρ can be normalized to be approximately around 1. Namely, property (i) is approached. Nonetheless, prevalent activation functions fail to achieve property (ii) as shown in figure 2. You may ask whether we can find a nonlinear activation function of which R converges to 0 and the post-activation is zero-mean. The answer is no. Lu et al. (2020) has proven it when the input is standard Gaussian. We further generalize their conclusion to any zero-mean Gaussian distribution and detailedly prove following proposition in Appendix.

EXPERIMENTS
With several expriments, we validate the effectiveness and advantage of our approach and investigate its compatibility with BN and residual networks. We benchmark NReLU (normalized ReLU) and NSwish (normalized Swish) against common activation functions, especially their unnormalized counterparts, as baseline. For convenience, we call a convolutional/linear layer plus the following BN and activation function a super-layer as a whole in this section.
All activation functions we compare are listed in Appendix.

LENET5
First, We compare NReLU and NSwish against all baseline activation functions on LeNet5 using MNIST as the dataset. We run experiments in 50 epochs 25 times for each activation function and use the same learning rate with SGD. Networks using Tanh, NReLU and NSwish are initialized with Xavier initialization; ReLU, LReLU, ELU (Clevert et al., 2016), SELU (Klambauer et al., 2017) and Swish (Hendrycks & Gimpel, 2016;Ramachandran et al., 2018) with He initialization; ReLU-GPN (Lu et al., 2020) with orthogonal initialization (Saxe et al., 2014). We compare the mean and median of the accuracy and the number of models that fail to reach 98% validation accuracy until the 5-th, 10-th, 15-th, 30-th and 50-th epoch. The parameter µ of all normalized activation functions are disabled in this experiment. Results are shown in Table 1.  Then, we validate how the relation between ρ and ρ affects the convergence. Namely, how activation functions in a network satisfy property (i) influences the speed of convergence to some extent.
We recorded the ρ and ρ during training and compare the score defined below among different methods.
By comparison, We find that this score is roughly inversely related to the speed of convergence. We plot the accuracy and scores in Figure 3. It embodies the impact of activation functions on convergence we mentioned in Section 2.1. We also monitored the mean of weight matrices. The result in Figure 4 shows our approach can inhibit weight matrices to shift from zero mean.

VGG
We benchmark our method on VGG19 (Simonyan & Zisserman, 2015) with CIFAR-100 dataset (Krizhevsky et al., 2009). We replaced ReLU with all activation functions we compare and train for 200 epochs. Super-layers using Tanh, NReLU, NLReLU and NSwish are initialized with Xavier Initialization; ReLU, LReLU, ELU, SELU and Swish with He Initialization; ReLU-GPN and LReLU-GPN with orthogonal initialization. We follow same learning rate with AdamW (Loshchilov & Hutter, 2019). For networks using normalized activation function, we use their unnormalized version as top three activation functions so that we need not change the model architecture and we remove affine transformation in BN followed by a normalized activation function.   The results in Table 3 show our approach consistently outperforms their unnormalized counterparts, particularly in terms of top-5 accuracy. NSwish outperforms Swish by a 1.7% in terms of top1accuracy.
Our approach makes the gradient on weights less easy to vanish when the network is enough deep so that it improves the trainability of neural networks. The operation of normalized activation function is similar to BN (batch normalization) operation containing a scale and a shift. Intuitively, we wonder whether these similar operations mutually deteriorate effectiveness. In this experiment, we empirically testify that BN works well with normalized activation functions.
We find there are two key points when normalized activation functions are used with BN in this experiment.
• If we use BN right before normalized activation function, BN without affine transformation will be preferable, whereas removing affine transformation from the super-layer using unnormalized activation function slightly degenerates the performance. We consider the reason is the affine transformation impairs the effort that BN tries to keep the zero-mean assumption and to stablize the variance of pre-activation and thus makes the normalization factor stable.
• It is suggested to use a BN layer between the highest layer using normalized activation function and the top layer as a buffer, typically when normalized activation function is used with BN in lower layers. The reason we consider is normalized activation function tries to keep the variance of output as the input's which is not necessarily same to the variance of target distribution. This buffer BN can prevent lower layers to raise the variance of output due to the increasing distance among classes during training and thus the moderate change of the variance gives raise to the relatively stable normalization factor.
The first point enables us to use less parameters, however, to achieve better performance. For SELU, ReLU-GPN and LReLU-GPN, we run experiments using learning rate 0.01 with SGD and decay the learning rate by a factor of 0.2 every 60 epochs since the original optimizer did not converge. Given that SELU, ReLU-GPN and LReLU-GPN are proposed to used withou BN, we remove all BN layers in VGG and compare them with normalized activation functions working without BN. We additionally tested the NLReLU (normalized LeakyReLU) for comparing with LReLU-GPN.

RESNET
We also compared normalized activation function to other methods on ResNet with CIFAR-100 and Tiny ImageNet -a subset of ImageNet (Russakovsky et al., 2015). Due to the compuational limitation, we choose the ResNet50 as the backbone. We follow the same set up and train for a fixed number of epochs.
For normalized activation function, we replaced all activation functions which is not following the element-wise addition in the bottleneck with their normalized version, since the shift operation will decrease the ratio of positive elements in output and the viariance of output will grow exponentially as the information flows to the deep block due to the scale operation, which will lead to extremely unstable normalzation factor, if we use normalized activation function right after the element-wise addition. Unnormalized activation function lowers the variance of the output raised by the elementwise addition, whereas normalized activation function keeps the variance and thus raise it exponen-tially as residual networks can be unraveled as hierarchical emsembles of relatively shallow networks (Veit et al., 2016).
We show the block for normalized activation function and unnormalized version in Appendix.   As the result shown in Table 4 and 5, NSwish outperforms Swish by nontrivial 1.4% at least on Tiny ImageNet and 1.2% on CIFAR-100. And NReLU outperforms ReLU by 1.1% at least on Tiny ImageNet and 1.0% on CIFAR-100. At the same time, the total number of trainable parameters for unnormalized activation function is 23705252, whereas the number for normalized version is 23690053 which is 15199 less than the unnormalized ones.

NATURAL LANGUAGE PROCESSING
We benchmark our approach on IMDB dataset (Maas et al., 2011) using TextRCNN (Lai et al., 2015). We train each network 3 times with same set up from scratch, and the median validation accuracy is 90.2% for NReLU against 89.0% for ReLU, 90.1% for NSwish against 88.9% for Swish. The best one is NReLU among compared activation functions. Note that, in this experiment we masked and ignored all padding tokens when calculating the mean and the variance.

DISCUSSION
We have three tips for using normalized activation function with BN and in residual networks, which we has talked detailedly in section 3.
• If we use BN right before normalized activation function, BN without affine transformation will be preferable. • We suggest to use a BN layer between the highest layer using normalized activation function and the top layer as a buffer when normalized activation function is used with BN in lower layers. • When normalized activation function is applied in residual networks, we suggest to not to replace the unnormalized activation functions right after the element-wise addition in each block.
We plan to further validate our approach on masked language modeling task, and will update our paper after finishing the experiments.

CONCLUSION
We propose a theoretically sound approach to normalize activation function. And across several experiments, we empirically conclude that NReLU and NSwish consistently surpass the accuracy of their unnormalized version and at least one of them can outperforms other activation functions, even with less parameters if BN is applied.
• Swish: Before we introduce normalized activation functions, let us predefine the following in order to avoid duplication: where x denotes the preactivation; y denotes the output of the unnormalized activation function. The definition of ρ is dependent on the unnormalized activation function and we define them differently in following parts.
• NReLU(Normalized ReLU) is defined as f (x i ) = (λ + βTanh(α)) (y i −ȳ) (33) Note that, the elements of the derivative of ReLU can be thought as a random variable following Bernoulli(p) where p is the ratio of positive elements in x. We take advantage of this property when talking about it working with residual connection. The gradient of α can be easily derived from the chain rule. ∂Loss where i runs over all position of the feature map. We also need compute the gradient with respect to the input feature map during training as following • NSwish(Normalized Swish) is defined as f (x i ) = (λ + βTanh(α)) (y i −ȳ) (38) For reason that Swish is a ReLU-like activatio function, we also can roughly deem the elements of its derivative as a random variable following Bernoulli(p). The gradient w.r.t. the parameter and the input feature map during training can be derived as: ∂Loss ∂x i = (λ + βTanh(α)) (y + xSigmoid(x) (1 − y)) Proof. Let Hermite polynomials of k degree be: Then we can derive that Since E x∼N (0,σ) δ (x) 2 < ∞; E x∼N (0,σ) δ (x) 2 < ∞ and δ (x) and δ (x) can be expanded in terms of Hermite polynomials, we have Due to E x∼N (0,σ) [δ (x)] = 0, we have According to Equation 44, 45 and we have Thus we can derive that For the reason that all terms in ∞ k=2 (k − 1)σ −2(k+1) a 2 k is nonnegative, the only solution is a k = 0 for all k ≥ 2. And for E δ (x) 2 = σ −4 a 2 1 = C, we have a 1 = ±σ 2 √ C. Hence δ(x) = ± √ Cx.
This proof is largely based on Lu et al. (2020) with minor generalization here.
A.3 BLOCK FOR RESNET50 Figure 6: Left: a block for normalized activation function.Right: a block for unnormalized activation function.