{'fraction_non_alphanumeric': 0.031201774691358024, 'fraction_numerical': 0.012393904320987654, 'mean_word_length': 4.760277777777778, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ProbAct: A Probabilistic Activation Function for Deep Neural Networks

Activation functions play an important role in the training of artificial neural networks and the Rectified Linear Unit (ReLU) has been the mainstream in recent years. Most of the activation functions currently used are deterministic in nature, whose input-output relationship is fixed. In this work, we propose a probabilistic activation function, called ProbAct. The output value of ProbAct is sampled from a normal distribution, with the mean value same as the output of ReLU and with a fixed or trainable variance for each element. In the trainable ProbAct, the variance of the activation distribution is trained through back-propagation. We also show that the stochastic perturbation through ProbAct is a viable generalization technique that can prevent overfitting. In our experiments, we demonstrate that when using ProbAct, it is possible to boost the image classification performance on CIFAR-10, CIFAR-100, and STL-10 datasets.


Introduction
Activation functions add a non-linearity to neural networks and thus the ability to learn complex functional mappings from data [1]. Different activation functions with different characteristics have been proposed thus far. Sigmoid [2] and hyperbolic tangent (Tanh) were especially popular during the early usage of neural networks [3]. Those functions were used due to their monotonicity, continuity, and bounded properties. However, their derivatives become small when the absolute value of the input becomes large, which leads to the vanishing gradient problem when using Stochastic Gradient Descent (SGD).
In recent times, the Rectified Linear Unit (ReLU) [4] has become an extremely popular activation function for neural networks. ReLU is defined as: and the derivative is: ReLU solves the vanishing gradient problem with an identity derivative. However, a zero value for all negative values creates the dead neurons problem. To counter the problem, several variants of ReLU have been proposed. For example, we can find simple extensions of ReLU, such as Leaky ReLU [5], Parametric ReLU (PReLU) [6], and Exponential Linear Unit (ELU) [7].
In this work, we propose a new activation function, called ProbAct, which is not only trainable but also stochastic. In fact, the idea of ProbAct is inspired by the stochastic behaviour of the biological neurons. Noise in neuronal spikes can arise due to uncertain bio-mechanical effects [8]. We try to emulate a similar behaviour in the information flow to the neurons by injecting stochastic sampling from a normal distribution to the activations. Simply speaking, even for the same input value x, the output value from the ProbAct varies by stochastic perturbation -this is the largest difference from conventional activation functions.
The induced perturbations prove to be effective in avoiding network overfitting during training. The operation has a resemblance to data augmentation, and hence we call it augmentation-by-activation. Furthermore, we show that ProbAct increases the classification accuracy by +2-3% compared to ReLU or conventional activation functions on established image datasets. The main contributions of our work are as follows: • We introduce a novel activation function, called ProbAct, whose output undergoes stochastic perturbation.
• We propose a method of governing the stochastic perturbation by using a parameter trained through back-propagation.
• We show that ProbAct improves the performance on various visual object classification tasks.
• We also show that the improvement by ProbAct is realized by the augmentation-by-activation, which acts as a stochastic regularizer to prevent overfitting of the network.
2 Related Work

Activation Functions
Various approaches have been applied to improve activation functions. Research for activation functions can be broadly defined as three trends: fixed activation functions, adaptive activation functions, and activation functions with non-parametric inner structures.
Fixed activation functions are functions that are fixed before training such as sigmoid, tanh, and ReLU. In particular, ReLU has led to significant improvements in neural network researches. However, the morphological characteristic of ReLU activation function is a ramp function and has a zero value for all negative numbers. This can cause neurons to die. Therefore variants of ReLU were suggested to solve this problem. For example, Leaky ReLUs [5] use a fixed 0.01x value for x < 0. Other activation functions like Swish [9] and Exponential Linear Sigmoid SquasHing (ELiSH) [10] take a different approach and use bounded negative regions.
Adaptive activation functions use trainable parameters in order to optimize the activation function. For example, Parametric ReLUs (PReLUs) [6] are similar to Leaky ReLUs but with a trainable parameter instead of a fixed value. In addition, S-shape ReLU (SReLU) [11], Parametric ELU (PELU) [12] were suggested to improve the performance of conventional ReLU functions.
Non-parametric activation functions were suggested to further increase the flexibility of the activation functions. The Maxout activation function was suggested by Goodfellow et al. [13] and uses a piece-wise linear approximation arbitrary convex functions through using the affine transform and the selecting maximum values part.
Also, kernel-based non-parametric activation functions for neural networks (Kafnets) was suggested by Scardapane et al. [14]. Kafnets allow for the non-linear combination of information from various paths in the network. In another work, the activation ensemble method was suggested by Harmon and Klabjan [15].

Generalization and Stochastic Methods
There has been less research on stochastic activation functions due to expensive sampling processes. Noisy activation functions [16] tried to deal with these problems by adding noises to the non-linearity in proportion to the magnitude of saturation of the non-linearity. RReLU [5] uses Leaky ReLUs with randomized slopes during training and a fixed slope during testing.
There are many other generalization techniques which use random distributions or stochastic functions. For example, dropout [17] generalizes by removing connections at random during training. Dropout can be interpreted as a way of model averaging. Liu et al. suggested Random Self-Ensemble (RSE) by combining randomness and ensemble learning [18]. Inayoshi and Kurita proposed back-propagating noise to the hidden layers [19]. Furthermore, the effects of adding noise to inputs [20,21], the gradient [22,23], and weights [24,21,25,26] have been studied. However, we adopt the concept that stochastic neurons with sparse representations allow internal regularization as shown by Bengio [27].
Introducing Bayesian inference on weights of a network [28] introduces regularization effects. However, the exact Bayesian form of the network is intractable and [29][30] [31] uses Variational Inference to approximate the posterior distribution. [32] showed that this approach attains performance similar to Dropout.
ProbAct can be thought of a way to add noise to the network. The difference from other noise injection methods is that we propose an adaptable and trainable variance that is injected to every layer. To the best of our knowledge, our proposed method is the first approach that adopts stochastic noise into activation function.

ProbAct: A Stochastic Activation Function
In this section, we define the proposed method, ProbAct. In general, each layer of neural networks computes its output y for the given input x: where w is the weight vector of the layer and f (·) is the activation function, such as ReLU. As shown in Figure 1, we introduce a random variable to ReLU to create a stochastic activation function. Specifically, ProbAct is defined as: where µ(x) is ReLU (i.e., µ(x) = max(0, x)) and the perturbation term z is: The perturbation parameter σ is a fixed or trainable value which specifies the range of stochastic perturbation and is a random value sampled from a normal distribution N (0, 1). The value of σ is either determined manually or trained along with other network parameters (i.e., weights) with simple implementation. If σ → 0, ProbAct behaves same as ReLU. This means that ProbAct is a generalization of ReLU.

Setting the Parameter for Stochastic Perturbation
The parameter σ specifies the range of stochastic perturbation. In the following, we will consider two cases of setting σ, fixed and trainable.

Fixed Case
There are several ways to choose the desired sigma. The simplest one is setting σ with a constant value as a hyper-parameter. Choosing one constant σ for all values is a good idea as is randomly sampled from a normal distribution and σ acts as a scaling factor to the sampled value . This can be interpreted as repeated addition of the Gaussian noise to the activation maps, which helps in better optimization of the network [27]. The network is optimized using gradient-based learning as it follows Theorem 1 as stated: The gradient propagation of a stochastic unit h based on a deterministic function g with inputs x (a vector containing outputs from other neurons), internal parameters φ (weights and bias) and noise z is possible, if g(x, φ, z) has non-zero gradients with respect to x and φ. [27] The nature of z = σ is Gaussian as ∼ N (0, 1) is sampled from a Gaussian distribution, and σ being a constant value does not affect the Gaussian properties. This ensures learning using gradientbased methods. The proposed method does not significantly affect the number of parameters in the architecture, so the computational cost is relatively low. However, choosing the best σ is a difficult task as setting any other hyper-parameter for a network.

Trainable Case
Using a trainable σ reduces the requirement to determine σ as a hyper-parameter and allows to learn the appropriate range of sampling. There are two ways of introducing a trainable σ: • Single Trainable σ: A shared trainable σ across the network. This introduces a single extra parameter used for all ProbActs. This is similar to the fixed σ but with the value of trained σ.
• Element-wise Trainable σ: This method uses a different trainable parameter for each ProbAct. This adds the flexibility to learn a different distribution for every neuron.
Training σ The trainable parameter σ is trained using a back-propagation simultaneously with other model parameters. The gradient computation of σ is done using the chain rule. Given an objective function E, the gradient of E with respect to σ l,i , where σ l,i is the perturbation parameter of the i-th unit in the l-th layer, is: where y l,i is the output of the i-th unit in the l-th layer. The term ∂E ∂y l,i is the gradient propagated from the deeper layer. The gradient of the activation is given by: Bounded σ Training σ without any bounds can create perturbations in a highly unpredictable manner when σ → ∞, making the training difficult. Taking the advantages of monotonic nature of the sigmoid function, we bound the upper and lower limit of σ to (0, α) using: where k represents the element-wise learnable parameter, and α and β are scaling parameters that can be set as hyper-parameters. We used α = 2 and β = 5 in the experiments. These values were found through exploratory testing. Figure 1 (c) illustrates the effect of stochastic perturbation by ProbAct in the feature space of each neural network layer. Intuitively, ProbAct adds perturbation to each feature vector independently, and this function acts as a regularizer to the network. It should be noted that while the noise added to each ProbAct is isotropic, the noise from early layers is propagated to the subsequent layers; hence, the total noise added to a certain layer depends on the noise and weights of the early layers. For example, even in the simplest case where the network has two layers with a unit for each layer, the distribution of the second layer's output y 2 differs depending on the first and second layer's weights (w 1 and w 2 ) and sigmas (σ 1 and σ 2 ) as:

Stochastic Regularizer
Incidentally, as shown in Figure 1 (c), a small noise variance tends to be learned in the final layer to make the network output stable (see Section 4.4 for a quantitative evaluation).
Using Eq. (6) from Theorem 1, assume g is noise injection function that depends on noise z, and some differentiable transformations d over inputs x and model internal parameters φ. We can derive the output, h as: If we use Eq. (11) for another noise addition methods like dropout [33] or masking the noise in denoising auto-encoders [34], we can infer z as noise multiplied just after a non-linearity is induced in a neuron. In the case of semantic hashing [35], the noise z is added just before the non-linearity.
In the case of ProbAct, we sample from Gaussian noise and add it while computing h. Or we can say, we add a noise to the pre-activation, which is used as an input to the next layer. In doing so, self regularization behaviour is induced in the network.
Further, the effect of regularization is proportional to the variance of the distribution. A high variance is induced with a higher σ value, allowing sampling from a high variance distribution which is further away from the mean. This way the prediction is not over-reliant on one value, helpful in countering overfitting problem. For the fixed σ case, the variance of the noise is fixed. However, it helps in optimizing the weights of the network. We also show the regularization behaviour empirically in the experiments section.

Experiments
In the experiments, we empirically evaluate ProbAct on image classification tasks to show the effectiveness of induced stochastic perturbations in the output. We also show that ProbAct acts as a regularizer to prevent overfitting experimentally. Our results are available on GitHub. 2

CIFAR-10 Dataset
The CIFAR-10 dataset consists of 60,000 images with 10 classes, with 6,000 images per class, each image 32 by 32 pixels. The dataset is split into 50,000 training images and 10,000 test images.
CIFAR-100 Dataset CIFAR-100 dataset has 100 classes containing 600 images per class. There are 500 training images and 100 test images per class. The resolution of the images is also 32 by 32 pixels.

Experimental Setup
To evaluate the performance of the proposed method on classification, we compare ProbAct to the following activation functions: ReLU, Leaky ReLU [5], PReLU [6], and Swish [9]. We utilize a 16-layer Visual Geometry Group network (VGG-16) [38] architecture. The specific hyper-parameters and training settings are shown in the Supplementary Materials.
For setting an experimental environment, we do not use regularization tricks, pre-training, and data augmentation. The inputs are normalized to [0, 1]. The STL-10 images are resized to 32 by 32 to match the CIFAR datasets to keep a fixed input shape to the network.

Quantitative Evaluation
The results of the experiment on CIFAR-10, CIFAR-100 and STL-10 are shown in Table 1. These results are obtained by averaging the training of the networks three times. When using Element-wise Trainable σ (bound) ProbAct, we achieved performance improvement +2.25% on CIFAR-10, +2.89% on CIFAR-100, and +3.37% on STL-10 compared to the standard ReLU. In addition, the proposed method performed better than any of the evaluated activation functions.
In order to demonstrate the viability of the proposed method, the training and testing times relative to the standard ReLU are also shown in Table 1. The time comparison shows that we can achieve higher performance with only a relatively small time difference.
The training and testing times for ProbAct is comparable to ReLU and other activation functions. This is mainly because the learnable σ values are few compared to the learnable weight values in a network. Hence, there is no significant extra time needed to train these parameters. This shows ProbAct as a strong replacement over popular activation functions.

Parameter Analysis
We visualized training aspects of the Single Trainable σ in Figure 2  with ReLU on the trained network. We confirmed that even with ReLU on the network trained with Single Trainable σ, we could achieve higher results than when training on ReLU. This shows that while training, σ helps to optimize the other learnable weights better than standard ReLU architecture, allowing better model performance. Figure 2 (b) shows the mean Element-wise Trainable σ over 200 epochs for all the layers. We demonstrate the ability of the network to train element-wise σ across all layers, even when the number of trainable parameters is increased due to Element-wise Trainable σ parameters. Figure 3 shows the frequency distribution for the bounded element-wise trained k values after 400 epochs. We observed two peak values for every distribution across all three datasets. We assume that the derivative of a sigmoid function becomes 0 at both the boundaries of the function. The points in the left peak lie in the lower boundary of σ (0 in our case) making ProbAct behave as ReLU. Right peak points lie in the upper boundary of the sigmoid (2 in our case) and take σ as 2. The values in between the peaks signify other σ values.
In the case of both CIFAR datasets, the distribution of parameter k in the last layer is quite narrow and concentrated in the negative domain. As shown in Figure 2 (b), the σ values becomes 0, which indicates that ProbAct conducts the ReLU-like operation.

Overfitting
A high σ value acts an inbuilt regularizer that generalizes better on data, preventing overfitting. To define the level of overfitting in a network, we use a term γ which shows the difference between the  training and test accuracy after training: The idea is that if γ is small then the learning on the training set generalized well to the test set. If γ is large, then the training set was memorized.
In addition, we compare the generalization ability with and without a dropout in Figure 4 for CIFAR-100. Results for CIFAR-10 can be found in the Supplementary Materials. We use a dropout layer before the linear classification layer with dropout probability of 0.5. ReLU without a dropout layer is around γ = 48% and does not change much with the use of a dropout. On the other hand, fixed σ = 1 achieves γ = 43% without a dropout and σ = 2 achieving γ = 39% without a dropout, showing the built-in regularization nature of ProbAct. With the increase in σ value, overfitting can be controlled largely due to a higher variance allowing varied sampling and better model averaging. Moreover, overfitting can further be reduced by introducing a dropout alongside with ProbAct with γ going down to γ = 30% for σ = 1 with a dropout.

Reduced Data
The training data size was reduced to 50% and 25% of the original data size for CIFAR-10 and CIFAR-100 dataset. We maintained the class distribution by randomly choosing 25% and 50% images for each class. The process was repeated three times to create three randomly chosen datasets. We run our experiments on all three datasets and average the results. Table 2 shows the test accuracy for ReLU and ProbAct with Element-wise Trainable σ (bound) on 25% and 50% data size. We achieve +3% average increase in test accuracy when the data size was halved and 2.5% increase when it was further halved. The higher test accuracy of ProbAct shows the applications of ProbAct in real life use cases when the training data size is small.

Conclusion
In this paper, we introduced a novel probabilistic activation function ProbAct, that adds perturbation in the every activation maps, allowing better network generalization. Through the experiments, we verified that the stochastic perturbation prevents the network from memorizing the training samples because of the change in samples every time, resulting in evenly optimized network weights and a more robust network that has a lower generalization error. Furthermore, we confirmed that the augmentation-like operation in ProbAct is very effective for classifying images when the number of images is insufficient to train networks. We also show that ProbAct has the potential to act as a regularizer like a dropout, thus preventing overfitting.
There are some areas that need to be explored more. Choosing the desired hyper-parameter for σ: a fixed value for a fixed σ case or slope value for bounded Element-wise Trainable σ requires several trial and error. Further, we need to explore the σ relationship with learning rate to find better hyper-parameters. With a clear relationship of σ with x, it is possible to learn and optimize σ better, which might further boost the network performance.