{'fraction_non_alphanumeric': 0.037732646942354474, 'fraction_numerical': 0.01702777620636986, 'mean_word_length': 4.5766561514195585, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets

This work introduces a novel activation unit that can be efficiently employed in deep neural nets (DNNs) and performs significantly better than the traditional Rectified Linear Units (ReLU). The function developed is a two parameter version of the specialized Richard's Curve and we call it Adaptive Richard's Curve weighted Activation (ARiA). This function is non-monotonous, analogous to the newly introduced Swish, however allows a precise control over its non-monotonous convexity by varying the hyper-parameters. We first demonstrate the mathematical significance of the two parameter ARiA followed by its application to benchmark problems such as MNIST, CIFAR-10 and CIFAR-100, where we compare the performance with ReLU and Swish units. Our results illustrate a significantly superior performance on all these datasets, making ARiA a potential replacement for ReLU and other activations in DNNs.


Introduction
The choice of activation function f (.) in deep neural nets is critical as it not only controls the firing of the neurons but also influences the training dynamics as well as directly affects the prediction accuracies [2]. To this date, rectified linear units (ReLU) [15] defined by f (x) = x + = max(0, x), have been a popular choice, especially in convolutional neural nets (CNNs) because of their simplicity, monotonicity and gradient preserving property as opposed to Tanh and/or Sigmoid activations that are cursed by the vanishing gradient problem [8]. However, ReLU activations profoundly suffer from the dying neuron problem as significant percent of the neurons get permanently deactivated if the pre-activation is ≤ 0 as ReLU regularizes the gradient to zero. To resolve this problem, numerous modifications to ReLU have been proposed and include the leaky ReLU [14], Max-out [9], ELU [4] [19] etc. These variations, albeit, are non-zero for x < 0, are monotonous and are unbounded to the region of significance, culminating into inconsistencies in their performance. Google Brain recently introduced a smooth and non-monotonous function based off the sigmoid function named Swish [17] defined by x * σ(β, x) . It uses a single hyper-parameter β which not only controls the slope of the function but also the rate at which the first derivative asymptotes to 0 and 1. The non-monotonous curvature present in the region of significance of Swish can be controlled by varying the value of β through which the network can be optimized. Here, we define the region of significance as the small area around the origin on x-axis [17][8] [6]. Consistent superior performance of Swish has been demonstrated on a number of benchmark classification problems as shown in Ramachandran et.al. [17]. Despite such remarkable performance, Swish is limited in its application as the hyper-parameter β cannot facilitate precise control over the non-monotonous convexity of the function as a small change in β affects the Swish curve in both the positive and negative regions. To this end, we introduce a novel activation function defined by x * σ(L) which is a function derived from the Richard's Curve and we call it as Adaptive Richard's Curve weighted Activation (ARiA). In this work, we focus primarily on a two parameter adaptation of ARiA that facilitates accurate control over the non-monotonous convexity of the function, independently, in both the first and third quadrants. We first discuss the mathematical premise for ARiA, its two parameter adaptation (ARiA2) and subsequently validate its performance on the standard benchmark problems namely MNIST, CIFAR-10 and CIFAR-100 data sets where we employ distinct neural net architectures. We compare performance of ARiA2 with Swish and ReLU activations for these datasets and demonstrate that ARiA not only exhibits faster learning but also outperforms its counterparts.

Methods
In this section, we introduce our novel activation function named 'ARiA' and demonstrate its mathematical relevance in comparison to other activation functions that include Swish and ReLU. We begin with recapitulating ReLU and Swish in brief and then describe the new function.

ReLU
ReLU is defined as where, x is the input to the neuron. Conventionally, ReLU has been the most popular activation function due to its gradient preserving property (i.e. having a derivative of 1 for x > 0) as opposed to Tanh which typically suffers from vanishing gradient problem [10]. By definition, ReLU is a monotonous and a smooth function as shown in Fig. 1(a) and (b)). However, for randomly initialized pre-activations, the networks extensively suffers from the dying neuron problem where a high percentage of neurons are deactivated, subsequently deteriorating the network efficiency [4].

Swish
The standard logistic sigmoid function is defined by Swish function is an adaptation of logistic sigmoid function given in Eq.(2) obtained by multiplying Eq.(2)with raw pre-activation. It introduces a hyper-parameter β and is defined by Eq. (3).
Next, we plot the derivatives for the Swish function as shown in Fig. 2(c). The scale of β controls the rate at which the first derivative approaches 0 and 1. Swish differs from ReLU in mainly two aspects: (1) the 'non-monotonic curvature' or convex curvature for x < 0 in third quadrant, mainly in the pre-activation region of the function as shown in Fig. 2(b) and (2) the smoother slopes for the linear part of the derivative plots in the first quadrant. From Fig. 2(c), we can infer that, with change in the value of β, the curvature in the third quadrant can be coarsely controlled, however, this also modulates the slope of the function in the first quadrant. For example, if β is increased, the curvature in the third quadrant reduces, controlling the f (.) for x < 0, however, the slope in the first quadrant increases that may subsequently increase the f (.) value through the multiple layers of the neural net leading to an exploding gradient [16].This limits the application of the Swish function, as it cannot control the function in both quadrants independently.

ARiA
We introduce a novel activation function that is based on a form of Richard's Curve [18] and we abbreviate it as ARiA. Richard's curve is a generalized logistic function that allows more flexibility in its S-shape. The sigmoid or the logistic functions are limited as these are rotationally symmetric around the point of inflection. Therefore, to generalize these for cases where rotational asymmetry is important, Richard's [18] introduced a multi-parameter function as defined in the Eq. (5).
where, A is the lower asymptote; K is the upper asymptote; B is the exponential growth rate; ν > 0 decides the direction of growth while Q is related to the initial value of function σ(L) and C is a constant which typically is chosen as one [7]. Based on this, we define our novel function ARiA, as the product of raw pre-activations and Richard's Curve and is give in Eq.(6) The ARiA as well as Richard's Curve contain five hyper-parameters that affect the shape of the function differently. We demonstrate the importance of each parameter in Fig. 3. Fig. 3(a-h) display Richard's Curve and the ARiA function for change in every parameter respectively while C is retained at 1. In Fig. 3(a & b), we vary parameter Q and observe that it affects only the non-monotonic curvature. However, this curvature does not asymptote to zero rapidly as seen by the orange line at Q = 2 in Fig. 3(b). Next, we study parameter B as shown in Fig. 3(c & d). This parameter is important as it introduces exponential non-linearity to the function (Fig. 3(c))and for ARiA, with increase in B the non-monotonic curvature asymptotes quickly to zero as can be observed from The role of parameter ν, is crucial as lower values of ν < 1, demonstrate a lower gradient in the first quadrant while in the third quadrant these display a similar behavior as B (Fig. 3(f)). Finally, changing K scales the whole curve whereas simultaneously varying both K & A has much complex effect on the shape of both Richard's as well as ARiA as shown in Fig. 3(g & h).
Overall, all the parameters described above, have the capability to control the shape of ARiA and therefore if ARiA is employed as an activation function and is optimized to a particular problem, it it has the potential to boost the network performance. However, optimizing 4 to 5 parameters heavily increases the number of operations per layer consequently increasing the computational cost. Hence, it is critical to restrict the number of hyper-parameters and we achieve this by selecting the most influential variables in the region of significance. Based on the plots in Fig. 3, these include only two hyper-parameters B (Fig. 3(d)) and ν (Fig. 3(f)), which individually as well as in combination affect the curve in the region of significance and for higher values on both +ve and -ve sides, the curve remains unaltered. This reduces the ARiA function to Eq.(7), where we call these parameters α and β where α = 1 ν and β = B. Henceforth, we call this two-parameter adaptation of ARiA as ARiA2.
f (x) = x * σ(α, β, x) Comparing Eq.(3) of Swish and Eq. (7), it can be observed that by employing two hyper-parameters α and β in combination, ARiA2 allows for a more complex control over the nature and shape of the curve facilitating better adaptation to a particular problem as compared to Swish and this has been described in the following section.

Comparison with Swish
Swish is a special case of ARiA, manifested at ARiA=f (x, 1, 0, 1, 1, β, 1) whereas, ARiA2 is defined at ARiA=f (x, 1, 0, 1, 1, 0, β, 1 α ). Fig. 4(a) shows the effect of varying both α and β on the logistic sigmoid function and Fig. 4(b) shows the corresponding ARiA2 function. By comparing Fig. 2(b) and Fig. 4(b), it can be observed that an additional hyper-parameter facilitates superior control over the shape of the curve subsequently offering higher adaptability to the learning process.  (Fig. 5a), β = 1 (Fig. 5-b), β = 2 (Fig. 5-c), β = 10( Fig. 5-d). The introduction of hyper-parameter α has a two-fold effect; it reduces the curvature in 3rd quadrant as well as increases the curvature in first quadrant while lowering the value of activation, for α > 1. Overall, ARiA2 provides an accurate control over the non-monotonic convex curvature in the 3rd quadrant as shown in Fig. 5. The derivative of ARiA2 is given by: The manipulation of α has more distinct effect than β on the function shape and its derivative as seen in Fig. 6. Fig. 6(a-d) show the first derivative plots for ARiA2 function as displayed in Fig. 5(a-d).
The analysis can be extended further by looking at the plots in Fig. 5 and Fig. 6. Let the activation function be defined as f (x).The function plot can be subdivided into four regions of interest namely, large positive, small positive, small negative, large negative values of x.

Observations
• For small values of x (positive and negative), ARiA2 (and Swish) exhibit a convex upside opening curvature which is completely absent in ReLU (Fig. 1). This lowers the activation value when small negative inputs are present, unlike ReLU for which such inputs do not contribute to the activation value. For small positive weights because of the non-linearity of Swish and ARiA2, the activation values are lower than ReLU. As can be observed from Fig. 5(b), the curve for ARiA2 (α = 0.5, β = 1) has lower convexity in the 1st quadrant and higher convexity in the third quadrant as compared to the curves for α > 1. Hence ARiA2 with α < 1, usually underperforms Swish and ReLU.
• The ARiA2 curves for α > 1, demonstrate a smaller curvature in the third quadrant and higher curvature in the first quadrant, more specifically, in the region of significance in comparison with Swish and ReLU functions. When such a function is employed in DNN architectures, the batch normalization layer [12] scales down the inputs to the next layer where the weights may perhaps are defined in the region of significance.
• Fig. 6 demonstrates that for large negative values, unlike ReLU, there exists a small gradient for ARiA2 (and Swish), while for large positive values the nature of the curve is similar to ReLU. The derivative of ARiA2 and Swish is continuous which may offer small computational advantage over ReLU whose derivative is undefined at 0 and needs to be defined piecewise. 96.54 ARiA2 (α = 1.5, β = 2) 96.7 • As stated earlier, the control on the convexity in the small positive and small negative regions is of utmost importance for the activation function to provide more precision and stability to the model. Therefore, when comparing between ARiA2 and Swish functions, ARiA2 offers this capability via two hyper-parameters while Swish only performs coarse modulation through varying β and this phenomenon can be observed through the derivative plots displayed in Fig. 6 where for (α = 1, β = 1), ARiA2 reduces to the base definition of Swish.

Experiments and Results
The following sections first describe our experiments and the obtained results in greater detail. We have incorporated appropriate architectures pertaining to each of the dataset to illustrate how ARiA2 can perform on a wide range of neural net architectures. It is important to note that due to differences in training setup and computational capacity, our results may not be directly comparable to the results reported in the analogous works.

MNIST
The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems. The MNIST database contains 60,000 training images and 10,000 testing images. We tested the MNIST dataset using two separate architectures; 1)user defined CNN and 2) Dense Convolutional Network (DenseNet). For training of both of these models, AWS P2-xlarge instance with Keras having TensorFlow [1] backend was employed. For the custom CNN, two blocks of convolutional layers were employed. Each block was followed by a pooling layer. The 2 nd pooling layer was followed by a dense layer consisting of 1024 neurons with dropout rate of 0.4, followed by 2 nd dense layer with 10 neurons, one for each target class (ranging from 0 to 9). Adam optimizer [13] with learning rate 0.001 was employed and the model was run for 30 epochs. Table 1 shows the testing accuracy for the MNIST dataset with this CNN model using ReLU, Swish and ARiA2 activation functions.
Next, we tested the MNIST dataset with state-of-art DenseNet to validate its usability in various architectures. It has been shown that DenseNet is an optimal model and requires less computational time to achieve better performance [11]. To train this model we employed Adam Optimizer with learning rate 0.0001 for 25 epochs. 2 cases of ARiA2 demonstrate better testing accuracy as shown in Table 2.

CIFAR-10 and CIFAR-100
The CIFAR-10 is a labeled subset of 60000 32x32 color images of 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images whereas CIFAR-100 is the labeled   [20] with dropouts. We train for 100 epochs with slightly higher initial learning rate of 0.125 with SGD learning algorithm [3] with a scheduled decay of 0.2. the experiments were performed using Keras with TensorFlow backend on an AWS P2-xlarge instance for CIFAR-10 and P2-8xlarge instance for CIFAR-100. Table 3 shows the prediction results using all three activation functions for CIFAR-10 and Table 4 shows the prediction results using all three activation functions for CIFAR-100.
We achieve comparable or better error rate for this model reported in [20], despite training for half the number of epochs by changing the activation function. From Table 4, it can be seen that ARiA2 with α = 1.5, β = 2 demonstrate highest accuracy.

ARiA1
To reduce the computational complexity of optimizing two hyper-parameters in ARiA2, instead, we can employ a simpler version of ARiA with only a single hyper-parameter α (while β is retained at 1). As described earlier in section 3, the effect of varying α is more significant than varying β. This is reflected in our results where, for both models of MNIST, ARiA2 performed better at β = 1 and α > 1. Further, for CIFAR-10 as well, we observe a similar phenomenon where, ARiA2 performs the best for α = 1.5, β = 1. Only for CIFAR-100, the most superior performance is achieved at β = 2. However, at α = 1, β = 1(equivalent to Swish), our model performs better than ReLU. In summary, when higher computational resources are not available, employing ARiA1 is a better choice than employing Swish and/or ReLU and its variants.

Conclusion
In this work, we introduced and implemented a novel activation function, ARiA, and demonstrated its mathematical relevance, its adaptation to ARiA2 and its comparison to ReLU and Swish activations. The ARiA2 function, that is derived from specialized Richard's Curve provides flexibility to the shape of the function via two hyper-parameters, and thus can be tuned to a particular neural net based task. We illustrated its working on three different datasets, where ARiA2 clearly outperformed ReLU and Swish. Moreover, we also demonstrated that ARiA1 could be a potential option, in cases where optimizing two parameters could be computationally challenging. In summary, the precise control over the non-monotonous convexity is key to ARiA's promising behavior and based on our results it holds the potential to replace ReLU, its variants and Swish in a variety of DNN based prediction tasks.