{'fraction_non_alphanumeric': 0.026586706646676662, 'fraction_numerical': 0.007646176911544228, 'mean_word_length': 4.702764320319179, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Neural Networks with Activation Networks

This work presents an adaptive activation method for neural networks that exploits the interdependency of features. Each pixel, node, and layer is assigned with a polynomial activation function, whose coefficients are provided by an auxiliary activation network. The activation of a feature depends on the features of neighboring pixels in a convolutional layer and other nodes in a dense layer. The dependency is learned from data by the activation networks. In our experiments, networks with activation networks provide significant performance improvement compared to the baseline networks on which they are built. The proposed method can be used to improve the network performance as an alternative to increasing the number of nodes and layers.


I. INTRODUCTION
N EURAL networks often use a single function as an activation function for all nodes in the networks. The activation of a node output only depends on the intermediate output of the node computed by a dense or convolutional layer. The shape of an activation function determines how the node output is triggered. In general, strong features extracted by a dense or convolutional layer are passed through to the next layer by an activation function. Neural networks with a single activation function have been successfully applied to various problems. However, some studies have used more complicated activation methods to improve the network performance [1], [2], [3], [4].
The activation of features in a biological system can be affected by many factors. For example, features that have similar characteristics with strong energy can inhibit the activation of a certain feature [5]. Features in a sensory input can be affected by the sensors at neighboring locations [6], [7]. Overt and covert attention can encourage or discourage the activation in certain regions [8], [9]. Neural networks that mimic the activation behavior of biological systems have been studied. In [10], [11], [12], the lateral inhibition model is introduced to neural networks. In [10], [11], the effect of the excitation of features at spatially coupled locations on the excitation of a feature in time is studied. By assuming that lateral inhibition instantaneously occurs, simpler lateral inhibition networks without time dependency can be considered. In [12], a box filter is applied to the intermediate outputs of a convolutional layer. The activation at the center pixel location in a convolutional layer is inhibited by strong intermediate outputs of neighboring pixels computed by a box filter. In [13], [14], the attention model is introduced to recurrent networks. In [13], the saliency of features are found in an input image, and a neural network is designed to weigh the features based on the saliency. In [14], the activation of a node in a recurrent network is modulated by a variable representing attention. The recurrent network shifts its attention to certain regions as it makes decisions.
This work presents an activation method for neural networks that exploits the interdependency of features. In particular, the shapes of the activation functions for each pixel, node, and layer and the dependency of the activation on other features are learned by an auxiliary activation network. A polynomial function is used as an activation function, whose coefficients are determined by an activation network. Each layer in a deep network is accommodated with the activation network. The network learns to adaptively use different activations for each pixel, node, and layer for a given input depending on the features of the neighboring pixels in a convolutional layer and other nodes in a dense layer. Unlike the lateral inhibition and attention models, where the interdependency of features is predefined based on the domain knowledge, the proposed method learns the dependency from the data.
The proposed activation networks are applied to deep convolutional neural networks (CNNs). We prepared deep networks for object recognition and denoising for analysis. The addition of activation networks increases the number of parameters in a network. However, the networks with activation networks were trained faster with lower training losses. The networks with activation networks outperformed both the baseline networks on which they were built and deeper and wider versions of the networks with higher computational complexity. The node outputs of the proposed networks show that the activation networks enable a network to utilize high-level features even from the first layers. This aspect contrasts the usual operations of deep networks, where deeper layers utilize high-level features that are built up from low-level features of shallower layers. We also compared the performance of the proposed networks to various activation methods. In our experiments, providing the adaptive activation that exploits the interdependency of features with activation networks significantly improved the network performance at modest increases of computational complexity. The proposed method can be used to improve the network performance as an alternative to increasing the number of nodes and layers.
The remainder of this paper is organized as follows. The use of activation networks is introduced in Section II-A. The proposed method is compared to various adaptive activation methods in Section II-B. The performance of the proposed networks is evaluated with deep CNNs for object recognition and denoising in Section III. Section IV concludes the paper.

II. NEURAL NETWORKS WITH ACTIVATION NETWORKS A. Activation Networks with Polynomial Activation Functions
This work considers a general form of activation, where a network learns to use different shapes of activation functions for the pixels, nodes, and layers depending on other features of neighboring pixels in a convolutional layer and other nodes in a dense layer. Consider a network whose operations of the lth layer are given as follows. For a dense layer, an intermediate output of the ith node u l i is obtained by for i = 1, 2, · · · , n l , where x l−1 j is the output of the jth node in the (l − 1)th layer, and W l ij is the weight. A set of polynomial coefficients is obtained by an auxiliary network: for k = 0, 1, · · · , K, where V l ij and b l i are the weights and a bias for a node that provides the kth-order coefficient of the ith node a l ki , respectively. The intermediate output is activated by a Kth-order polynomial function as follows: For a convolutional layer, an intermediate output of the ith node u l i is obtained by for i = 1, 2, · · · , n 1 , where w l i is the filters, and x l−1 is the set of node outputs of the (l − 1)th layer. A set of polynomial coefficients is obtained by for k = 0, 1, · · · , K, where v l ki and b l ki are a set of filters and bias for a node that provides the kth-order coefficients a l ki , respectively. The intermediate output is activated by where (ξ, ζ) are pixel indices. The layer schematics of the proposed architecture are shown in Fig. 1 The activation function for the ith node of a dense layer and the (ξ, ζ)th pixel of a convolutional layer is the Kth-order polynomial function. The shapes of the activation functions are determined by the coefficients provided by the auxiliary dense and convolutional networks in (2) and (5), respectively. We name the auxiliary network an activation network. The activation network provides different shapes of polynomial activation functions for each pixel, node, and layer. The activation depends on the features of other nodes in a dense layer and neighboring pixels in a convolutional layer. The dependency is learned by the activation network from the data during the training. The activation networks are simultaneously trained with the original network. The weights of the original network are updated by for a dense or convolutional layer, where E is the cost function for the training. The weights and biases of the activation networks are simultaneously updated by for a dense or convolutional activation network, resepectively, and using the same cost function E.
The activation networks includes the use of usual activation functions. If V l kj 's or v l ki 's are set to zero for all k and j, and b l ki 's are set to predefined constants in (2) or (5), one can use a well-known activation function. For example, b l ki 's can be set to approximate the sigmoid or hyperbolic tangent functions, whose Talylor expansions are given by If b l ki 's are set to the same constants for all nodes, one can use a single activation function in a network. Hence, the network with activation networks includes networks that use a single activation function.

B. Comparisons to Other Activation Methods
Even though neural networks with the same activation function for all nodes have been used successfully for many applications, some studies have used variations of the activation functions to improve the performance. In [1], [2], multiple activation functions, φ k 's, are applied to the intermediate output of a node, and their linear combination is used as an activation, i.e., The coefficients for the linear combination, α l k 's, are learned from the data during the training of a network. The network uses a different activation function for each layer. In [15], a sigmoid activation function is fed with a noisy intermediate output, where n is the noise. The result is to randomly change the slope of the activation function for a given input. Hence, it has an effect of using a different activation function for each node. Compared to these approaches, the proposed method uses more general shapes of activation functions, and the variations of the activation functions are not limited to being node-wise and can be extended to being pixel-wise.
In [3], [4], a polynomial function is used as an activation function for the node-wise variant activation. Once the shapes of the polynomial activation functions are learned during the training, the shapes remain fixed during the usage of the network. In contrast, we train the activation networks that provide different shapes of activation functions adaptively when the input changes.
In [12], the lateral inhibition by neighboring nodes is simulated by shifting the bias of an activation function by The quantity ι l i is the output of the so-called receptor field given by where u l 's are the intermediate outputs of the nodes and N (i) is the set of neighboring nodes of the ith node. In this approach, how other node outputs affect the activation of a node is predetermined by the design of the activation function and receptive fields. In contrast, we allow a network to learn how other node outputs should affect the activation of a node from data during training.
In [14], the attention model is used to enable a recurrent network to pay attention to particular features as time proceeds. The activation function in the last layer is modulated by the attention, or where α(u l i ) represents the attention of a network. The attention is applied in the last layer that makes a final decision. The recurrent network can rely more heavily on some features when the attention shifts in time. In contrast, with the proposed method, the variation of activation is not limited to the modulation but learned for a specific task from the data.

A. LeNet for Object Recognition
To analyze the deep networks with activation networks, we prepare LeNet [16] by adding the proposed activation network for each convolutional and dense layer except for the last layer, which is prepared for object recognition. The order of the polynomial activation function K is determined experimentally and set to 5. The LeNet with the activation network is denoted by LeNet-AN. A baseline LeNet with the ReLU activation function is prepared for comparison. The networks are implemented with Keras deep learning package [17] and trained with the CIFAR-10 dataset [18]. To rule out other factors on the performance, no data augmentation is used for training. Fig. 2 shows the training and validation losses for the baseline LeNet and LeNet-AN. LeNet-AN was trained much faster with smaller training and validation losses. The activation networks provide polynomial coefficients for an activation function of each pixel in a convolution layer and each node in a dense layer. Fig. 4 shows examples of the activation functions used by LeNet-AN for the two input images presented in Fig. 3. The activation functions for the pixels in the first and second convolutional layers and those for the nodes in the first dense layer are shown. LeNet-AN uses different shapes of activation functions for its pixels and nodes. Moreover, LeNet-AN adaptively uses different activation functions for the two different inputs.   The activation network adaptively activates only the features suitable for a given task. The activated node outputs show impressions of object being recognized. The use of activation networks enables a network to analyze and adaptively utilize features related to a given task. Node outputs show higherlevel features earlier from the shallow layers.
The network performance is reported in Table I. LeNet-AN provides higher accuracy than the baseline LeNet. However, the use of activation networks increases the number of parameters in a network. As a result, we use a more complex network than the baseline network for a given task. For fair comparison, deeper versions of LeNet with more layers and wider versions of LeNet with more nodes in layers are prepared. The networks are trained using the same training set. The test accuracy and number of parameters of the baseline, deeper, and wider LeNets are reported in Table I. The number of parameters of LeNet-AN is only 121.5% of that of the baseline LeNet. However, LeNet-AN provides higher accuracy than the more complex deeper versions of LeNets. The wider versions 1 and 2 of LeNet provide higher accuracy, but their complexity is 392.5% and 877.4% of the baseline LeNet, respectively. We prepare a deeper version of LeNet with activation networks. The number of parameters of the deeper LeNet-AN is only 143.6% of that of the baseline LeNet. However, it provides higher accuracy than the much more complex wider versions of LeNets.  We compared the network performance to other activation methods. For comparison, the polynomial activation with polynomial coefficients learned using the training set [3], [4], the inhibition model that modifies the activation with the sum of neighboring node outputs [12], and the attention models that modulate the activation with the attention learned with a simple network [14] are prepared for LeNet. Fig. 7 shows the training and validation losses for the networks with different activation methods. The proposed network was trained much faster with a lower training loss than the other networks. Table  II shows the performance and number of parameters of the networks. For object recognition, all networks with variant activation methods provided higher accuracy than the baseline LeNet. The proposed method outperformed all methods in providing the highest accuracy at a modest increase in complexity.

B. U-net for Denoising
We prepared U-net [19] by adding the proposed activation network for each convolutional layer for denoising. A baseline U-net with the ReLU activation function for all nodes was also prepared for comparison. The U-net with activation network is denoted by U-net-AN. The variance of noise added to the ground truth images to prepare noisy input images is 0.05. The networks were trained using the MNIST dataset [20]. Fig. 8 shows the training and validation losses for the baseline U-net and U-net-AN. U-net-AN was trained much faster with lower training and validation losses.    Fig. 9 shows examples of denoised images. The original and noisy images are shown in Fig. 9 (a) and (b), respectively. The letter zero image denoised by U-net-AN is much closer to the ground truth than the image denoised by the baseline U-net. The letter does not have as many disconnections in the stroke in Fig. 9 (d) as that in Fig. 9 (c). Examples of the features used by the U-nets are shown in Fig. 10. The node outputs of the first and second convolutional layers are shown. The node outputs of the baseline U-net in Fig. 10  We compared the network performance to the polynomial activation, the inhibition model, and the attention model. Fig.  11 shows the training and validation losses for the networks with different activation methods. The proposed network was trained much faster with a lower training loss than the other networks. Table IV shows the performance and number of parameters of the networks. For object recognition, all variant activation methods provided a performance improvement. However, for denoising, only the inhibition model and proposed method provided a performance increase. The use of a learned but fixed polynomial activation or a heavier reliance on some parts with attention did not improve the MSE of the denoised images. The proposed method learned the interdependency features for denoising from data. U-net-AN provided the lowest MSE. Because U-net has more layers for upconversion than a network for a recognition task, the increase in complexity is higher than that observed for LeNet-AN for object recognition.

C. VGG for Object Recognition
In this section, we consider designing a transferred deeper network for object recognition. We prepared a deep network based on VGG16 [16] by adding activation networks. The VGG has cascades of processing blocks that consist of multiple convolutional layers and a pooling layer. We added an activation network after the first convolutional layer of each processing block. The first convolutional layer takes the output of a pooling layer and effectively uses the features with larger supports. By adding an activation network, we can impose how to activate the features based on the features with larger supports. The VGG16-based network with activation networks is denoted as VGG16-AN. For comparison, the baseline VGG16 with the ReLU activation function is also prepared. The networks are trained with the CIFAR10 dataset for object recognition. Fig. 12 shows the training and validation losses for the baseline VGG16 and VGG16-AN. VGG16-AN was trained much faster, with lower training and validation losses.  Table V shows comparisons of the network performance and complexity. VGG16-AN provides a higher accuracy than the baseline VGG16. The computational complexity of VGG16-AN is 120.5% of that of the baseline VGG16. We prepared another network on VGG16 with less computational complexity. All processing blocks consist of two convolutional layers and a pooling layer. The number of nodes in each layer is reduced by half. The narrower and shallower networks are denoted as VGGns and VGGns-AN. Fig. 12 shows the training and validation losses. The network performance and complexity are also shown in Table V. The computational complexity of VGGns-AN is only 21.7% of that of the baseline VGG. However, VGGns-AN outperforms both the VGGns and baseline VGG16. With activation networks to exploit the interdependency on features, the network with shallower layers and smaller nodes outperforms the much deeper network.

IV. CONCLUSION
Deep networks equipped with auxiliary activation networks are presented. The proposed method represents a comprehensive form of activation. Each pixel, node, and layer can be assigned with a different activation function. The activation of a feature depends on other features with the dependency learned from data. In our experiments, providing an adaptive activation that exploits the dependency of features with activation networks improved the network performance significantly at a modest increase in the computational complexity. The proposed method can be used to improve the network performance as an alternative to increasing the number of nodes and layers.