{'fraction_non_alphanumeric': 0.029528227175019798, 'fraction_numerical': 0.006844665686163593, 'mean_word_length': 4.512628624883068, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Visualization Regularizers for Neural Network based Image Recognition

The success of deep neural networks is mostly due their ability to learn meaningful features from the data. Features learned in the hidden layers of deep neural networks trained in computer vision tasks have been shown to be similar to mid-level vision features. We leverage this fact in this work and propose the visualization regularizer for image tasks. The proposed regularization technique enforces smoothness of the features learned by hidden nodes and turns out to be a special case of Tikhonov regularization. We achieve higher classification accuracy as compared to existing regularizers such as the L2 norm regularizer and dropout, on benchmark datasets with no change in the training computational complexity.


Introduction
Regularization is an important aspect of deep neural network training. It prevents over-fitting in the absence of sufficient data. Usually regularizers restrict the norms of the weight parameters. A commonly used regularizer the L2 norm regularizer, penalizes the sum of squares of the weight parameters. Other regularizers include soft-weight sharing, introduced by Nowlan and Hinton [1]. Erhan et. al. showed that layer-wise unsupervised pre-training acts as a regularizer for deep neural networks [2]. Dropout [3] is a more recent regularizer proposed for deep neural networks. There has also been recent works on adaptive dropout [4], which is an improvement over the original dropout. In this paper, we propose a novel regularizer which we call as the visualization regularizer (VR), based on the visual quality of features learned by the internal nodes.
Vision tasks benefit from features consisting of primitives recognized by mid-level vision systems. Deep neural networks have been known to learn hierarchical layers of feature representation [5,6]. On observing the features learned by deep neural networks trained using back propagation, it is seen that in contrast to well defined mid-level features, the node features are often noisy. More meaningful features (smoother features for example) can be favorable to training the network. Our proposed regularizer imposes a constraint on the hidden nodes of a neural network to learn smoother features. Since, the definition of the regularizer depends on the visual property of smoothness, it is only pertinent to domains with a notion of spatial locality, such as images.
We show that the VR regularizer is a special case of Tikhonov regularization [7]. The Tikhonov matrix of the conventional L2 norm regularizer corresponds to an identity matrix multiplied by the regularization weight. Whereas, the Tikhonov matrix for the VR regularizer is more generalised but sparse.
Using our regularizer, we achieve better performance in terms of accuracy than the baselines when compared on benchmark datasets such as MNIST [8] and CIFAR-10 [9]. Moreover the computational complexity of the training algorithm is same as that of the unregularized training algorithm. Our experimental results show that the features features learned by the nodes are smoother than the features learned without the VR regularizer. This paper is organized in five sections. Section II gives a brief introduction to the architecture of deep neural networks and the notation used. The notion of visualization of a node is formally defined in section III. The proposed VR regularizer and the training algorithm along with the complexity analysis are described in section IV. Section V establishes the relationship to Tikhonov regularization. Section VI presents all the experimental results and observations. Finally Section VII concludes with a summary of the achievements and scopes for future work.

Deep neural network architecture
The investigations presented in this paper are based on a muti-class classification setting. The notation followed in this paper is as follows: x denotes the input, W i and b i respectively denote the weights and biases corresponding of the layers, u i denotes the pre-activation of the layers, h i denotes the activation of layers and y denotes the output of the neural network. The subscripts i in the notation denotes the ith hidden layer. The following equations describe a deep neural network with l hidden layers.
where g, is the activation function, a monotonically increasing non-linear function such as sigmoid, tanh or the rectified linear unit.
The loss function used for training is the sum of the classification loss and the regularization term. For a neural network M and a dataset D, the loss function can be written as, where L c (M, D) denotes the classification loss between the output of the neural network and the true class labels, Reg(M ) denotes the regularization term and λ denotes the regularizer weight. However, dropout cannot be included in the loss function. It is incorporated into the training algorithm.

The notion of visualization of a node
Visualization of a node refers to the visualization of the features learned by the node. Visualization of the nodes of a neural network have been studied by Erhan et. al. [10]. They proposed the activation maximization algorithm for visualizing features learned by a node. Following the notion in [10] we define the visualization of a node as follows.
Visualization of a node is defined to be the input pattern(s) which activate(s) the node maximally under the restriction of the L2 norm of the input to be equal to unity. The L2 norm of the input is restricted to unity to prevent the input from becoming unbounded. Formally, the visualization vis(n) of a node n is defined as, vis(n) = argmax where n(x) denotes the activation of node n for input x. Depending on the non-linearity g used in equation 2, solution to the above equation can be unique or multiple. For example, there exists an unique solution for invertible functions like g = σ (sigmoid) or g = tanh. Whereas there can be multiple solutions for non-invertible functions such as g(·) = max(0, ·) also known as the rectified linear unit (ReLU). The visualization of an internal node of the neural network can be computed using gradient-ascent as described in [10] as activation maximization.
Observe that the pre-activation of a node n in the first hidden layer for an input vector x is w x, where w denotes the weights of the connections coming into the node n. The pre-activation is maximized when x is aligned in the same direction as w in the appropriate vector space. Since the activation function is an increasing function, maximization of the pre-activation maximizes the activation too. Consequently, we get the following closed form as one of the possible solutions to equation 6 for nodes n in the first hidden layer.
Note that finding a closed form algebraic expression for the visualization of nodes in the higher hidden layers is difficult due to the non-linearity of the activation function.

Proposed visualization based regularizer
We first define the notion of smoothness of a visualization.

Smoothness of a visualization
Intuitively, one can determine whether an image is smooth or noisy by looking at the gradients in the image. An image is smooth if it has small gradients. The gradient of an image can be computed by convoluting it with a high pass filter. Larger the pixel values in the convolution, the larger the gradients in the original image, and noisier the image. We utilize this intuition to give a formal definition of smoothness.
Consider a convolution I ⊗ K of image I with kernel K, where K is a high pass filter like the laplacian kernel. We define the smoothness of an image to be the negative sum of squares of pixel values of I ⊗ K. Equivalently, we can define the visualization loss VL of image I as, where "·" denotes the element-wise product, also known as the Schur or Hadamard product. The visualization loss is the negative of the smoothness of an image. Lower the visualization loss, smoother is the image. Table 1 shows the visualization loss for some example visualizations.

Visualization loss as a regularizer
Classification tasks using deep neural networks benefit from the high-level of abstractions achieved in the higher layers of the neural network. Deep neural networks are intended to utilize low-level pixels to learn mid-level features and finally high-level features. We propose the visualization regularizer (VR) to constrain the nodes in the first hidden layer to learn features with qualities similar to mid-level visual features. This constraint is intended to facilitate the discovery of high-level abstractions more effectively.
Informally, we define the VR regularizer as a regularizer to reduce the visualization loss of the nodes of the neural network. In other words the VR regularizer makes the nodes learn smooth or less noisy features.
The following sub-sections give a more detailed description of the VR regularizer:

Regularizer expression
Let U (M ) denote the set of nodes of the first hidden layer of a neural network M and for a node n ∈ U (M ), let w n denote the weights of the connections incoming into the node.
From equation 7 we know that visualization of a node in the first hidden layer is proportional to the weights of the connections coming into the node. Hence the visualization loss of the node is proportional to the visualization loss of the weight vector coming into the node. Therefore we can use VL(w n ) as a surrogate for the visualization loss of the visualization of the node n. The difficulty of computing an algebraic expression for the visualization of nodes in higher hidden layers, limits the usage of the surrogate to nodes in the first hidden layer only.
We define the visualization loss VL of a neural network M as, The network training loss function can thus be defined as, where L 2 (M ) denotes the L2 norm regularization term for all weights except the weights coming into the first hidden layer. The remaining notation are as described in section 2.
Note that instead of defining the visualization of loss as the sum of squares as in equation 8, equivalently one can also define it as the mean of squares. The only change required is to divide the sum by the number of elements (a constant) over which the mean is taken. We denote this by VL m (I).

Gradient of the visualization regularizer
For the visualization loss to be used as a regularizer, its gradient must be computed with respect to the model parameters. We derive the expression of the derivative for a general kernel of size (2k + 1) × (2k + 1).
For simplicity in computing the expression of the gradient, we index the elements of the kernel relative to the central element as shown in equation 12. The element at the center is indexed (0, 0). All other elements are indexed according to their position relative to the central element.
Let N K denote the set of indices of the kernel matrix.
Consider an image I with dimensions n, m. Let S(i, j), corresponding to the (i, j)th pixel of I, be defined as follows.
S(i, j) = {(r, (p, q)) | (p, q) = (i, j) + r, r ∈ N K and 0 ≤ p < n, 0 ≤ q < m} (14) Informally, S(i, j) contains the set of valid indices (p, q), along with their position r relative to (i, j), that need to be considered while computing the convolution for the (i, j)th pixel.
A full convolution of the image I can be described as It follows from the definition that a pixel I pq present at a position r, relative to (i, j), has the coefficient a r in I ij .
The visualization loss is The partial derivative of the visualization loss with respect to a pixel I ij of the image I is Observe that in the above equation I ij occurs in I pq only for (p, q) such that (r, (p, q)) ∈ S(i, j). Moreover, if (p, q) is present at position r relative to (i, j), then (i, j) is present at position −r relative to (p, q). It follows that I ij has the coefficient a −r in I pq . Hence the derivative can be computed as, 2I pq a −r .
Further, using equation 15, we can write, where K I denotes the kernel matrix formed by flipping K both horizontally and vertically. Formally, Equation 18 allows us to compute the gradient efficiently and in a scalable manner. All popular GPU programming frameworks provide libraries for scalable convolutions. Figure 1 illustrates the gradient computation for the visualization loss.

Regularized training algorithm
Training requires computing the gradient for the regularized loss function with respect to the parameters of the network. As evident from equation 10  Note that the gradient ∂ ∂w VL(w n ) is zero if w / ∈ w n . In other words, the gradient is zero if w does not belong to the set of weights incoming to node n. Hence we only need to compute the gradients ∂ ∂wn VL(w n ) for all n ∈ U (M ). These can be computed using equation 18. The full algorithm described in figure 2. The algorithm can be extended to using dropout and momentum.
Computing each z n in figure 2 takes O(|w n |). Consequently, computing the gradient of the VR regularizer takes O(|w|) time, which is of the same order as computing the gradient of the L2 norm regularizer. Thus, the VR regularizer does not impose additional overhead in the computational complexity per iteration.

Relationship with Tikhonov regularization
Tikhonov regularization was originally developed for solutions to ill-posed problems [7]. For example, L2 regularization, a special case of Tikhonov regularization is used to compute solutions to regression problems for which rank deficient matrices are encountered while computing their solutions. The solution to regularized least squares regression is given by, where Γ is the Tikhonov matrix. The L2 regularizer corresponds to Γ = I.
Let w be the concatenation of all the weights in w n for n ∈ U (M ). From equation 16, we can see that VL(w n ) is the sum of squared terms of the form w ij The expression of s i consists of only a constant number of non-zero c it . The maximum number of non-zero c it is bounded by the size of the kernel matrix K. Hence, the number of non-zero entries in Γ is O(p · size(K)), whereas the total number of entries in Γ is p|w|. Since size(K) is practically a constant, we can conclude the sparsity of Γ. for minibatch sample D m do 14: Step 1: Compute using back-propagation. 15: Step 2: Gradient for L2 regularizer.

Experiments and observations
We experimented on the MNIST [8] and CIFAR-10 [9] datasets and compared the classification accuracy of our algorithm using the VR regularizer with other regularizers. 1

Experimental setting
We used a neural network with two fully connected hidden layers for both the datasets. For the CIFAR-10 dataset, we additionally used a convolution and max pooling layer before the fully connected layers. For simplicity in searching for optimal parameters, we fixed the number of fully connected hidden layers to two, with the number of nodes in each hidden layer equal to 1000. We also fixed the number of feature maps of the convolution layer to 10, and the kernel size to 3 × 3.
We used the laplacian kernel for the VR regularizer defined as follows.
We used the mean cross-entropy loss over the mini-batch examples as the classification loss, mean of squares of the weight parameters as the L2 regularizer, and VL m (M ) as the VR regularizer. We trained the neural network using stochastic gradient descent as described in figure 2, and also used Nesterov's accelerated gradient (momentum) [11]. Note that the first fully connected layer is connected to the input for MNIST, whereas it is connected to the max-pooling layer in the case of CIFAR-10. Thus for the CIFAR-10 network, U (M ) is defined as the set of nodes in the first fully connected layer. The VR regularizer utilizes this modified definition of U (M ) in the case of CIFAR-10.
We fixed the number of training epochs to 200 and a mini-batch size of 100. For both MNIST and CIFAR-10, we reduced the learning rate by a factor of 10, respectively after every 100 and 70 epochs. We fixed the momentum parameter to 0.9 and dropout rate to 0.3 for both hidden layers.

Accuracy
We compared the different neural networks with combinations of L2, VR, and dropout regularizers. For finding the optimal parameters, we performed a randomized hyper-parameter search with manual fine-tuning. The accuracy and optimal hyper-parameters for various regularizer settings are given in table 2. The parameters µ, λ and α respectively denote the VR regularizer weight, the L2 regularizer weight, and the learning rate.
From the table it is observed that the VR regularizer is an improvement over the standard L2 regularizer. This is observed both for MNIST and CIFAR-10.   Figure 7 shows the visualizations of the internal nodes of the MNIST classifier. The first two rows of the sub-figures correspond to nodes in the first hidden layer, and the last two rows correspond to nodes in the second hidden layer.     The difference between the visualizations of networks trained using the VR regularizer and the ones not trained using it is apparent. The visualizations of architectures using VR regularizer are smoother and less noisy as was expected.

Conclusion and future work
We observe that the VR regularizer is an improvement over the popular L2 regularizer in terms of classification accuracy. It is also observed that the networks trained using a VR regularizer have smoother visualizations. Though the VR regularizer imposes the smoothness constraint only on the first hidden layer nodes, our observations from figure 7 show that the smoothness constraint is propagated to the next layer.
The VR regularizer may be extended to a new class of training algorithms which use domain knowledge in the form of regularizers.