{'fraction_non_alphanumeric': 0.026634382566585957, 'fraction_numerical': 0.011975656043452654, 'mean_word_length': 4.948423511093811, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Neural Networks Regularization Through Class-wise Invariant Representation Learning

Training deep neural networks is known to require a large number of training samples. However, in many applications only few training samples are available. In this work, we tackle the issue of training neural networks for classification task when few training samples are available. We attempt to solve this issue by proposing a new regularization term that constrains the hidden layers of a network to learn class-wise invariant representations. In our regularization framework, learning invariant representations is generalized to the class membership where samples with the same class should have the same representation. Numerical experiments over MNIST and its variants showed that our proposal helps improving the generalization of neural network particularly when trained with few samples. We provide the source code of our framework https://github.com/sbelharbi/learning-class-invariant-features .


Introduction
For a long time, it has been understood in the field of deep learning that building a model by stacking multiple levels of non-linearity is an efficient way to achieve good performance on complicated artificial intelligence tasks such as vision [24,41,43,19] or natural language processing [12,46,22,16]. The rationale behind this statement is the hierarchical learned representations throughout the depth of the network which circumvent the need of extracting handcrafted features.
For many years, the non-convex optimization problem of learning a neural network has prevented going beyond one or two hidden layers. In the last decade, deep learning has seen a breakthrough with efficient training strategies of deeper architectures [21,33,6], and a race toward deeper models has began [24,41,43,19]. This urge to deeper architectures was due to (i) large progress in optimization, (ii) the powerful computation resources brought by GPUs 3 and (iii) the availability of huge datasets such as ImageNet [14] for computer vision problems. However, in real applications, few training samples are usually available which makes the training of deep architectures difficult. Therefore, it becomes necessary to provide new learning schemes for deep networks to perform better using few training samples.
A common strategy to circumvent the lack of annotated data is to exploit extra informations related to the data, the model or the application domain, in order to guide the learning process. This is typically carried out through regularization which can rely for instance on data augmentation, L 2 regularization [44], dropout [42], unsupervised training [21,33,6,36,35,5], shared parameters [26,34,15], etc.
Our research direction in this work is to provide a new regularization framework to guide the training process in a supervised classification context. The framework relies on the exploitation of prior knowledge which has already been used in the literature to train and improve models performance when few training samples are available [29,37,20,31,25,48,47,49].
Indeed, prior knowledge can offer the advantage of more consistency, better generalization and fast convergence using less training data by guiding the learning process [29]. By using prior knowledge about the target function, the learner has a better chance to generalize from sparse data [29,1,2,3]. For instance, in object localization such as part of the face, knowing that the eyes are located above the nose and the mouth can be helpful. One can exploit this prior structure about the data representation: to constrain the model architecture, to guide the learning process, or to post-process the model's decision.
In classification task, although it is difficult to define what makes a representation good, two properties are inherent to the task: Discrimination i.e. representations must allow to separate samples of distinct classes. Invariance i.e. representations must allow to obtain robust decision despite some variations of input samples. Formally, given two samples x 1 and x 2 , a representation function Γ(·) and a decision function Ψ(·); when x 1 ≈ x 2 , we seek invariant representations that provide Γ(x 1 ) ≈ Γ(x 2 ), leading to smooth decision Ψ(Γ(x 1 )) ≈ Ψ(Γ(x 2 )). In this work, we are interested in the invariance aspect of the representations. This definition can be extended to more elaborated transformations such as rotation, scaling, translation, etc. However, in real life there are many other transformations which are difficult to formalize or even enumerate. Therefore, we extend in this work the definition of the invariant representations to the class membership, where samples within the same class should have the same representation. At a representation level, this should generate homogeneous and tighter clusters per class.
In the training of neural networks, while the output layer is guided by the provided target, the hidden layers are left to the effect of the propagated error from the output layer without a specific target. Nevertheless, once the network trained, examples may form (many) modes on hidden representations, i.e. outputs of hidden layers, conditionally to their classes. Most notably, on the penultimate representation before the decision stage, examples should agglomerate in distinct clusters according to their label as seen on Figure 1. From the aforementioned prior perspective about the hidden representations, we aim in this work to provide a learning scheme that promotes the hidden layers to build representations which are class-invariant and thus agglomerate in restricted number of modes. By doing so, we constrain the network to build invariant intermediate representations per class with respect to the variations in the input samples without explicitly specifying these variations nor the transformations that caused them.
We express this class-invariance prior as an explicit criterion combined with the classification training criterion. It is formulated as a dissimilarity between the representations of each pair of samples within the same class. The average dissimilarity over all the pairs of all the classes is considered to be minimized. To the best of our knowledge, none has used this class membership to build invariant representations. Our motivation in using this prior knowledge, as a form of regularization, is to be able to train deep neural networks and obtain better generalization error using less training data. We have conducted different experiments over MNIST benchmarck using two models (multilayer perceptrons and convolutional networks) for different classification tasks. We have obtained results that show important improvements of the model's generalization error particularly when trained with few samples.
The rest of the paper is organized as follows: Section 2 presents related works for invariance learning in neural networks. We present our learning framework in Section 3 followed by a discussion of the obtained results in Section 4.

Related Work
Learning general invariance, particularly in deep architectures, is an attractive subject where different approaches have been proposed. The rational behind this framework is to ensure the invariance of the learned model toward the variations of the input data. In this section, we describe three kinds of approaches of learning invariance within neural networks. Some of these methods were not necessarily designed to learn invariance however we present them from the invariance perspective. For this description, f is the target function to be learned.

Invariance through data transformations:
It is well known that generalization performance can be improved by using larger quantity of training samples. Enlarging the number of samples can be achieved by generating new samples through the application of small random transformations such as rotation, scaling, random noise, etc [4,10,40] to the original examples. Incorporating such transformed data within the learning process has shown to be helpful in generalization [31]. [1] proposes the use of prior information about the behavior of f over perturbed examples using different transformations where f is constrained to be invariant over all the samples generated using these transformations. While data transformations successfully incorporate certain invariance into the learned model, they remain limited to some predefined and well known transformations. Indeed, there are many other transformations which are either unknown or difficult to formalize.

Invariance through model architectures:
In some neural network models, the architecture implicitly builds a certain type of invariance. For instance, in convolutional networks [26,34,15], combining layers of feature extractors using weight sharing with local pooling of the feature maps introduces some degree of translation invariance [32,28]. These models are currently state of the art strategies for achieving invariance in computer vision tasks. However, it is unclear how to explicitly incorporate in these models more complicated invariances such as large angle rotation and complex illumination. Moreover, convolutional and max-pooling techniques are somewhat specialized to visual and audio processing, while deep architectures are generally task independent.
Invariance through analytical constraints: Analytical invariance consists in adding an explicit penalty term to the training objective function in order to reduce the variations of f or its sub-parts when the input varies. This penalty is generally based on the derivatives of a criterion related to f with respect to the input. For instance, in unsupervised representation learning, [36] introduces a penalty for training auto-encoders which encourages the intermediate representation to be robust to small changes of the input around the training samples, referred to as contractive autoencoders. This penalty is based on the Frobenius norm of the first order derivative of the hidden representation of the auto-encoder with respect to the input. Later, [35] extended the contractive auto-encoders by adding another penalty using the norm of an approximation of the second order derivative of the hidden representation with respect to the input. The added term penalizes curvatures and thus favors smooth manifolds. [17] exploit the idea that solving adversarial examples is equivalent to increase the attention of the network to small perturbation for each example. Therefore, they propose a layer-wise penalty which creates flat invariance regions around the input data using the contractive penalty proposed in [36]. [38,39] penalize the derivatives of f with respect to perturbed inputs using simple distortions in order to ensure local invariance to these transformations. Learning invariant representations through the penalization of the derivatives of the representation function Γ(·) is a strong mathematical tool. However, its main drawback is that the learned invariance is local and is generally robust toward small variations.
Learning invariance through explicit analytical constraints can also be found in metric learning. For instance, [9,18] use a contrastive loss which constrains the projection in the output space as follows: input samples annotated as similar must have close (adjacent) projections and samples annotated as dissimilar must have far projections. In the same way, Siamese networks [7] proceed in learning similarity by projecting input points annotated as similar to be adjacent in the output space. This approach of analytical constraints is our main inspiration in this work, where we provide a penalty that constrains the representation function Γ(·) to build similar representation for samples from the same class, i.e. in a supervised way.
In the following section, we present our proposal with more details.

Proposed Method
In deep neural networks, high layers tend to learn abstract representations that we have assumed to be closer and closer for the same class along the layers. We would like to promote this behavior. In order to do so, we add a penalty to the training criterion of the network to constrain intermediate representations to be class-invariant. We first describe a model decomposition, the general training framework and then more specific implementation details.

Model Decomposition
Let us consider a parametric mapping function for classification: M(.; θ) : X → Y, represented here by a neural network model, where X is the input space and Y is the label space. This neural network is arbitrarily decomposed into two parametric sub-functions: 1. Γ(·; θ Γ ) : X → Z, a representation function parameterized with the set θ Γ . This sub-function projects an input sample x into a representation space Z. 2. Ψ(·; θ Ψ ) : Z → Y, a decision function parameterized with the set θ Ψ . It performs the classification decision over the representation space Z.
The network decision function can be written as follows: Such a possible decomposition of a neural network with K = 4 layers is presented in Fig.2. Here, the decision function Ψ(·) is composed of solely the output layer while the rest of the hidden layers form the representation function Γ(·). Figure 2: Decomposition of the neural network M(·) into a representation function Γ(·) and a decision function Ψ(·).

General Training Framework
In order to constrain the intermediate representations Γ(·) to form clusters over all the samples within the same class we modify the training loss by adding a regularization term. Thus, the training criterion J is composed of the sum of two terms. The first term J sup is a standard supervised term which aims at reducing the classification error. The second and proposed regularization term J H is a hint penalty that aims at constraining the intermediate representations of samples within the same class to be similar. By doing so, we constrain Γ(·) to lean invariant representations with respect to the class membership of the input sample. even if D and D s contains tuples of (feature,target), x represents only the feature part in the notation x ∈ D.

Proposed Hint Penalty
Let x i be an input sample. We want to reduce the dissimilarity over the space Z between the projection of x i and the projection of every sample x j ∈ D s with j = i. For this sample x i , our hint penalty can be written as follows: where C h (·, ·) is a loss function that measures how much two projections in Z are dissimilar and |D s | is the number of samples in D s .

Regularized Training Loss
The full training loss can be formulated as follows: where γ and λ are regularization weights, C sup (·, ·) the classification loss function. If one use a dissimilarity measure C h (·, ·) in J h that is symmetrical such as typically a distance, summations in the term J H could be rewritten to prevent the same sample couple to appear twice.
Eq.3 shares a similarity with the contrastive loss [9,18,7]. This last one is composed of two terms. One term constrains the learned model to project similar inputs to be closer in the output space. In Eq.3, this is represented by the hint term. In [9,18,7], to avoid collapsing all the inputs into one single output point, the contrastive loss uses a second term which projects dissimilar points far from each other by at least a minimal distance. In Eq.3, the supervised term prevents, implicitly, this collapsing by constraining the extracted representations to be discriminative with respect to each class in order to minimize the classification training error.

Implementation and Optimization Details
In the present work, we have chosen the cross-entropy as the classification loss C sup (·, ·).
In order to quantify how much two representation vectors in Z are dissimilar we proceed using a distance based approach for C h (·, ·). We study three different measures: the squared Euclidean distance (SED), the normalized Manhattan distance (NMD), and the angular similarity (AS), Minimizing the loss function of Eq.3 is achieved using Stochastic Gradient Descent (SGD). Eq.3 can be seen as multi-tasking where two tasks represented by the supervised term and the hint term are in concurrence. One way to minimize Eq.3 is to perform a parallel optimization of both tasks by adding their gradient. Summing up the gradient of both tasks can lead to issues mainly because both tasks have different objectives that do not steer necessarily in the same direction. In order to avoid these issues, we propose to separate the gradients by alternating between the two terms at each mini-batch which showed to work well in practice [8,45,11,5]. Moreover, we use two separate optimizers where each term has its own optimizer. By doing so, we make sure that both gradients are separated.
On a large dataset, computing all the dissimilarity measures in J H in Eq.3 over the whole training dataset is computationally expensive due to the large number of pairs. Therefore, we propose to compute it only over the mini-batch presented to the network. Consequently, we need to shuffle the training set D periodically in order to ensure that the network has seen almost all the possible combinations of the pairs. We describe our implementation in Alg.1.

Experiments
In this section, we evaluate our regularization framework for training deep networks on a classification task as described in Section 3. In order to show the effect of using our regularization on the generalization performance, we will mainly compare the generalization error of a network trained with and without our regularizer on different benchmarks of classification problems.

Classification Problems and Experimental Methodology
In our experiments, we consider three classification problems. We start by the standard MNIST digit dataset. Then, we complicate the classification task by adding different types of noise. We consider the three following problems: • The standard MNIST digit classification problem with 50000 , 10000 and 10000 training, validation and test set. We refer to this benchmark as mnist-std. (Fig.4, top row).
• MNIST digit classification problem where we use a background mask composed of a random noise followed by a uniform filter. The dataset is composed of 100000 , 20000 and 50000 samples for train, validation and test set. Each set is generated from the corresponding set in the benchmark mnist-std. We refer to this benchmark as mnist-noise. (Fig.4, middle row).
• MNIST digit classification problem where we use a background mask composed of a random picture taken from CIFAR-10 dataset [23]. This benchmark is composed of 100000 samples for training built upon 40000 training samples of CIFAR-10 training set, 20000 samples for validation built upon the rest of CIFAR-10 training set (i.e. 10000 samples) and 50000 samples for test built upon the 10000 test samples of CIFAR-10. We refer to this benchmark as mnist-img. (Fig.4, bottom row). All the images are 28 × 28 gray-scale values scaled to [0, 1]. In order to study the behavior of our proposal where we have few training samples, we use different configurations for the training set size. We consider four configurations where we take only 1000 , 3000 , 5000 , 50000 or 100000 training samples from the whole available training set. We refer to each configuration by 1k , 3k , 5k , 50k and 100k respectively. For the benchmark mnist-std, only the configurations 1k , 3k , 5k and 50k are considered.
For all the experiments, we consider the two following neural network architectures: • Multilayer perceptron with 3 hidden layers followed by a classification output layer. We use the same architecture as in [13] which is 1200 − 1200 − 200. This model is referred to as mlp.
• LeNet convolutional network [27], which is well known in computer vision tasks, (with similar architecture to LeNet-4) with 2 convolution layers with 20 and 50 filters of size 5 × 5, followed by a dense layer of size 500, followed by a classification output layer. This model is referred to as lenet.
Each model has three hidden layers, we refer to each layer from the input toward the output layer by: h 1 , h 2 and h 3 respectively. The output layer is referred to as h 4 . When using our hint term, we refer to the model by mlp + hint and lenet + hint for the mlp and lenet models respectively.
Each experiment is repeated 7 times. The best and the worst test classification error cases are discarded. We report the mean ± standard deviation of the validation (vl) and the test (tst) classification error of each benchmark. Models without regularization are trained for 2000 epochs. All the models regularized with our proposal are trained for 400 epochs which we found enough to converge and find a better model over the validation set. All the trainings are performed using stochastic gradient descent with an adaptive learning rate applied using AdaDelta [50], with a batch size of 100 .
Technical Details: • We found that layers with bounded activation functions such as the logistic sigmoid or the hyperbolic tangent function are more suitable when applying our hint term. Applying the regularization term over a layer with unbounded activation function such as the Relu [30] did not show an improvement.
The source code of our implementation is freely available 4 .

Results
As we have described in Sec.3, our hint term can be applied at any hidden layer of the network. In this section, we perform a set of experiments in order to have an idea which one is more adequate to use our regularization. To do so, we trained the mlp model for classification task over the benchmark mnist-std using different configurations with and without regularization. The regularization is applied for one hidden layer at a time h 1 , h 2 or h 3 . We used the squared Euclidean distance (Eq.4) as a dissimilarity measure. The obtained results are presented in Tab  From Tab.1, one can see that regularizing low layers h 1 , h 2 did not help improving the performance error but it did increase it in the configuration 1k, for instance. This may be explained by the fact that low layers in neural networks tend to learn low representations which are shared among high representations. This means that these representations are not ready yet to discriminate between the classes. Therefore, they can not be used to describe each class separately. This makes our regularization inadequate at these levels because we aim at constraining the representations to be similar within each class while these layers are incapable to deliver such representations. Therefore, regularizing these layers may hamper their learning. As a future work, we think that it would be beneficial to use at low layers a regularization term that constrains the representations of samples within different classes be dissimilar such as the one in the contrastive loss [9,18,7].
In the case of regularizing the last hidden layer h 3 , we notice from Tab.1 an important improvement in the classification error over the validation and the test set in most configurations. This may be explained by the fact that the representations at this layer are more abstract, therefore, they are able to discriminate the classes. Our regularization term constrains these representations to be tighter by re-enforcing their invariance which helps in generalization. Therefore, applying our hint term over the last hidden layer makes more sense and supports the idea that high layers in neural networks learn more abstract representations. Making these discriminative representations invariant helps the linear output layer in the classification task. For all the following experiments, we apply hint term over the last hidden layer. Moreover, one can notice that our regularization has less impact when adding more training samples. For instance, we reduced the classification test error by: 1.74%, 0.92% and 0.58% in the configurations 1k, 3k and 5k. This suggests that our proposal is more efficient in the case where few training samples are available. However, this does not exclude using it for large training datasets as we will see later (Tab.2, 3). We believe that this behavior depends mostly on the model's capacity to learn invariant representations. For instance, from the invariance perspective, convolutional networks are more adapted, conceptually, to process visual content than multilayers perceptrons.
In another experimental setup, we investigated the effect of the measure used to compute the dissimilarity between two feature vectors as described in Section.3.3. To do so, we applied our hint term over the last hidden layer h 3 using the measures SED, NMD and AS over the benchmark mnist-std.  In another experiment, we evaluated the benchmarks mnist-noise and mnist-img, which are more difficult compared to mnist-std, using the model lenet which is more suitable to process visual content. Similarly to the previous experiments, we applied our regularization term over the last hidden layer h 3 using the SED measure. The results depicted in Tab.3 show again that using our proposal improves the generalization error of the network particularly when only few training samples are available. For example, our regularization allows to reduce the classification error over the test set by 2.98% and by 4.16% over the benchmark mnist-noise and mnist-img, respectively when using only 1k training samples.  Based on the above results, we conclude that using our hint term in the context of classification task using neural networks is helpful in improving their generalization error particularly when only few training samples are available. This generalization improvement came at the price of an extra computational cost due the dissimilarity measures between pair of samples. Our experiments showed that regularizing the last hidden layer using the squared Euclidean distance give better results. More generally, the obtained results confirm that guiding the learning process of the intermediate representations of a neural network can be helpful to improve its generalization.

On Learning Invariance within Neural Networks
We show in this section an intriguing property of the learned representations at each layer of a neural network from the invariance perspective. For this purpose and for the sake of simplicity, we consider a binary classification case of the two digits "1" and "7". Furthermore, we consider the mlp model over the lenet in order to be able to measure the features invariances over all the layers. We trained the mlp model over the benchmark mnist-std where we used all the available training samples of both digits. The model is trained without our regularization. However, we tracked, at each layer and at the same time, the value of the hint term J H in Eq.3 over the training set using the normalized Manhattan distance as a dissimilarity measure. This particular dissimilarity measure allows comparing the representations invariance between the different layers due to the normalization of the measure by the representations dimension. The obtained results are depicted in Fig.5 where the x-axis represents the number of mini-batches already processed and the y-axis represents the value of the hint term J H at each layer. Low value of J H means high invariance (better case) whereas high value of J H means low invariance.  of the mlp over the train set of mnist-std benchmark for a binary classification task: the digit "1" against the digit "7".
In Fig.5, we note two main observations: • The value of the hint term J H is reduced through the depth of the network which means that the network learns more invariant representations at each layer in this order: layer 1, 2, 3, 4. This result supports the idea that abstract representations, which are known to be more invariant, are learned toward the top layers.
• At each layer, the network does not seem to learn to improve the invariance of the learned representations by reducing J H . It appears that the representations invariance is kept steady all along the training process. Only the output layer has learned to reduce the value of J H term because minimizing the classification term J sup reduces automatically our hint term J H . This shows a flaw in the back-propagation procedure with respect to learning intermediate representations. Assisting the propagated error through regularization can be helpful to guide the hidden layers to learn more suitable representations.
These results show that relying on the classification error propagated from the output layer does not necessarily constrain the hidden layers to learn better representations for classification task. Therefore, one would like to use different prior knowledge to guide the internal layers to learn better representations which is our future work. Using these guidelines can help improving neural networks generalization especially when trained with few samples.

Conclusion
We have presented in this work a new regularization framework for training neural networks for classification task. Our regularization constrains the hidden layers of the network to learn class-wise invariant representations where samples of the same class have the same representation. Empirical results over MNIST dataset and its variants showed that the proposed regularization helps neural networks to generalize better particularly when few training samples are available which is the case in many real world applications.
Another result based on tracking the representation invariance within the network layers confirms that neural networks tend to learn invariant representations throughout staking multiple layers. However, an intriguing observation is that the invariance level does not seem to be improved, within the same layer, through learning. We found that the hidden layers tend to maintain a certain level of invariance through the training process.
All the results found in this work suggest that guiding the learning process of the internal representations of a neural network can be helpful to train them and improve their generalization particularly when few training samples are available. Furthermore, this shows that the classification error propagated from the output layer does not necessarily train the hidden layers to provide better representations. This encourages us to explore other directions to incorporate different prior knowledge to constrain the hidden layers to learn better representations in order to improve the generalization of the network and be able to train it with less data.