{'fraction_non_alphanumeric': 0.01801634472511144, 'fraction_numerical': 0.005386329866270431, 'mean_word_length': 4.972823072656683, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Robustness to Augmentations as a Generalization metric

Generalization is the ability of a model to predict on unseen domains and is a fundamental task in machine learning. Several generalization bounds, both theoretical and empirical have been proposed but they do not provide tight bounds .In this work, we propose a simple yet effective method to predict the generalization performance of a model by using the concept that models that are robust to augmentations are more generalizable than those which are not. We experiment with several augmentations and composition of augmentations to check the generalization capacity of a model. We also provide a detailed motivation behind the proposed method. The proposed generalization metric is calculated based on the change in the output of the model after augmenting the input. The proposed method was the first runner up solution for the NeurIPS competition on Predicting Generalization in Deep Learning.


Introduction
Deep learning models are more of a black box technique and it is hard to explain why a model behaves in the way it does. Despite this limitation, there is the tremendous success of deep neural networks in a wide variety of tasks ranging from image classification to speech recognition [8,15]. But there is still a need to able to understand why these models work so well. Generalization is one of the fundamental problems in machine learning and estimating the generalization performance of a model is not trivial. Many generalization metrics have been proposed but they tend to overestimate the generalization performance. Recent developments in the field are focused on the calculation of a complexity measure, which is a quantity to directly indicate the extent of generalizability of a model. There have been theoretical [1,18] and empirical [13] measures that can assess the degree of generalization of a model. Theoretical complexity measures such as VC-dimension [3] has been the standard for measuring generalizability as there has been an established monotonic relation between the VC-dimension and the generalizability of a model. Generalization performance is very crucial in practical computer vision and models need to adapt and generalize well to unseen domains. Current computer vision models don't generalize very well to unseen domains and there has been extensive work in domain adaptation [5] and domain generalization to fit new domains. The generalization gap of a model is defined as the difference between the estimated risk of a target function and the empirical risk of a target function. The estimated risk is not computable but is estimated as the risk of the target function on a validation set and the empirical risk is estimated as the risk of the target function on the training set. The task of the competition was to predict the generalization of a model through a complexity measure that maps the model and dataset to a real number. This real number indicates the generalization ability of the model.

Related Work
Jiang* et al. [10] presents a large scale study of various generalization metrics that have been applied in deep neural networks. They explore 40 complexity measures, both theoretical and experimental, and provide metrics for the evaluation of generalization bounds. They discuss conducting a conditional independent test by calculating the conditional mutual information between the generalization metric and the observed generalization in a causal probabilistic graph. This causality is conditioned over a set of hyper-parameters of the model. Keskar et al. [13] suggested using sharpness as a generalization measure and sharpness measures the robustness to adversarial perturbations. They also showed the success of small batch methods over large batch methods when trained with Stochastic Gradient Descent and showed that training with large batch methods tends to converge to sharp minima which leads to poorer generalization. Neyshabur et al. [19] investigates different complexity measures like norm, PAC-Bayes to explain the generalization of deep networks. They also show that combining sharpness with PAC-Bayes analysis with the norm of the weights. Jiang et al. [9] proposed to use the margin distribution as a predictor for the generalization gap in deep neural networks.
Dziugaite and Roy [4] proposes direct optimization on PAC-Bayes bounds to compute non-vacuous numerical bounds on generalization error in deep neural networks with more model parameters than training data. Kawaguchi et al. [12] has discussed theoretical motivations for the ability of deep neural networks to generalize, and provides new open problems in the field.

Motivation
Many deep neural networks have used data augmentation to prevent overfitting. But recent work [2], shows convolutional neural networks are very sensitive to small geometric transformations to the input that are imperceptible to humans even when the model is trained on a wide range of augmented data. The proposed method is based on a simple hypothesis that a model capable of generalizing must be robust to augmentations. A model's output should not change significantly when certain augmentations are performed on the input. In other words, a model's predicted class for input should not change when we augment the input without erasing key features from the input. For example, if a picture of a cat which the model correctly classified is augmented with a rotation by 180 degrees, then the model must still predict it as a cat. The model should confidently predict the augmented input if it has learned the correct features of a particular class. Humans predominantly classify objects based on their shape rather than texture whereas most CNNs classify based on texture rather than shape [7]. Augmentations that perform changes to the texture would be a suitable test of a model's generalizability.
The generalization power can also be looked at from the point of view of the model's performance on input from a shifted distribution. Based on this, we augment images in such a way that important features of the image are retained.

Proposed Metric
Algorithm 1 describes the proposed generalization metric. For every sample, we augment the input and then compare the class prediction of the model for the original input and augmented input. If the class prediction is the same even after the input is augmented, then we add a penalty equal to the difference between probabilities of the predicted class on the original and augmented input. This Algorithm 1: Proposed metric calculation Input: Consider a model θ; x is the input; λ is the penalty for an augmentation.

Result: Generalization metric
represents the difference in the confidence in the class when the input is augmented. The penalty is determined based on the strength of the augmentation. The strength of augmentation is determined by the ability of the augmentation to change the texture in the input. Augmentations that do not alter the texture of the image, but to tend to alter the shape in the image, are weak. For example, flipping an image left to right does not change any texture in the input, only the shape. Augmentations that alter the texture in an image are referred to as strong augmentations. These augmentations may or may not alter the shape of the image, but they affect the texture in the image. For example, applying a Sobel filter [11] on an image removes most of the texture in the image and retains only an edge map. A model that misclassifies samples on weak augmentations are considered to generalize poorly are penalized heavily. Models that misclassify samples with strong augmentations are penalized less. The generalization metric reflects the penalty that has been incurred over an augmented subset of the training data. Thus, models that accumulate a large penalty have a higher negative score and tend to generalize poorly.   [20]: Erases a random grid of the image 6. Sobel Filter [11]: Provides an edge map of the image 7. Virtual Adversarial Perturbation [16]: Inspired by the metric of sharpness [13], a similar method which measures the robustness of the model against local perturbation was used as an augmentation. Figure 1 is a visual depiction of some of the augmentations. We experimented with individual and composition of augmentations like flip and then add saturation, which yielded good results. We also experimented with neural style transfer [6,7] which makes the image texture-invariant, but were unable to produce good results probably because of the texture bias of most of the models.

Results
These are the results on the public dataset calculated on the evaluation metric, Conditional Mutual Information [10]. The public dataset of this competition involves two groups of models trained on two publicly available datasets, namely, CIFAR-10 [14] and SVHN [17]. The models were trained on different model architectures and different training schemes. The models trained on CIFAR-10 had a VGG like architecture trained with different batch sizes, learning rate, dropout rate, convolution layers, and dense layers. The training datasets consisted of a total of 150 models trained on different architectures with different hyperparameters. Higher the score, the better the generalization metric. Table 1 shows the penalties for each of the augmentations and the respective scores on both the public and private datasets. λ indicates the penalty value for the respective augmentation and λ v indicates the penalty for the virtual adversarial perturbation. It can be seen that a high penalty value for the composition of simple augmentations seem to work well in the datasets and using stronger augmentations like cutout seem to have a positive effect on the score.
We had experimented with each of these augmentations individually, although we were able to score good results, we found that compounding each augmentation with relative penalties provided greater scores. This ensemble of augmentations allowed us to measure the robustness of a model against various strong and weak augmentations. The penalties for each augmentation were empirically calculated.

Conclusion
Calculation of the generalization bound on a deep learning network is a critical field of study and extensive work is being done in the field. We have provided a measure of a model's ability to generalize based on conditional mutual information. This method tests the model with augmented samples from the training data and penalizes models that are unable to correctly classify augmented samples. We have also provided an insight into the role of augmentations in testing the generalizability of a model. This metric can be used to assess the generalization capacity of any deep learning model.