{'fraction_non_alphanumeric': 0.029021394611727418, 'fraction_numerical': 0.009508716323296355, 'mean_word_length': 4.809167625623322, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Nuisance-Label Supervision: Robustness Improvement by Free Labels

In this paper, we present a Nuisance-label Supervision (NLS) module, which can make models more robust to nuisance factor variations. Nuisance factors are those irrelevant to a task, and an ideal model should be invariant to them. For example, an activity recognition model should perform consistently regardless of the change of clothes and background. But our experiments show existing models are far from this capability. So we explicitly supervise a model with nuisance labels to make extracted features less dependent on nuisance factors. Although the values of nuisance factors are rarely annotated, we demonstrate that besides existing annotations, nuisance labels can be acquired freely from data augmentation and synthetic data. Experiments show consistent improvement in robustness towards image corruption and appearance change in action recognition.


Introduction
Model robustness is an important topic because it decides on whether a model can work well in actual applications. Many factors can affect model robustness, and spurious correlation is one of them. Spurious correlation refers to using inaccurate information to make predictions. For example, playing football and grass are always bundled up in action datasets; if a model uses grass to predict the task label playing football, it is relying on the spurious correlation between grass and the task label, and when people play football in other scenes, it fails. For a certain task, we call factors related to the task as essential factors and others as nuisance factors. When training a model, the goal is to accurately make predictions depending on essential factors, while invariant to the change of nuisance factors. Therefore, we should prevent models from relying on spurious correlation between task labels and nuisance factors.
Increasing data diversity is a common way to solve spurious correlation however it suffers from problems. The Figure 1. Simple illustration of the core concept. Our proposed method Nuisance-label Supervision (NLS) module makes use of nuisance labels, which is usually threw out by other methods, to improve model robustness.
strong correlations are hard to break, e.g. the football and grass pair commonly appear in most action datasets. Simply adding to training data diversity is not sufficient for removing the correlation between nuisance factors and task labels. Recent works illustrate this problem that ImageNet-trained models are still biased towards textures [11], action recognition models trained on UCF101 or Kinetics are sensitive to viewpoint and human appearance changes [21]. Therefore, exploration about proactively disentangling nuisance factors and task label is worth thinking about.
We propose to explicitly use nuisance labels to train a feature representation robust to nuisance factor perturbations. We design a Nuisance-label Supervision (NLS) module, which utilizes nuisance labels as extra supervision signals and makes the whole network training in an adversary way. We first prove the core idea on MNIST-C which is widely used to study image corruption, then we demonstrate the method effectiveness on UCF101, and NTU RGB-D, which are both action datasets; the former is a popular dataset, but models trained on it usually fail to generalize to other datasets; the latter is a controlled dataset with multiple available nuisance labels. By using NLS, our trained model will be insensitive to viewpoint or human appearance changes.
Though NLS is simple and intuitive, collecting enough training data with nuisance labels is a non-trivial issue. We answer this challenging question by giving out three practical and low-cost ways of acquiring nuisance labels: 1) image processing parameters when doing data augmentation; 2) metadata in the data collection procedure; 3) rendering parameters from synthetic data generation. These three types of data are free and often ignored; Fig. 1 vividly illustrates the idea of making waste profitable. We describe in detail how we get nuisance labels from these three sources in Sec. 3.3 and demonstrate the effectiveness in the experiment section. Previously, extra labels other than the task label have been proved effective in intermediate supervision [16] and multi-task learning [38]. Compared with them, nuisance labels in our method are easy to collect because they all come from the processing steps of data generation. Our module is generic and easy to use. It is not restricted to tasks or models and can be attached to any existing models, which makes our method widely applicable. The improvement by NLS is totally for free because the nuisance labels come from the data generation process, which costs no time to annotate. Besides, NLS only works during the training time and adds no extra calculation to inference.
In addition to NLS, we explore using nuisance labels for understanding model robustness. We diagnose model dependency on a certain nuisance factor and analyze its relationship with robustness. The results show the more a model relies on nuisance factors, the worse it performs on test set with different nuisance factor values.
The contribution can be summarized as follows: • We propose a Nuisance-label Supervision (NLS) module to improve model robustness by breaking spurious correlation. The module is generic and can be applied to different tasks and models.
• We propose three practical and low-cost ways of acquiring nuisance labels. Besides existing annotations, parameters from data augmentation and synthetic data can also be used for generating nuisance labels.
• Extensive experiments demonstrate the effectiveness of improving model robustness to image corruption and appearance changes. Nuisance labels are used to further understand over-fitting and model robustness improvement.

Related Work
Robustness against irrelevant perturbations. Recent works attempt to solve generalization failure by data augmentation and bias mitigation. Cubuk and Zoph [7] proposed AutoAugment for searching improved data augmentation policies; Hendrycks and Mu [13] proposed image processing technique AugMix, Rusak and Schott [29] introduced adding Gaussian noise to training data (GNT) and novel adversarial noise generator (ANT) to overcome unseen corruptions. These approaches focus on increasing data diversity while not making use of intermediate labels.
Singh et al. [31] used CAM [41] and feature-slitting methods to decorrelate category and its co-occurring context but only applicable to fixed category pairs. Approaches [1,14] utilized extra labels to mitigate bias in feature representations, such as removing gender bias from age classification. While we focus on multiple factors, especially those rarely explored, and make use of parameters in synthetic data and data augmentation, which are cheaper labels compared with other work.
Action Recognition. Action recognition model robustness to nuisance factors has gradually received more attention in recent years and is found to be fragile. Lyu et al. [21] showed TSN [37] and I3D [4] make predictions relying on irrelevant information, such as related objects. Li et al. [17] revealed that current action datasets are biased towards the scene, objects or people. Thus our work focuses on RGBbased models and tries to increase their robustness even learning from a biased dataset. Choi et al. [6] introduce two adversarial losses to remove background effect while our work focus on multiple nuisance factors, including both 2D image corruptions and 3D factors e.g. viewpoint.
Usage of extra labels. Intermediate supervision and multi-task focus on the usage of extra labels. Multi-task learning focuses on related tasks. e.g. pose estimation and action recognition [12], surface normals and depth [24], etc. While we use labels not specifically annotated for any tasks, but parameter labels with less semantic meanings. DSN [15] is the first to propose the concept of deep supervision where the final label is used for supervising intermediate layers in a network. Furthermore, Li et al. [16] utilized intermediate shape concepts acquired from rendered data to supervise CNN hidden layers and achieved synthetic to real generalization. Whereas nuisance labels used in our method is different from the final label or intermediate concepts, which are easy-acquired but often ignored.
Synthetic data in vision tasks. Synthetic data is widely used as augmentation and shows merits on vision tasks. The domain gap between synthetic and real is always a problem to solve and the various intermediate information during rendering is rarely utilized. Previously, domain randomization [33] is a popular way for bridging the real-sim gap. On the one hand, our proposed NLS helps reduce the domain gap and offers a good way for using extra information. On the other hand, synthetic data serves as a source of nuisance labels, providing a larger application space for our method, including action recognition [8,39,34], pose estimation [5,26,27], segmentation [35,25,28], etc.

Method
This section can be roughly divided into four parts. First, we formulate the problem of using nuisance labels to improve model robustness (Sec. 3.1). Second, we illustrate the structure and adversarial training procedure of our proposed NLS module (Sec. 3.2). Third, we show the generation of nuisance labels and synthetic data (Sec. 3.3). Finally, we define Dependency Degree on Nuisance Factor and study it with nuisance labels (Sec. 3.4).

Problem Statement
The objective of our work is to weaken the influence of nuisance factor perturbations by making full use of nuisance labels. We demonstrate our method on action recognition tasks because the action model is easily affected by various factors.
We aim to train a model robust to nuisance factor perturbations, especially when training and testing data have different nuisance factor values. We suppose the process of making prediction y is written as: where F denotes the model, y denotes model prediction, the factors for generating input x ∈ X can be divided into essential factors ϕ ∈ Φ and nuisance factors ψ ∈ Ψ, D represents data generator. Thus an ideal model robust to nuisance factors should meet the following characteristics: where no matter what value nuisance factors take, as long as essential factors do not change, final prediction should stay the same. In order to better illustrate the problem, we give the definition of important terms. Examples specific to action recognition are given after the definition of each term. Nuisance Factors (Ψ): factors irrelevant to final prediction. The values of nuisance factors are nuisance labels and can be acquired from the data generation procedure. In this work, we divide the nuisance labels into three types according to their resources: 1) image processing parameters; 2) metadata in the data collection procedure; 3) rendering parameters from synthetic data generation. The details are shown in Sec. 3.3 Essential Factors (Φ): factors decisive to final prediction. For action recognition, Φ is human motion, which is directly relevant to action definition. Task Label (Y ): label for the main task. Task label does not necessarily equal essential factor labels, it is decided by essential factor values. Task label is unique in a certain task while essential factors may be more than one. For example, the essential factor in action recognition is human motion, and the task label is an action class.
To achieve the goal, we want the extracted features to contain the least information about nuisance factors. Most classification model F can be divided into feature extractor f with parameter θ f and classifier g with parameter θ g . In this way, the objective can be defined as: where input data x ∈ X, nuisance factor ψ ∈ Ψ, I(a; b) denotes the mutual information between a and b.
The intuition behind minimizing mutual information between ψ and f (x) is that the main task classifier g takes extracted features f (x) as input, so if f (x) contains no information about nuisance factors, g will not utilize nuisance factor features, thus being robust to nuisance factor perturbations.

Nuisance-label Supervision (NLS) Module
In this section, we present the formulation of the proposed nuisance-label supervision module. The module utilizes nuisance labels as extra supervision, in order to train a feature representation with the least information about nuisance factors.
Overview. We give an overview of the whole network architecture in Fig. 2. We utilize adversarial training to achieve the goal of "removing nuisance information" inspired by [10]. We divide input data into two parts: 1) data w/o nuisance labels, 2) data w/ nuisance labels. On the one hand, both inputs are put through a feature extractor and classifier the same as normal action recognition pipeline. On the other hand, data with nuisance labels is additionally put into NLS module for training feature representation. In this process, nuisance labels are used as extra supervision signals and output an adversarial loss.
Architecture. The NLS module consists of two parts: gradient reversal layer [10] and nuisance factor classifier as is shown in Fig. 3. The module input is extracted features and nuisance labels. The nuisance factor classifier is to predict labels of a certain nuisance factor, while corresponding nuisance labels are used for calculating a cross-entropy loss, which is widely used for classification tasks. The gradient reversal layer multiplies the gradient by a negative constant during the back-propagation process, making the input features as indistinguishable as possible for the nuisance factor classifier and ensuring the training is in an adversary way.
where L ce denotes cross-entropy loss. Usually, there is more than one nuisance factor in a task, so we use multiple nuisance factor classifiers (h p ) to deal  Figure 2. The pipeline of the proposed method. The blue arrow represents data w/o nuisance labels while the red arrow represents data w/ nuisance labels. Both inputs are put through feature extractor f and classifier g as yellow arrow shows. Additionally, data with nuisance labels is put into NLS module for training feature representations. (Better view in color) with each nuisance factor, then aggregate the outputs. The nuisance adversarial loss for multiple factors is defined as: where superscript p denotes the p th type of nuisance factors, and N denotes the number of nuisance factors matter in the task. Fig. 3 also illustrates how we deal with multiple nuisance factors in NLS module.
Training Procedure. Additional to removing nuisance factor effects, we certainly need the model to learn useful features for the main classification task: So the final objective consists of classification loss and nuisance adversarial loss, which is formulated as follows: where, the parameter λ controls the strength of the nuisance adversarial loss.
Based on the above objective function, the optimization  Figure 4. Synthetic data is an effective way of getting nuisance labels and we build a pipeline for generating synthetic data. We follow the concept of domain randomization to set camera position, actor appearance and background in the virtual world. Together with action labels and video, we save these rendering parameters as nuisance labels.
problem is defined as follows: At training time, the nuisance factor classifier parameters θ h minimize the nuisance adversarial loss and the action classifier parameters θ g minimize classification loss as normal classification task. The feature extractor parameters θ f minimize classification loss while maximizing nuisance adversarial loss simultaneously.

Nuisance-label Generation
In this section, we formulate the generation of nuisance labels. As stated in Sec. 3.1, we define data generation as D(ϕ, ψ). Usually, we only record labels for ϕ and throw out labels for ψ.
Data augmentation actually offers more information than people used to expect; the image processing parameters can be used in training as nuisance labels. For example, when generating images with Gaussian Noise, we can save the standard deviation values; when applying Median Blur, we

Control Viewpoint
Control Actor Appr.
Control BG Real Data Synthetic Data Figure 5. Examples of synthetic data. The appearance is similar in each row and different in each column. But the action label of each row is the same. So a robust model needs to be invariant to these dramatic nuisance factors change. We utilize nuisance labels from synthetic data to achieve this goal.
can save the kernel size, these can both be used as nuisance labels.
Similarly, some existing real datasets also contain such information, i.e. metadata during data collection. People record metadata only for evaluation or showing randomization, while it can be used in training as well. For example, people collect NTU RGB-D dataset in a controlled way, that for each video, the recording environment, subject appearance, and camera viewpoint are all saved. This information can be regarded as nuisance labels.
Synthetic data generation offers a convenient way to manipulate multiple factors in the virtual world and we can easily reach the ground truth of these factors. In our work, we use CG renderer Blender 1 to manipulate viewpoint, textures in the virtual world, as is shown in Fig. 4 (more details are in Appendix). We save camera position, actor appearance id and background id as nuisance labels. The examples of our CG synthetic data are shown in Fig. 5.

Dependency Degree on Nuisance Factor
Besides serving as extra supervision signals, nuisance labels are also used to probe model dependency degree on certain factors.
We design Dependency Degree on Nuisance Factor to quantitatively measure how much a trained feature extractor focuses on nuisance factors. We attach a classifier to a trained feature extractor to predict the correct label of a certain nuisance factor. During training, weights in the feature extractor are frozen while parameters in the attached classifier are optimized. For a classification task, the worst performance corresponds to the random guess of inputs to classes, i.e. chance level performance; the chance level performance on a nuisance factor ψ is denoted as Acc Rand (ψ). Thus we define the model dependency degree on a certain nuisance factor ψ based on trained feature extractor f as: Dep(ψ|f (x)) = log P rec(ψ|f (x)) Acc Rand (ψ) where P rec(ψ|f (x)) is the precision of the attached classifier. When the dependency degree goes down to zero, the nuisance factor classifier reaches chance-level performance, thus extracted features contain no information related to the nuisance factor. The definition is inspired by Dataset Bias in [17] while it presents the dataset dependency and ours presents the model dependency on some factor.

Experiments
In this section, we explain setups and show results of our experiments. The setups including dataset and implementation details are shown in Sec. 4.1 and Sec. 4.2. The experiment results are organized by nuisance-label types, i.e. image processing parameters, real metadata and rendering parameters (Sec. 4.3, 4.4, 4.5). And the ablation study of the relationship between model robustness and nuisance factor dependency is shown in Sec. 4.6.
MNIST-C [23]. This is a small-scale handwritten digits dataset, including a training set of 60,000 examples, and a testing set of 10,000 examples with 15 types of corruptions applied to the original MNIST test set.
UCF101 [32]. This is a large dataset with 101 human actions from videos in the wild; it consists of over 13k clips and 27 hours of video data with five types of action classes. We use it to prove our proposed method on 2D nuisance factors. We build a special test set to evaluate the model robustness on image corruptions, as shown in Fig. 6.
NTU RGB-D [30]. This is a controlled human action dataset with 60 different action classes; it contains metadata besides action labels, including camera viewpoint, actor ap-pearance, and recording environment. The metadata can be used as nuisance labels in NLS module. In addition, with these annotations, we create a more challenging data split, denoted as Cross-Nuisance (CN) split, where training and testing sets share no overlap on these three factors (details shown in Appendix). This split is for evaluating model robustness to 3D nuisance factor perturbations.

Implementation Details
Network architecture. For MNIST-C, we use the same architecture as in [22]. For action recognition, we choose I3D [4] network as the baseline. We initialize the network with inflated weights from the 2D Inception network pretrained on ImageNet [9]. At training time, we randomly sample 16 consecutive frames temporally, rescale the shorter side of all input frames to 226 pixels and randomly crop a 224 × 224 patch from each frame. The left-right flip is randomly applied as 2D data augmentation. At test time, we temporally sample three fragments evenly from the full video.
Our proposed NLS module is applied to the input feature map of the main task classifier. We choose α = 0.5 for gradient reversal layer [10], which is the coefficient for reversed gradient during backpropagation. The nuisance classifier in NLS module is a three-layer MLP, with 1024 nodes in each layer.
Training Details. For MNIST-C, we train the network following the same rule as in [29]. For action recognition, we set batch size as 8 on each GPU. We use SGD optimizer with momentum = 0.9, weight decay = 1e-4. The initial learning rate equals 0.01 and decays to 10% when accuracy on validate set saturates. In order to prevent NLS from back-propagating noisy signals at the beginning of training, we set λ=0 for 10 epochs, then increase it to 0.05 for the rest of the training. We set λ so small because we want L ψ to work as a regularization term, not to interfere with the main task training.
Synthetic Data Generation. When generating synthetic data for NTU RGB-D, animation data captured from Microsoft Kinect [40] are used to drive 3D human skeletal meshes from MakeHuman 2 . Following the concept of domain randomization [33], the textures for human appearance and background are randomly sampled from 1000 images within the MSCOCO dataset [18]; the camera is randomly set on 240 positions to record videos.

NLS on Image Processing Parameters
In this section, we use corrupted data as augmentation and regard its processing parameters as nuisance labels. We regard GNT and ANT1×1 in [29] as the comparison, where Gaussian Noise and adversarial noise is added to part of the training data respectively as data augmentation.  Table 1. Accuracy on MNIST (clean data) and MNIST-C (corrupted data) test sets. We compare our NLS with GNT and ANT1×1, where models trained with noise and our NLS module.
MNIST-C Experiments. We use experiments on MNIST-C [23] to evaluate models on image corruptions. MNIST-C is a simple but popular dataset for people to study image corruption. Our proposed method NLS module is applied to GNT, where we regard Gaussian Noise standard deviations as nuisance labels. The results are shown in Table 1. Our method surpasses GNT and ANT on both clean data and corrupted data.
UCF101 Experiments. We further evaluate NLS module dealing with image corruptions on UCF101 dataset. We build a corrupted test set to evaluate the model robustness on image corruptions, as shown in Fig. 6. For each corruption, we study three types. As for the data augmentation baseline Aug, we choose only one for augmentation during training. During testing, all the types are applied respectively. We add one nuisance factor classifier for each type, through which only augmented data with corresponding corruption will pass. For comparison, we train an ANT1 × 1 and apply adversarial noise to each frame in a video sequence.
The results are shown in Table 2. We display accuracy on the clean test set and our designed corrupted test sets. The number is the mean value on three splits of UCF101. Aug has a good improvement on the corrupted test set especially on noise compared to the baseline but brings harm to clean data. Adding NLS further lifts model accuracy on clean data by 0.43% and corrupted data by 0.55% while adding adversarial noise to video data has marginal improvement. Though in some cases NLS is not the best, we highlight the consistent improvement compared with Aug.

NLS on Real Metadata
In this section, we evaluate NLS on real metadata in NTU RGB-D dataset, using its metadata, including camera viewpoint, actor appearance, and recording environment, as nuisance labels.
We conduct experiments on CN split (Sec. 4.1) and study three factors: viewpoint, actor appearance, and background in action recognition tasks. We study both real metadata and rendering parameters on the same dataset NTU RGB-D, so we denote adding NLS on real metadata as Real NLS and adding NLS on synthetic rendering parameters as Sim NLS (shown in Sec. 4.5).
The   Comparison with State-of-the-art. We conduct experiments on NTU RGB-D standard CV and CS splits, and compare it with state-of-the-art methods using RGB only. Note the data we use is totally the same as that in other methods. The results are shown in Table 4. Based on the strong baseline I3D, our proposed method reaches the best performance within RGB-based models, bringing accuracy improvement by 0.5% and 0.4% on two splits respectively. Besides, comparing with the results on CN split in Table 3, we observe that when the split is harder, the improvement brought by NLS is larger.

Model
Real Accuracy DR 18.16 DR + NLS 20.40 Table 5. Top-1 accuracy on NTU RGB-D real test set of I3D trained on domain randomization (DR) synthetic data w/ and w/o adding NLS. We train models purely on synthetic data to show that NLS has ability of reducing domain gap.

NLS on Rendering Parameters
In this section, we evaluate NLS on rendering parameters, denoted as Sim NLS. The real dataset setup is the same as in Sec. 4.4. We generate CG synthetic data following rules in Sec. 3.3 as data augmentation, randomizing viewpoint, actor and background respectively, denoted as Sim Aug. We add ANT1×1 noise to synthetic training data as a comparison.
The results of Sim NLS are shown in Table 3. Adding CG synthetic data into training brings an accuracy improvement of 4.25% while using Sim NLS further lifts the performance by 1.65%. We notice ANT1×1 has no improvement for 3D factor robustness, which shows the limitation of current 2D augmentation methods. We also study the effectiveness of adding Real and Sim NLS simultaneously, which reaches the best performance on CN test set.
Domain Generalization. We explore NLS module potential in domain generalization. The models are trained on pure synthetic data and tested on real data. The synthetic data is generated following the rule of domain randomization [33].
The results of adding NLS on viewpoint, actor appearance, and background together are shown in Table 5 The results in Table 6 show that NLS on a single nuisance factor also improves model robustness. No matter NLS is applied to which factor, the performance is better than only using augmentation. One interesting phenomenon is that NLS brings an even larger lift to unseen corruptions, such as Salt and Pepper Noise and Median Blur, showing the ability for domain generalization.

Whether NLS helps reduce over-fitting
We design experiments to study whether NLS helps reduce over-fitting on the corresponding nuisance factor using NTU RGB-D dataset. In CN test set, all of the three factors are changed, so it is hard to say whether all the factors benefit from NLS. We apply NLS to one factor each time and evaluate it on three special test sets: View Diff, BG Diff and Actor Diff. These sets have only one factor different from training data, e.g. View Diff has the same actor and background settings but different viewpoint settings. We regard the performance improvement on Diff test set as nuisance factor over-fitting reduction. Table 7 shows the results. After adding NLS, the corresponding factor over-fitting reduces obviously. Adding background NLS brings the smallest improvement, we explain it as the background in NTU RGB-D is less distinguishable than other factors. In addition, adding NLS on one nuisance factor also benefits robustness to other factors, e.g. accuracy on View Diff and BG Diff lifts sharply when adding NLS only on actor. While in most cases, adding NLS on three nuisance factors reaches the best performance.

How much does NLS reduce model dependency on nuisance factors
We use nuisance labels to diagnose the model dependency degree on each 3D nuisance factor, further explaining the reason for robustness improvement. As defined in Sec. 3 Table 7. Top-1 accuracy on special test sets, the higher accuracy, the less over-fitting to a certain nuisance factor. View Diff has only different viewpoint settings from training set. 3F refers to adding NLS on three nuisance factors and V, B, A refer to adding NLS on viewpoint, background, actor respectively. feature extractor weights frozen. Fig. 7 shows the dependency degree changes, all the results are trained and tested on synthetic data. After adding NLS, the corresponding nuisance factor dependency degree decreases sharply.

Conclusion
In this paper, we focus on the robustness problem caused by spurious correlation. We propose a novel module NLS using nuisance labels as extra supervision to break up spurious correlations, and we give three low-cost ways of acquiring nuisance labels, including image processing parameters, real metadata, and rendering parameters. Our proposed module effectively improves model robustness to nuisance factor perturbations. We demonstrate its effectiveness on MNIST-C, UCF101 and NTU RGB-D, handling image corruption and appearance changes, where our method all reaches the best performance. Furthermore, we utilize nuisance labels to diagnose the model dependency degree on a certain nuisance factor, explaining the reason why NLS helps reduce over-fitting. We hope that this work can contribute to the study of nuisance labels usage and enable more robust models.