{'fraction_non_alphanumeric': 0.03503899455759412, 'fraction_numerical': 0.01854345508612467, 'mean_word_length': 4.700783623860547, 'pattern_counts': {'":': 1, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Open-set Label Noise Can Improve Robustness Against Inherent Label Noise

Learning with noisy labels is a practically challenging problem in weakly supervised learning. In the existing literature, open-set noises are always considered to be poisonous for generalization, similar to closed-set noises. In this paper, we empirically show that open-set noisy labels can be non-toxic and even benefit the robustness against inherent noisy labels. Inspired by the observations, we propose a simple yet effective regularization by introducing Open-set samples with Dynamic Noisy Labels (ODNL) into training. With ODNL, the extra capacity of the neural network can be largely consumed in a way that does not interfere with learning patterns from clean data. Through the lens of SGD noise, we show that the noises induced by our method are random-direction, conflict-free and biased, which may help the model converge to a flat minimum with superior stability and enforce the model to produce conservative predictions on Out-of-Distribution instances. Extensive experimental results on benchmark datasets with various types of noisy labels demonstrate that the proposed method not only enhances the performance of many existing robust algorithms but also achieves significant improvement on Out-of-Distribution detection tasks even in the label noise setting.


Introduction
The success of deep neural networks (DNNs) heavily relies on a large number of training instances with fully accurate labels [16]. In real-world applications, it is expensive and time-consuming to collect such large-scale datasets. To alleviate this problem, some surrogate methods are commonly used to improve labelling efficiency, such as online queries [3] and crowdsourcing [74]. These cheap but imperfect methods usually suffer from unavoidable noisy labels that can be easily memorized by Deep Neural Networks (DNNs), leading to poor generalization performance [2,79]. Therefore, designing robust algorithms against noisy labels is of great practical importance.
Label noise can be separated into two categories: 1) closed-set noise, where instances with noisy labels have true class labels within the noisy label set [15,24,50,51,60,63]. 2) open-set noise, where instances with noisy labels have some true class labels outside the noisy label set [30,54,61,67]. In the existing literature, open-set noises are always considered to be harmful to the training of DNNs like closed-set noises [30,61,54,67]. For example, ILON [61] reduces the effect of open-set noises by using noisy label detector and contrastive loss to pull away noisy samples from clean samples. Extended T [67] uses an extended transition matrix to mitigate the impacts of open-set noises and EvidentialMix [54] filters out the open-set examples by modelling the loss distribution. Therefore, there arises a question to be answered: Do open-set noises always play a negative role in the training?
In this work, we answer this question by experiments with an open-set auxiliary dataset. We empirically show that additional open-set noises can be harmless to generalization if the number of open-set noises is sufficiently large. More surprisingly, we observe that the open-set noises can even benefit the robustness of neural networks against noisy labels from the original training dataset, while the closed-set counterpart could not. Based on the "insufficient capacity" hypothesis [2], an intuitive explanation for this phenomenon is that increasing the number of open-set auxiliary samples slows down the fitting of inherent noises. In other words, fitting open-set label noises consumes extra capacity of the network, thereby reducing the memorization of inherent noisy labels. The details of experimental results are presented in Subsection 2.2.
Inspired by the above observations, we propose an effective and complementary method to improve robustness against noisy labels. The high-level idea is to introduce Open-set samples with Dynamic Noisy Labels (ODNL) into training. In each epoch, we assign random labels to instances of open-set auxiliary dataset by uniformly sampling from the label set. In other words, the label of each open-set sample is independently random and is constantly changing over training epochs. In this manner, ODNL consistently produce "benign hard examples" to consume the extra representation capacity of neural networks, thereby preventing the neural network from overfitting inherent noisy labels.
To further understand the effect of ODNL, we provide an analysis from the lens of SGD noise [7,65], and formalize the differences and connections between our regularization and related methods. From the analysis, we show that the SGD noises induced by our method are 1) random-direction, helping the model converge to a flat minimum; 2) conflict-free, leading to good stability for the training; 3) biased, enforcing the model to produce more conservative predictions on OOD examples. In this manner, our method not only improves the robustness against noisy labels, but also provides benefits for OOD detection performance, which is also essential for the deployment of deep learning in safety-critical applications.
To the best of our knowledge, we are the first to explore the benefits of open-set auxiliary dataset in learning with noisy labels. To verify the efficacy of our regularization, we conduct extensive experiments on both simulated and real-world noisy datasets, including CIFAR-10, CIFAR-100 and Clothing1M datasets. In summary, the proposed method can be easily adapted for a wide range of existing algorithms to prevent overfitting noisy labels. Furthermore, experimental results validate that our regularization could also achieve impressive performance for detecting OOD examples, even if the labels of training dataset are noisy.

Problem setting and motivation
In this section, we first briefly introduce the problem setting of learning from training data with noisy labels. Then we start in Subsection 2.2 with an observational study highlighting the effect of open-set noisy examples from the auxiliary dataset on generalization and robustness against noisy labels.

Preliminaries: learning with inherent noisy labels
In this work, we consider multi-class classification with k classes. Let X ⊂ R d be the feature space and Y = {1, . . . , k} be the label space, we suppose the training dataset with N samples is given as , where x i ∈ X is the i-th instance sampled from a certain data-generating distribution D train in an i.i.d. manner and y i ∈ {1, . . . , k} represents its observed label that might be different from the ground truth. To prevent confusion, the noisy labels in the training dataset are referred to as inherent noisy labels. A classifier is a function that maps the feature space to the label space f : X → R k with trainable parameter θ ∈ R p . In this paper, we consider the common case where the function f is a DNN with the softmax output layer. The objective function of standard training can be represented as a cross-entropy loss: where f (x i ; θ) denotes the output of the classifier f with θ for x i and e yi := (0, . . . , 1, . . . , 0) ∈ {0, 1} k is a one-hot vector and only the y i -th entry of e yi is 1.  D train . Therefore, they are also called Out-of-Distribution (OOD) instances. In real applications, it is easy to get such an auxiliary dataset, which is commonly used in OOD detection tasks [20,48].

Motivation: the role of open-set label noise in generalization and robustness
To  [20] as the open-set auxiliary dataset and CIFAR-5m 3 [21,43] as the closed-set auxiliary dataset. Specifically, all images that belong to CIFAR classes are removed in 300K Random Images so that D train and D out are disjoint. More training details can be found in Appendix B.
Learning with clean labels. Fig. 1a presents the results on clean training set of CIFAR-10, training with the open/closed-set noisy auxiliary dataset. The results show that open-set noisy labels from the auxiliary dataset become harmless gradually with the increase of the sample size of the auxiliary dataset, while closed-set noisy labels consistently deteriorate the generalization performance. Furthermore, we observe that closed-set noisy labels result in unstable training shown by the large standard deviations in the figure. It is worthy to note that open-set noisy labels do not encounter this issue.
Learning with inherent noisy labels. Fig. 1b presents the results on CIFAR-10 with inherent noisy labels, training with the open/closed-set noisy auxiliary dataset. Surprisingly, we can observe that as we increase the sample size of the open-set noisy dataset, the open-set noises can even improve the generalization performance beyond standard training without the noisy auxiliary dataset. In this case, closed-set noisy auxiliary dataset still plays a negative role in the training.
An intuitive interpretation. From the results, one can hypothesize that open-set noisy labels in the auxiliary dataset would be harmless and can even benefit the robustness against inherent noisy labels, if the sample size of the auxiliary dataset is large enough. This phenomenon is somewhat counter-intuitive since one would expect that open-set noisy labels should have been "poison" to the training of network, like closed-set noises [30,54,61,67]. Here we provide an intuitive interpretation from "insufficient capacity" [2]: increasing the number of examples while keeping representation capacity fixed would increase the time needed to memorize the data set. Hence, the larger the size of auxiliary dataset is, the more time it needs to memorize the open-set noises in the auxiliary dataset as well as the inherent noises in the training set, relative to clean data. This interpretation is supported by our plot of training losses of inherent noises with different-sized auxiliary datasets in Fig. 1c, where increasing the number of open-set auxiliary samples slows down the fitting of inherent noises.

Method: Open-set regularization with Dynamic Noisy Labels
Motivated by the previous observations, we propose to utilize open-set auxiliary dataset to further improve the robustness against inherent noisy labels. This approach can be considered as an auxiliary task that continuously consumes the extra capacity of neural networks but does not interfere with learning patterns from clean data. Such auxiliary tasks will prevent deep networks from overfitting noisy data.
Dynamic Noisy Labels. To maintain the effectiveness of the auxiliary task during training, an ideal method is to use an auxiliary dataset with unlimited instances based on "insufficient capacity" [2]. In this way, the network would always try to fit a stream of new instances sequentially in an online learning setting. However, it is unrealistic to provide such an auxiliary dataset in real scenarios.
To relieve the requirement on sample size, we propose an alternative approach called Dynamic Noisy Labels. For each open-set instance x i , a noisy label y i is uniformly sampled from the label set {1, . . . , k} independently, regardless of the instance x i . More importantly, the generated labels are not fixed and would be changed dynamically over training epochs. In this way, the auxiliary task becomes an "impossible mission", continuously consuming the extra capacity of network. Without Dynamic Noisy Labels, the effectiveness of the auxiliary task would be degraded gradually during training, because deep network can easily memorize the finite open-set samples, even though the labels of those samples are randomly generated [2,79].
Training Objective. For the original training dataset, we use the standard cross entropy as the training objective function: For the auxiliary dataset, we enforce the outputs to be consistent with dynamic noise labels, uniformly sampled from the label set in each epoch. Therefore, the training objective function is as follows: where U k represents a discrete uniform distribution on the label set {1, . . . , k}.
Combining the original training dataset and the auxiliary dataset in the training, now we can formalize ODNL as minimizing the final objective: where η denotes the trade-off hyperparameter. The details of ODNL are provided in Algorithm 1.
Extensions to Other Robust Training Algorithms. It is worthy to note that our regularization is a general method and can be easily incorporated into existing robust training algorithms, including sample selection [15,63], loss correction [50], robust loss function [80,62], etc. Given the original learning objective L robust of the robust methods and the hyperparameter η, we can formalize the final objective as:

Understanding from SGD noise
To further understand the effect of our regularization, we provide an analysis from the lens of SGD noise. Following SLN [7], here we consider an original training sample (x, y) and an open-set sample x at each SGD step for convenience. In parameter updates, our regularization adds noises to the original gradients with the open-set noisy sample x and dynamic noisy label y ∼ U k : Perform gradient descent on f with L total from Equation (4); 6: end for where denotes the cross entropy loss for single sample. For convenience, we omit the hyperparameter η and use f , f to indicate the model output f (x; θ) for the original training sample and the model output f ( x; θ) for the open-set noisy sample. Following the standard notation of the Jacobian matrix, The proofs of the following propositions are presented in Appendix A.
The effects of SGD noise. From Proposition 1, we find that our regularization induces SGD noise sampled from a uniform distribution over The effects of the noises can be analyzed from three aspects: • Firstly, our regularization yields SGD noises with random direction, uniformly sampled from the gradients of the open-set noisy sample. In this way, the SGD noises make the training difficult to converge and help escape local minima [4,22,25,26,46]. Without the noises, the model would quickly converge to a local minimum because the optimization of parameters always follows the direction of gradient descent during the training.
• Secondly, the noises z are independently random to the gradients ∇ θ (f , y) from the original training example as the noises are conducted with the open-set sample x from D out . If closed-set samples are used here, the generated noises might be conflicted with the original gradients, leading to unstable training and limiting the improvement from noises. This argument is clearly supported by the experimental results in Fig. 1 and Fig. 3a.
• Finally, the expected value of noise is not zero so the noise is biased. In particular, the expected value of noise enforces the model f to produce more conservative predictions on Out-of-Distribution samples. Consequently, this feature would lead to performance improvement on OOD detection tasks as shown in Subsection 5.4.
To further demonstrate the importance of open-set samples in the auxiliary dataset, we conduct experiments with synthetic auxiliary datasets consisting of convex combinations x of open-set samples x and closed-set samples x with a hyperparameter α: Here, we use 300K Random Images [20] and CIFAR-5m [21,43] as the open-set and closed-set dataset. More training details can be found in Appendix B. The results in Fig. 3a show that using open-set auxiliary dataset achieves better performance on robustness against inherent noises.
Relations to label randomization methods. Label randomization is also applied in some existing work in the context of deep learning [7,36,71]. For example, DisturbLabel [71] intentionally generates incorrect labels on a small fraction of training data to improve generalization performance under the settings of clean labels, interpreted as an implicit model ensemble. It can be seen as using a part of training data as a closed-set auxiliary dataset. In such a way, DisturbLabel can be interpreted as a special case of our method with a closed-set auxiliary dataset.
To mitigate the issue of label noise, Stochastic Label Noise (SLN) [  by SLN are given as follows:∇ where σ y > 0, z y ∈ R k , z y ∼ N (0, I k×k ). Proposition 2 (Proposition 3 in Chen et al. [7]). For the cross-entropy loss, Eq. (7) in SLN induces From Proposition 2, we show that SLN induces SGD noises that obey a Gaussian distribution, where the standard deviation can be adjusted by the hyperparameter σ y . Therefore, SLN and our ODNL can be considered as different variants of SGD noise, helping the network converge to a flat minimum, which is expected to provide better generalization performance [4,22,25,26,46]. The effects are explicitly presented in Fig. 2c and Fig. 2d. Moreover, the results in Fig. 3b illustrate that our method could further improve the performance of SLN 4 under various types of label noise, showing the complementarity between SLN and our method.
Relations to Outlier Exposure. Outlier Exposure (OE) [20] is an effective method designed for OOD detection tasks. Specifically, OE also makes use of a large, unlabeled auxiliary dataset sampled from D out , and regularizes the softmax probabilities to be a uniform distribution for out-ofdistribution data in the auxiliary dataset. In such manner, OE train the model to discover signals and learn heuristics to detect whether a query is sampled from D train or D out . The training objective is to minimize:

Experiments
In this section, we verify the effectiveness of our method on three benchmark datasets: CIFAR-10/100 and Clothing1M. We show that ODNL could not only significantly improve the robustness against various types of inherent noisy labels in standard training, but also enhance the performance of existing robust training techniques. Then we perform a sensitivity analysis to validate the effect of η. Finally, we conduct experiments to evaluate the performance of ODNL on OOD detection task.

Setups
We comprehensively verify the utility of ODNL 5 on different types of label noise, including symmetric noise, asymmetric noise [80], instance-dependent noise [6] and open-set noise [61] synthesized on CIFAR-10/100 and real-world noise on Clothing1M [70]. Symmetric noise assumes each label has the same probability of flipping to any other classes. We uniformly flip the label to other classes with an overall probability 20% and 50%. Asymmetric noise assumes labels might be only flipped to similar classes [7,50,80]. Noisy labels are generated by mapping TRUCK → AUTOMOBILE, BIRD → AIRPLANE, DEER → HORSE, and CAT ↔ DOG with probability 40% for CIFAR-10.
For CIFAR-100, we flip each class into the next circularly with probability 40%. Instance-dependent noise assumes the mislabeling probability is dependent on each instance's input features [6,7,68]. We use the instance-dependent noise from PDN [68] with a noisy rate 40%, where the noise is synthesized based on the DNN prediction error. Open-set noise contains samples that do not belong to the label set considered in the classification task [61]. We generate open-set noises on CIFAR-10 by randomly replacing 40% of its training images with images from CIFAR-100.
We perform training with WRN-40-2 [77] on CIFAR-10 and CIFAR-100. The network is trained for 200 epochs using SGD with a momentum of 0.9, a weight decay of 0.0005. In each iteration, both the batch sizes of the original dataset and the open-set auxiliary dataset are set as 128. We set the initial learning rate as 0.1, and reduce it by a factor of 10 after 80 and 140 epochs. We use 5k noisy samples as the validation to tune the hyperparameter η in {0.1, 0.5, 1, 2.5, 5}, then train the model on the full training set and report the average test accuracy in the last 5 epochs. All experiments are repeated five times with different seeds and more training details can be found in Appendix B. We use 300K Random Images [20] as the open-set auxiliary dataset and more discussions on the different choices of open-set auxiliary dataset are presented in Appendix C.

CIFAR-10 and CIFAR-100
On CIFAR-10 and CIFAR-100, we verify that ODNL can boost existing robust training methods by integrating ODNL with the following methods: 1) Decoupling [40], which updates the parameters only using instances that have different predictions from two classifiers. 2) F-correction [50], which corrects the prediction by the label transition matrix. As the setting in F-correction, we first train a standard network to estimate the transition matrix Q. 3) PHuber-CE [41], which uses a composite loss-based gradient clipping, a variation of standard gradient clipping for label noise robustness. 4) Co-teaching [15], which trains two networks simultaneously and cross-updates parameters of peer networks. 5) JoCoR [63], which trains two networks simultaneously and reduces the divergence between the two classifiers by explicit regularization. The average test accuracy with WRN-40-2 [77] are reported in Table 1 and To further illustrate the influence of η, we present a sensitivity analysis on CIFAR-10 with symmetric-40% noisy labels in Fig. 4. Specifically, we highlight the differences in the trend of test accuracy after the second decay of learning rate at the 140th epoch. We can observe that with a large η, the decreasing trend of test accuracy caused by inherent noisy labels is nearly eliminated. The result verifies that our regularization is an effective method to reduce the overfitting of inherent noisy labels.  [53] and remove all classes related to clothes. Other training details strictly follow the settings of DivideMix [33], presented in Appendix B. As shown in Table 3, best denotes the score of the epoch where the validation accuracy is optimal, and last denotes the scores at the last epoch. The vanilla ODNL outperforms many simple baselines, and it can further improve the performance of SLN and DivideMix. The results show that our method is applicable for large-scale real-world scenarios.

OOD detection with noisy labels
Out-of-distribution (OOD) detection is an essential problem for the deployment of deep learning especially in safety-critical applications [20,35]. From the analysis in Section 4, we know that our regularization should have a similar regularization effect to OE [20], which enforces the model to give conservative outputs on OOD examples. Here, we conduct experiments on CIFAR-10 with symmetric-40% noisy labels to verify the advantage of our method on OOD detection tasks even in a weakly supervised setting. We expect the trained models could perform well in the OOD detection task, even if the labels of training dataset are noisy. Experiments on CIFAR-10 with clean labels are also conducted and the results can be found in Appendix C. The training settings are the same as those in subsection 5.2. Following OE [20], we consider the maximum softmax probability (MSP) baseline [18], which gives the maximum of softmax output as the OOD score. For the OOD test datasets, we use three types of noises and seven common benchmark datasets: Gaussian, Rademacher [20], Blobs [20], Textures [9], SVHN [45], CIFAR100, Places365 [82], LSUN-Crop [75], LSUN-Resize [75], and iSUN [72]. We measure the following metrics: (1) Table 4 presents the test accuracy and the average performance over the three types of noises and seven OOD test datasets. We can observe that our method achieves impressive improvement on both the test accuracy and the detection performance, while OE does not bring benefit on the generalization performance. The results can further serve as evidence for the analysis in Section 4.
Utilizing auxiliary dataset. To the best of our knowledge, we are the first to explore the benefits of open-set auxiliary dataset in learning with noisy labels. Auxiliary dataset is also utilized in other problem settings of deep learning. For example, pre-training a network on the large ImageNet database [53] can produce general representations that are useful in many fine-tuning applications [78]. Representation learned from images scraped from search engines and photo-sharing websites could improve objective detection performance [8,39]. OE uses an auxiliary dataset to teach the network better representations for OOD detection [20]. To improve robustness against adversarial attack, a popular method is to train on adversarial examples which can be seen as a generated auxiliary dataset [14]. Unlabelled data is also shown to be beneficial for the adversarial robustness [5,59]. Note that the unlabelled data for improving adversarial robustness is required to be in-distribution, our method utilizes out-of-distribution instances to improve robustness against noisy labels. Recently, OAT [31] designed to use out-of-distribution data for improving robustness against adversarial examples, a detailed comparison between OAT and our work is provided in Appendix D.
OOD detection. OOD detection is an essential building block for safely deploying machine learning models in the open world [18,20,35]. A common baseline for OOD detection is the softmax confidence score [18]. It has been theoretically shown that neural networks with ReLU activation can produce arbitrarily high softmax confidence for OOD inputs [17]. To improve the performance, previous methods have explored using artificially synthesized data from GANs [13] or unlabeled data [20,29] as auxiliary OOD training data. Energy scores are shown to be better for distinguishing inand out-of-distribution samples than softmax scores [35]. As a side effect, our method could also achieve superior performance in OOD detection tasks even if the labels of training dataset are noisy.

Conclusion and Discussion
In this paper, we proposed Open-set regularization with Dynamic Noisy Labels (ODNL), a simple yet effective technique that enhances many existing robust training methods to mitigate the issues of inherent noisy labels across various settings. To the best of our knowledge, our method is the first to utilize open-set auxiliary dataset in the problem of noisy labels. We show that ODNL can be considered as a variant of SGD noises that usually leads to a flat minimum. Extensive experiments show that ODNL can help improve the robustness against various types of noisy labels and it is applicable for large-scale real-world scenarios. Moreover, ODNL also brings a "side effect": it can improve the OOD detection performance, which is also essential for the deployment of deep learning in safety-critical applications. On the other hand, ODNL is computationally inexpensive and can be easily integrated into existing algorithms. Overall, our method is an effective and complementary approach for boosting robustness against inherent noisy labels.
In the future, the observations and analyses in this work can help understand the effect of open-set noises and inspire more specially designed methods using auxiliary dataset to combat label noise in deep learning. There are still some limitations in our current exploration and method as follows.
instance-dependent open-set noisy labels. In this work, we only consider open-set noisy labels that are independent from the instances. In real scenarios, an open-set sample with a noisy label might be similar to the clean samples in this class but still do not belong to this class. We may get different observations when we extend the setting to instance-dependent open-set noisy labels.
Synthesized auxiliary dataset. In this work, we use realistic open-set dataset in our method. It is also natural to ask whether synthesized data can be used to combating noisy labels. For example, one may generate adversarial examples with dynamic noisy labels to improve robustness against noisy labels.
B More details on experiment setup B.1 Datasets Auxiliary datasets. 300K Random Images [20] is a cleaned and debiased dataset with 300K natural images. In this dataset, Images that belong to CIFAR classes from it, images that belong to Places or LSUN classes, and images with divisive metadata are removed so that D train and D out are disjoint. We use the dataset as the open-set auxiliary dataset for experiments with CIFAR-10 and CIFAR-100.
In particular, For experiments with Clothing1M, ImageNet 2012 dataset [53] is used as the open-set auxiliary dataset and we remove all classes related to clothes. CIFAR-5m [21,43] is a dataset of 6 million synthetic CIFAR-10-like images. For experiments in Fig. 1 and Fig. 3a, we build the closed-set dataset by sampling class-balanced subsets from CIFAR-5m.
Blobs noises consist in algorithmically generated amorphous shapes with definite edges. Textures [9] is a dataset of describable textural images. SVHN dataset [44] contains 32 × 32 color images of house numbers. There are ten classes comprised of the digits 0-9. Places365 [82] consists in images for scene recognition rather than object recognition. LSUN [75] is another scene understanding dataset with fewer classes than Places365. Here we use LSUN-Crop and LSUN-Resize to denote the cropped and resized version of the LSUN dataset respectively. iSUN [72] is a large-scale eye tracking dataset, selected from natural scene images of the SUN database [69].

B.2 Setups
We conduct all the experiments on NVIDIA GeForce RTX 3090, and implement all methods with default parameters by PyTorch [49]. Following GCE [80] and SLN [7], we use 5k noisy samples as the validation set to tune the hyperparameters. We then train the model on the full training set and report the average test accuracy over the last 5 epochs.
For experiments in Fig. 1, Fig. 2, Fig. 3a and Fig. 3c. We use WRN-40-2 [77] and the network trains for 200 epochs with a dropout rate of 0.3, using SGD with a momentum of 0.9, a weight decay of 0.0005. In each iteration, both the batch sizes of the original dataset and the open-set auxiliary dataset are set as 128. Note that we compare the performance of training with different-sized auxiliary datasets in Fig. 1, the batch size of the auxiliary dataset is fixed as 128 in all cases. We set the initial learning rate as 0.1, and reduce it by a factor of 10 after 80 and 140 epochs. For drawing loss landscapes in Fig. 2, we use the technique from the loss-landscapes library [32]. For experiments in Fig. 3c, we use 40% noise rate in all types of inherent noisy labels.
For experiments in Fig. 3b. We use the same setting as that of SLN [7]. Specifically, we train WRN-28-2 [77] for 300 epochs using SGD with learning rate 0.001, momentum 0.9, weight decay 0.0005 and a batch size of 128. For the hyperparameter σ of SLN, we use σ = 1 for symmetric noise and σ = 0.5 otherwise. Here, we use 40% noise rate in all types of inherent noisy labels, following the official implementation of SLN.
For experiments on Clothing1M. We use the same setting as that of DivideMix [33]. Specifically, we use ResNet-50 with ImageNet pretrained weights for 80 epochs. The initial learning rate is set as 0.002 and reduced by a factor of 10 after 40 epochs. For each epoch, we sample 1000 mini-batches from the training data while ensuring the labels are balanced.
Method-specific hyperparameters. The backbone and training settings of the compared methods are not unified in previous paper and we reimplement all methods in the same backbone for a fair comparison. For experiments on CIFAR-10/100, we set λ = 0.5 for JoCoR [63]. For experiments on Clothing1M, we use the default hyperparameters for DivideMix: M = 2, T = 0.5, τ = 0.5, λ u = 0, α = 0.5, and we set the standard deviation of SLN as σ = 0.2.
Guideline for tuning η. We tune the hyperparameter η following a guideline that has been widely used in existing literature like SLN [7]. Given a new dataset with unknown noise, we suggest quickly searching the best value of η (denoted as η o ) based on the binary search using the validation accuracy throughout training. 1) If we observe a decrease of validation accuracy at the late stage of training, it implies overfitting and η < η o . 2) Otherwise, we have η ≥ η o . Based on 1) and 2), we can conduct a binary search to quickly find the best value of η even if one would like to search η in a very detailed range. The best values of η for vanilla ODNL are reported in Table 5.  Our regularization method improves robustness against label noise for more than just WRN architectures. Table 6 shows that ODNL also improves performance under various types of noisy labels with ResNet-18 [16] and VGG-11 [56]. In particular, we set weight decay as 0.001 for experiments with ResNet-18 and VGG-11.

C.2 Different choices of open-set auxiliary datasets.
In the experiments in Subsection 2.2, we found that the sample size of auxiliary dataset is important for the generalization performance when the labels of open-set samples are fixed. To relieve this requirement, we introduce dynamic noisy labels to strengthen our method in Section 3. In particular, our method can use only 50,000 examples from 300K Random Images to achieve comparable performance with using all examples from this dataset. Moreover, we show that using simple Gaussian noises as auxiliary dataset can also improve robustness against noisy labels, but do not work as well as real data. For example, in the case of symmetric-40% noise on CIFAR-10, using ODNL with simple Gaussian noises achieves 80.24% accuracy while using ODNL with 300K Random Images achieve 82.03%. In addition to size and realism, we also find that diversity of auxiliary dataset is not important for robustness against noisy labels, while OE [20] claims that it is an important factor for OOD detection. We conduct experiments on CIFAR-10 with symmetric-40% noises and use the subset of CIFAR-100 with different number of classes as auxiliary dataset. The sample size of these subsets are fixed as 5,000. The results of using different subsets are very close, achieving 81.56% test accuracy.
C.3 Detailed results for OOD detection. Table 7 presents the detailed results of OOD detection performance on CIFAR-10 under symmetric-40% noisy labels with various OOD test datasets. From the results, we show that our method can outperform OE [20] in OOD detection tasks when the labels of training dataset are noisy. Table 8 presents the detailed results of OOD detection performance on CIFAR-10 under clean labels with various OOD test datasets. We can observe that our method achieve comparable performance to OE [20]. The results also support the analysis in Section 4.

D Discussion
Relations to OAT. OAT [31] aims to use out-of-distribution data to improve generalization in the context of adversarial robustness. While OAT is relevant to our work, we note that there are key differences in both technique and insight perspectives: • Technique: OAT regularizes the softmax probabilities to be a uniform distribution for out-ofdistribution data, which is the same as Outlier Exposure [20]. As analyzed in Section 4, the regularization in OE and OAT cannot improve the robustness against noisy labels, because it does not induce dynamic noises and the optimization of parameters always follows the direction of gradient descent. • Insight: OAT shows that OOD data samples share the same undesirable features as those of the in-distribution data so that these samples could also be used to remove the influence of undesirable features in adversarial robustness. In our work, we show that open-set noisy labels could be harmless and even benefit the robustness against inherent noisy labels. We give an intuitive interpretation for the phenomena by "insufficient capacity" and understand the effects of open-set noises from the perspective of SGD noises.