{'fraction_non_alphanumeric': 0.038130078441028524, 'fraction_numerical': 0.017024868971824546, 'mean_word_length': 4.942725752508361, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning

Deep learning with noisy labels is a challenging task. Recent prominent methods that build on a specific sample selection (SS) strategy and a specific semi-supervised learning (SSL) model achieved state-of-the-art performance. Intuitively, better performance could be achieved if stronger SS strategies and SSL models are employed. Following this intuition, one might easily derive various effective noisy-label learning methods using different combinations of SS strategies and SSL models, which is, however, reinventing the wheel in essence. To prevent this problem, we propose SemiNLL, a versatile framework that combines SS strategies and SSL models in an end-to-end manner. Our framework can absorb various SS strategies and SSL backbones, utilizing their power to achieve promising performance. We also instantiate our framework with different combinations, which set the new state of the art on benchmark-simulated and real-world datasets with noisy labels.


Introduction
Deep Neural Networks (DNNs) have achieved great success in different computer vision problems, e.g., image classification [20], detection [36], and semantic segmentation [27]. Such success is demanding for large datasets with clean human-annotated labels. However, it is costly and time-consuming to correctly label massive images for building a large-scale dataset like ImageNet [5]. Some common and less expensive ways to collect large datasets are through online search engines [37] or crowdsourcing [51], which would, unfortunately, bring noisy labels to the collected datasets. Besides, an in-depth study [52] showed that deep learning with noisy labels can lead to severe performance deterioration. Thus, it is crucial to alleviate the negative effects caused by noisy labels for training DNNs.
A typical strategy is to conduct sample selection (SS) and to train DNNs with selected samples [12,18,40,45,50]. Since DNNs tend to learn simple patterns first before fitting noisy samples [3], many studies utilize the small-loss trick, where the samples with smaller losses are taken as clean ones. For example, Co-teaching [12] leverages two networks to select small-loss samples within each mini-batch for training each other. Later, Yu et al. [50] pointed out the importance of the disagreement between two networks and proposed Co-teaching+, which updates the two networks using the data on which the two networks hold different predictions. By contrast, JoCoR [45] proposes to reduce the diversity between two networks by training them simultaneously with a joint loss calculated from the selected small-loss samples. Although these methods have achieved satisfactory performance by training with selected small-loss samples, they simply discard other large-loss samples which may contain potentially useful information for the training process.
To make full use of all given samples, a prominent strategy is to consider selected samples as labeled "clean" data and other samples as unlabeled data, and to perform semi-supervised learning (SSL) [2,4,22,42]. Following this strategy, SELF [32] detects clean samples by progressively removing noisy samples whose selfensemble predictions of the model do not match the given labels in each iteration. With the selected labeled and unlabeled data, the problem becomes an SSL problem, and a Mean-Teacher model [42] can be trained. Another recent method, DivideMix [26], leverages Gaussian Mixture Model (GMM) [34] to distinguish clean (labeled) and noisy (unlabeled) data, and then uses a strong SSL backbone called MixMatch [4]. DivideMix achieves state-of-the-art results across different benchmark datasets.
As shown above, both methods rely on a specific SS strategy and a specific SSL model. The two components play a vitally important role for combating label noise, and stronger components are expected to achieve better performance. This motivates us to investigate a general algorithmic framework that can leverage various SS strategies and SSL models. In this paper, we propose SemiNLL, which is a versatile framework to bridge the gap between SSL and noisy-label learning (NLL). Our framework can absorb various SS strategies and SSL backbones, utilizing their power to achieve promising performance. Guided by our framework, one can easily instantiate a specific learning algorithm for NLL, by specifying a commonly used SSL backbone with an SS strategy. The key contributions of our paper can be summarized as follows: • To avoid reinventing the wheel for NLL using SSL algorithms, we propose a versatile framework that can absorb various SS strategies and SSL algorithms. Our framework is advantageous since better performance would be achieved if stronger components (including the ones proposed in the future) are used.
• To instantiate our framework, we propose DivideMix+ by replacing the epoch-level selection strategy of DivideMix [26] with a mini-batch level one. We also propose GPL, another instantiation of our framework that leverages a two-component Gaussian mixture model [26,34] to select labeled (unlabeled) data and uses Pseudo-Labeling [2] as the SSL backbone.
• We conduct extensive experiments on benchmark-simulated and real-world datasets with noisy labels. Empirical results show that the stronger SS strategies and SSL backbones we use, the better performance SemiNLL could achieve. In addition, our instantiations, DivideMix+ and GPL, outperform other stateof-the-art noisy-label learning methods.

Related work
In this section, we briefly review several related aspects on which our framework builds.

Learning with noisy labels
For NLL, most of the existing methods could be roughly categorized into the following groups: Sample selection. This family of methods regards samples with small loss as "clean" and trains the model only on selected clean samples. For example, self-paced MentorNet [18], or equivalently self-teaching, selects small-loss samples and uses them to train the network by itself. To alleviate the sample-selection bias in self-teaching, Han et al. [12] proposed an algorithm called Co-teaching [12], where two networks choose the next batch of data for each other for training based on the samples with smaller loss values. Co-teaching+ [50] bridges the disagreement strategy [29] with Co-teaching [12] by updating the networks over data where two networks make different predictions. In contrast, Wei et al. [45] leveraged the agreement maximization algorithm [21] by designing a joint loss to train two networks on the same mini-batch data and selected small-loss samples to update the parameters of both networks. The mini-batch SS strategy in our framework belongs to this direction. However, instead of ignoring the large-loss unclean samples, we just discard their labels and exploit the associated images in an SSL setup. Noise transition estimation. Another line of NLL is to estimate the noise transition matrix for loss correction [11,16,30,31,33,44,47]. Patrini et al. [33] first estimated the noise transition matrix and trained the network with two different loss corrections. Hendrycks et al. [16] proposed a loss correction technique that utilizes a small portion of trusted samples to estimate the noise transition matrix. However, the limitation of these methods is that they do not perform well on datasets with a large number of classes.
Other deep learning methods. Some other interesting and promising directions for NLL include metalearning [8,39] based, pseudo-label estimation [24] based, and robust loss [7,10,28,43,48,54] based approaches. For meta-learning based approaches, most studies fall into two main categories: training a model that adapts fast to different learning tasks without overfitting to corrupted labels [9,25], and learning to reweight loss of each mini-batch to alleviate the adverse effects of corrupted labels [35,38,55]. Pseudo-label estimation based approaches reassign the labels for noisy samples. For example, Joint-Optim [41] corrects labels during training and updates network parameters simultaneously. PENCIL [49] proposes a probabilistic model, which can update network parameters and reassign labels as label distributions. The family of pseudo-label estimation has a close relationship with semi-supervised learning [13,24,41,49]. Robust loss based approaches focus on designing loss functions that are robust to noisy labels.

Semi-supervised learning
SSL methods leverage unlabeled data to provide additional information for the training model. A line of work is based on the concept of consistency regularization: if a perturbation is given to an unlabeled sample, the model predictions of the same sample should not be too different. Laine and Aila [22] applied consistency between the output of the current network and the exponential moving average (EMA) of the output from the past epochs. Instead of averaging the model outputs, Tarvainen and Valpola [42] proposed to update the network on every mini-batch using an EMA of model parameter values. Berthelot et al. [4] introduced a holistic approach that well combines MixUp [53], entropy minimization, and consistency regularization. Another line of SSL is pseudo-labeling, the objective of which is to generate pseudo-labels for unlabeled samples to enhance the learning process. Recently, Arazo et al. [2] proposed a method to improve previous pseudo-labeling methods [17,24] by adding MixUp augmentation [53] and setting a minimum number of labeled samples per mini-batch to reduce accumulated error of wrong pseudo-labels.

Combination of SS and SSL
Some previous studies that combine a specific SS strategy and a specific SSL backbone could be regarded as special cases in our framework. Ding et al. [6] used a pre-trained DNN on the noisy dataset to select labeled samples. In the SSL stage, Temporal Ensembling [22] was used to handle labeled and unlabeled data. Nguyen et al. [32] proposed a progressive noise filtering mechanism based on the Mean-Teacher model [42] and its self-ensemble prediction. Li et al. [26] used a Gaussian Mixture Model (GMM) to divide noisy and clean samples based on their training losses and fitted them into a recent SSL algorithm called MixMatch [4]. Each specific component used in these methods has its own pros and cons. This motivates us to propose a versatile framework that can build on a variety of SS strategies and SSL backbones. In other words, all the methods mentioned in this subsection could be taken as instantiations of our framework.  Figure 1: The schematic of SemiNLL. First, each mini-batch of data is forwarded to the network to conduct SS, which divides the original data into the labeled/unlabeled sets. Second, labeled/unlabeled samples are used to train the SSL backbone to produce accurate model output.

4:
Fetch mini-batch D n from D train ;

5:
end for 8: end for Output: f θ

The overview of SemiNLL
In this section, we present SemiNLL, a versatile framework of learning with noisy labels by SSL. The idea behind our framework is that we effectively take advantage of the whole training set by trusting the labels of undoubtedly correct samples and utilizing only the image content of potentially corrupted samples. Previous sample selection methods [12,18,45,50] train the network only with selected clean samples, and they discard all potentially corrupted samples to avoid the harmful memorization of DNNs caused by the noisy labels of these samples. In this way, the feature information contained in the associated images might be discarded without being exploited. Our framework, alternatively, makes use of those corrupted samples by ignoring their labels while keeping the associated image content, transforming the NLL problem into an SSL setup. The mechanism of SSL that leverages labeled data to guide the learning of unlabeled data naturally fits well in training the model with the clean and noisy samples divided by our SS strategy. The schematic of our framework is shown in Figure 1. We first discuss the advantages of the mini-batch SS strategy in our framework and then introduce several SSL backbones used in our framework.

Mini-batch sample selection
During the SS process, a hazard called confirmation bias [42] is worth noting. Since the model is trained using the selected clean (labeled) and noisy (unlabeled) samples, wrongly selected clean samples in this iteration may keep being considered clean ones in the next iteration due to the model overfitting to their labels. Most existing methods [26,32] divide the whole training set into the clean/noisy set on an epoch level. In this case, the learned selecting knowledge is incorporated into the SSL phase and will not be updated till the next epoch. Thus, the confirmation bias induced from those wrongly divided samples will accumulate within the whole epoch. To overcome this problem, our mini-batch SS strategy divides each mini-batch of samples into the clean subset X m and the noisy subset U m (Line 5 in Algorithm 1) right before updating the network using SSL backbones. In the next mini-batch, the network can know better to distinguish clean and noisy samples, alleviating the confirmation bias mini-batch by mini-batch.

SSL backbones
The mechanism of SSL that uses labeled data to guide the learning of unlabeled data fits well when dealing with clean/noisy data in NLL. The difference lies in an extra procedure, as introduced in Subsection 3.1, that divides the whole dataset into clean and noisy data. After the SS process, clean samples are considered labeled data and keep their annotated labels. The others are considered noisy samples, and their labels are discarded to be treated as unlabeled ones in SSL backbones. SemiNLL can build on a variety of SSL algorithms without any modifications to form an end-to-end training scheme for NLL. Concretely, we consider the following representative SSL backbones ranging from weak to strong according to their performance in SSL tasks: (i) Temporal Ensembling [22], where the model uses an exponential moving average (EMA) of label predictions from the past epochs as a target for the unsupervised loss. It enforces consistency of predictions by minimizing the difference between the current outputs and the EMA outputs.
(ii) MixMatch [4], which is a holistic method that combines MixUp [53], entropy minimization, consistency regularization, and other traditional regularization tricks. It guesses low-entropy labels for augmented unlabeled samples and mixes labeled and unlabeled data using MixUp [53].
In the next section, we will instantiate our framework by applying specific SS strategies and SSL backbones to the select and semi placeholders in Algorithm 1.

Instantiation 1: DivideMix+
In Algorithm 1, if we (i) specify the select placeholder as a GMM [34], (ii) specify the semi placeholder as MixMatch [4] mentioned in Subsection 3.2, and (iii) train two independent networks wherein each network selects clean/noisy samples in the SS phase and predicts labels in the SSL phase for the other network, then our framework is instantiated into a mini-batch version of DivideMix [26]. Specifically, during the SS process, DivideMix [26] fits a two-component GMM to the loss i of each sample using the Expectation-Maximization technique and obtains the posterior probability of a sample being clean or noisy:

Mini-batch n+1
Mini-batch n Mini-batch n-1  where k = 0 (1) denotes the clean (noisy) set. During the SSL phase, the clean set X e and the noisy set U e are fit into an improved MixMatch [4] strategy with label co-refinement and co-guessing. As shown in Figure 2b, the SS strategy (GMM) of DivideMix [26] is conducted on an epoch level. Since X e and U e are updated only once per epoch, the confirmation bias induced from the wrongly divided samples will be accumulated within the whole epoch. However, our mini-batch version, which is called DivideMix+ (Figure 2c), divides each mini-batch of data into a clean subset X m and a noisy subset U m , and updates the networks using the SSL backbone right afterwards. In the next mini-batch, the updated networks could better distinguish clean and noisy samples.

Instantiation 2: GPL
Intuitively, the choice of stronger SS strategies and SSL models would achieve better performance based on our framework. Thus, we still choose GMM to distinguish clean and noisy samples due to its flexibility in the sharpness of distribution [26]. As for the SSL backbone, we choose the strongest Pseudo-Labeling [2] introduced in Subsection 3.2. We call this instantiation GPL (GMM + Pseudo-Labeling). Note that we do not train two networks in GPL as in DivideMix [26] and DivideMix+. To our understanding, training two networks simultaneously might provide significant improvements in performance. However, this is outside the scope of this paper, since our goal is to demonstrate the versatility of our framework.

Self-prediction divider
Inspired by SELF [32], we introduce the self-prediction divider, a simple yet effective SS strategy which leverages the information provided by the network's own prediction to distinguish clean and noisy samples. Based on the phenomenon that DNN's predictions tend to be consistent on clean samples and inconsistent on noisy samples in different training iterations, we select the correctly annotated samples via the consistency between the original label set and the model's own predictions. The self-prediction divider determines potentially clean samples in a mini-batch if the samples' maximal likelihood predictions of the network match their annotated labels. Specifically, the samples are divided into the labeled set only if the model predicts the annotated label to be the correct class with the highest likelihood. The others are considered noisy samples, and their labels will be discarded to be regarded as unlabeled ones in SSL backbones. Compared to previous small-loss SS methods [12,45,50], which depend on a known noise ratio to control how many small-loss samples should be selected in each training iteration, self-prediction divider does not need any additional information to perform SS strategy where the clean subset and the noisy subset are determined by the network itself. Concretely, we instantiate three learning algorithms by combining our self-prediction divider (SPD) with three SSL backbones introduced in Subsection 3.2 and denote them as SPD-Temporal Ensembling, SPD-MixMatch, and SPD-Pseudo-Labeling, respectively.

Effects of the two components
This section demonstrates the effects of SS strategies and SSL backbones in our framework. To prove that a more robust SS strategy can boost performance for our framework, we propose DivideMix- (Figure 2a) Table 2: Average test accuracy (%) and standard deviation (5 runs) of all the methods in various datasets under asymmetric label noise. The best accuracy is bold-faced. The second-best accuracy is underlined.
divider is supposed to be weaker than GMM, DivideMix-is expected to achieve lower performance than DivideMix [26]. To prove the effectiveness of the SSL backbone, we remove it after the SS process and only update the model using the supervised loss calculated from the clean samples. We will give detailed discussions in Subsection 5.3 and Subsection 5.4.

Experiments
In this section, we first compare two instantiations of our framework, DivideMix+ and GPL, with other state-of-the-art methods. We also analyze the effects of SS strategies by comparing DivideMix-, DivideMix [26], and DivideMix+, then analyze the effects of SSL backbones by combining three representative SSL methods with our self-prediction divider. More information of our experiments can be found in supplementary materials.

Experiment setup
Datasets. We thoroughly evaluate our proposed DivideMix+ and GPL on five datasets, including MNIST [23], FASHION-MNIST [46], CIFAR-10, CIFAR-100 [19], and Clothing1M [47]. MNIST and FASHION-MNIST contain 60K training images and 10K test images of size 28 × 28. CIFAR-10 and CIFAR-100 contain 50K training images and 10K test images of size 32 × 32 with three channels. According to previous studies [26,45,54], we experiment with two types of label noise: symmetric noise and asymmetric noise. Symmetric label noise is produced by changing the original label to all possible labels randomly and uniformly according to the noise ratio. Asymmetric label noise is similar to real-world noise, where labels are flipped to similar classes. Clothing1M is a large-scale real-world dataset that consists of one million training images from online shopping websites with labels annotated from surrounding texts. The estimated noise ratio is approximately 40% [47]. Network Structure and Optimizer. Following previous works [2,26,45,54], we use a 2-layer MLP for MNIST, a ResNet-18 [14] for FASHION-MNIST, the well-known "13-CNN" architecture [42] for CIFAR-10 and CIFAR-100, and an 18-layer PreAct Resnet [15] for Clothing1M. To ensure a fair comparison between the instantiations of our framework and other methods, we keep the training settings for MNIST, CIFAR-10, CIFAR-100, and Clothing1M as close as possible to DivideMix [26] and FASHION-MNIST close to GCE [54].
For FASHION-MNIST, the network is trained using stochastic gradient descent (SGD) with 0.9 momentum and a weight decay of 1 × 10 −4 for 120 epochs. For MNIST, CIFAR-10, and CIFAR-100, all networks are trained using SGD with 0.9 momentum and a weight decay of 5 × 10 −4 for 300 epochs. For Clothing1M, the momentum is 0.9, and the weight decay is 0.001. Baselines. We compare DivideMix+ and GPL with the following state-of-the-art algorithms and implement all methods by PyTorch on NVIDIA Tesla V100 GPUs.
(i) Coteaching [12], which trains two networks and cross-updates the parameters of peer networks.
(ii) GCE [54], which uses a theoretically grounded and easy-to-use loss function, the L q loss, for NLL.
(iii) F-correction [33], which corrects the prediction by the label transition matrix. As suggested by the authors, we first train a standard network using the cross-entropy loss to estimate the transition matrix.
(iv) M-correction [1], which models clean and noisy samples by fitting a two-component BMM and applies MixUp data augmentation [53].
(v) DivideMix [26], which divides clean and noisy samples by using a GMM on an epoch level and leverages MixMatch [4] as the SSL backbone.

Performance Comparison
The results of all the methods under symmetric and asymmetric noise types on MNIST, FASHION-MNIST, CIFAR-10, and CIFAR-100 are shown in Table 1 and Table 2. The results on Clothing1M are shown in Table 5.
Results on MNIST. DivideMix+ surpasses DivideMix across symmetric and asymmetric noise at all noise ratios, showing the effectiveness of the mini-batch SS strategy in our framework. M-correction performs well under low noise ratios. However, in the hardest symmetric 80% case, DivideMix+ achieves best test accuracy.
Results on FASHION-MNIST. FASHION-MNIST is quite similar to MNIST but more complicated. DivideMix+ still outperforms DivideMix on symmetric and asymmetric noise at all noise ratios. In the harder asymmetric 40% noise, DivideMix+ and GPL outperform the other methods by a large margin.
Results on CIFAR-10. DivideMix+ constantly outperforms DivideMix, especially in the cases with higher noise ratios. We believe the reason is that the mini-batch SS strategy used in our framework can better mitigate the confirmation bias induced from wrongly divided samples in more challenging scenarios. Overall, GPL and DivideMix+ surpass the other methods over a large margin, with the latter performing extremely well on asymmetric noise. Results on CIFAR-100. In most cases, DivideMix+ and DivideMix achieve higher test accuracy than the other approaches, with DivideMix+ performing better. Specifically, DivideMix+ surpasses DivideMix by 14.82% in the hardest symmetric 80% case. An interesting phenomenon is that all the approaches suffer from performance deterioration in the asymmetric 40% cases except GPL, which significantly outperforms the second-best algorithm over +14%.  Table 4: Test accuracy (%) of the baseline and three SSL backbones integrated into our proposed framework.
Results on Clothing1M. To show the robustness of our framework under real-world noisy labels, we demonstrate the effectiveness of DivideMix+ and GPL on Clothing1M. As shown in Table 5, the performance of DivideMix+ is better than that of DivideMix and other methods.

The effects of SS strategies
To study how SS strategies can affect the performance of our framework, we propose DivideMix-by replacing the GMM component in DivideMix with our self-prediction divider yet maintaining the epoch-level SS strategy for a fair comparison. Due to constraints of space, we only provide the mean value of the results in Table 3, which can show the overall tendency. Results with mean and standard deviation can be found in supplementary materials. In CIFAR-10, the difference between DivideMix-and DivideMix is not obvious in the lower noise ratios. However, in the most difficult symmetric 80% case, the test accuracy of DivideMix is +12.69% higher than DivideMix-. The difference is even greater in CIFAR-100, showing that GMM is better able to distinguish clean and noisy labels in most cases. An impressive phenomenon to note is that DivideMix-excels in the asymmetric 40% case in CIFAR-100, which means the self-prediction divider performs better in nosier asymmetric cases than GMM. The reason is explained in the original paper of DivideMix [26], that GMM cannot effectively distinguish clean and noisy samples under asymmetric noise with high noise ratio in datasets with a large number of classes. At the same time, the fact that DivideMix+ constantly outperforms DivideMix in most cases shows that the mini-batch SS strategy in our framework is better than the epoch-level one in DivideMix.

The effects of SSL backbones
We evaluate the effects of SSL backbones in our framework by combining the self-prediction divider (SPD) with three different SSL methods and a baseline which only updates the model using the cross-entropy loss calculated from clean samples. We denote them as SPD-Temporal Ensembling, SPD-MixMatch, SPD-Pseudo-Labeling, and SPD-Cross-Entropy, respectively. For a fair comparison, we use the "13-CNN" architecture [42] for all methods across different datasets. We keep most hyperparameters introduced by the SSL methods close to their original papers [2,4,22], since they can be easily integrated into our framework without massive adjustments.
In Table 4, we list these four algorithms in the left column from weak to strong according to their performance in their original papers. The test accuracies demonstrate their corresponding performance for NLL based on our framework. SPD-MixMatch and SPD-Pseudo-Labeling outperform SPD-Temporal Ensembling by a large domain in both CIFAR-10 and CIFAR-100, especially under 80% noise ratio (over Methods Test Accuracy Cross-Entropy 69.21 F-correction [33] 69.84 M-correction [1] 71.00 Joint-Optim [41] 72.16 Dividemix [26] 73.91 GPL(ours) 73.19 Dividemix+(ours) 74.14 Table 5: Test accuracy (%) on Clothing1M.
40% in CIFAR-10). This phenomenon is reasonable because Temporal Ensembling [22] only uses consistency regularization for unsupervised loss, while MixMatch [4] and Pseudo-Labeling [2] also leverage entropy regularization as well as MixUp data augmentation [53]. Moreover, SPD-Pseudo-Labeling achieves remarkable test accuracy under 80% noise ratio in CIFAR-100, which is +21.44% higher than SPD-MixMatch and +42.66% higher than SPD-Temporal Ensembling. We assume that this is due to the additional loss used in SPD-Pseudo-Labeling that prevents the model from assigning all labels to a single class at the early training stage.
From the results of SPD-Cross-Entropy, we can see that after the removal of the SSL backbone, the test accuracy drops dramatically compared to SPD-MixMatch and SPD-Pseudo-Labeling, especially in high noise ratios and datasets with more classes (e.g., CIFAR-100). This is possibly due to the substantial amount of data that has been removed by the self-prediction divider, leaving very few samples per class. Thus, instead of discarding noisy samples, transferring them to unlabeled ones in SSL backbones is an effective way to combat noisy labels.