{'fraction_non_alphanumeric': 0.03520702418868899, 'fraction_numerical': 0.020056813290866834, 'mean_word_length': 4.7800995024875625, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation

As the adoption of deep learning techniques in industrial applications grows with increasing speed and scale, successful deployment of deep learning models often hinges on the availability, volume, and quality of annotated data. In this paper, we tackle the problems of efficient data labeling and annotation verification under the human-in-the-loop setting. We showcase that the latest advancements in the field of self-supervised visual representation learning can lead to tools and methods that benefit the curation and engineering of natural image datasets, reducing annotation cost and increasing annotation quality. We propose a unifying framework by leveraging self-supervised semi-supervised learning and use it to construct workflows for data labeling and annotation verification tasks. We demonstrate the effectiveness of our workflows over existing methodologies. On active learning task, our method achieves 97.0% Top-1 Accuracy on CIFAR10 with 0.1% annotated data, and 83.9% Top-1 Accuracy on CIFAR100 with 10% annotated data. When learning with 50% of wrong labels, our method achieves 97.4% Top-1 Accuracy on CIFAR10 and 85.5% Top-1 Accuracy on CIFAR100.


Introduction
As deep learning models and algorithms continue to evolve with increasing capacity and complexity, existing research has shown success across a plethora of domains and tasks [1,2,3] at the cost of a growing amount of data and compute. However, many industrial applications do not have readily available high-quality datasets. As a result, a large part of the machine learning life cycle is data engineering [4], which often requires painstaking manual annotation and inspection that are expensive and time-consuming [5].
To reduce the amount of human effort, it is necessary to automate the data curation process and reduce the number of labels needed for good performance. For example, active learning can reduce the amount of manual labor required by prioritizing the most informative data sample for labeling. Recent progress in active learning has shown promising results in speeding up human-in-the-loop annotation [6,7,8,9,10]. To help model learn with fewer labels, both self-supervised learning [11,12,13,14,15,16] and semi-supervised learning [17,18,19,20,21,22,23,24,25,26,27,28] has shown substantial progress, achieving competitive results against supervised baselines with limited supervisions.
Inspired by the latest advances in self-supervised learning and semi-supervised learning, we combine the best of both worlds and build a simple yet versatile image similarity-based framework for robust and efficient labeling and data verification. The contributions of the paper are • We leverage the latest advances in self-supervised learning and computer vision architectures to obtain image representations that is useful for versatile downstream usages.
• We demonstrate the effectiveness of the self-supervised representation in a variety of scenarios including active learning-based human-in-the-loop annotation, label error detection, and robust classification with noisy labels.
• Unlike expensive semi-supervised approaches that require updating neural networks to incorporate label information [28], our approach leverages label propagation based on nearest neighbor graph to quickly incorporate new label information via simple matrix multiplication. As a result, our method can be seamlessly incorporated into real-time human-in-the-loop systems without sacrificing throughput.
We hope our approach will be a simple and strong baseline to motivate future progress in building labor-efficient data curation pipelines and label-efficient machine learning systems. Code will be made available.

Methods
Our general workflow consists of two parts. We first leverage self-supervised learning, specifically, contrastive learning methods to obtain an unsupervised representation for the unlabeled data. Then we construct a nearest neighbor graph over data samples based on the learned representations. Finally, we can use the nearest neighbor graph for various downstream tasks. In this section, we will discuss each component of our method in detail.

Problem formulation
We assume a dataset of n examples X = {x 1 , . . . , x n } ∈ R n×d with d being the feature dimension. For our purpose, we define l ⊆ [n] to be the index set for samples with human verified labels and u ∈ [n] to be the index set for data samples without trusted labels. The label matrix Y = {y 1 , . . . , y n } ∈ R n×c with c being the number of classes. Our goal is to leverage feature matrix X and known label matrix Y l to generate and improve the estimateỸ u for the unknown label matrix Y u .

Self-supervised Learning
Recent developments in self-training have seen a substantial progress exemplified by a series of work [11,14,12,15,16] in contrastive learning, where the goal is to learn representation that is invariant across two views of the same image created via data augmentation. Specifically, we leverage BYOL [14], which uses an asymetric siamese architecture including online encoder f θ , online projector g θ and predictor q θ for one branch and a target encoder f ξ and projector g ξ for the other branch with polyak averaged weights ξ ← τ ξ + (1 − τ )θ. Given two views v 1 and v 2 of the same image x, we obtain projections , 2}, and we train θ with the following loss ) as a similarity metric between x i and x j .

Nearest Neighbor Graph
Based on the metric (f θ (x i ), f θ (x j )), we can build a nearest neighbor graph in the form of sparse adjacency matrix W where where NN(i, k) denotes the index set of the k nearest neighbors of x i , and T is a temperature parameter. The symmetrically normalized counterpart of W is given by is the degree matrix and 1 n is an all-ones n-vector.

Semi-supervised Pseudo-Labeling with Label Propagation
Based on the consistency assumption [29] that nearby nodes are likely to have the same label, we can perform label propagation (LP) on the nearest neighbor graph to propagate information from samples with known labels to samples without label or with noisy labels as follows where Y (0) = {y 1 , · · · , y n } is the initial label matrix, l ⊆ [n] denotes index set for samples with annotated/verified labels, u ⊆ [n] denotes index set for samples without trusted labels.Ỹ (t+1) is the soft label matrix at iteration t + 1, from which we takeỸ l . Since LP is mostly sparse matrix multiplication, incorporating information from added labels into the pseudo-labels is much faster than training a deep learning model with partial labels.

Experimental Analysis and Results
In this section, we showcase the performance of two convenient workflows that we built around the aforementioned techniques with minimal modifications.

Experiment Settings and Implementation Details
We use CIFAR10 and CIFAR100 [30] as our benchmark datasets. To perform self-supervised learning on target dataset, we leverage the Vision Transformer [1], ViT-B/16, as our encoder architecture while following the exact projector and predictor definition in [1]. We initialize the ViT encoder with BEiT [31] pretrained weights. We perform all training with batchsize 64 with images resized to 224 × 224 resolution. We use the AdamW optimizer [32]. For each experiment, we determine the learning rate, weight decay, and τ through grid search. On CIFAR10, we run BYOL for 3k steps with a learning rate of 2.3e − 5, 5.e − 4 weight decay, τ = 0.998. On CIFAR100, we run BYOL for 5k steps with a learning rate of 4.6e − 5, 2.1e − 6 weight decay, τ = 0.9993. Following [1], we gradually anneal τ to 1 during training. To construct k-NN graph, we use k = 10 and T = 0.01 for CIFAR10, and k = 15 and T = 0.02 for CIFAR100. We use t = 20 iterations for LP.

k-NN Classification Performance with Learned Representations
To assess the quality of our learned representation, we directly performed weighted k-NN classification based on the nearest neighbor graph in Equation 2. We achieved 98.45% validation Top-1 accuracy on CIFAR10, and 89.58% validation Top-1 accuracy on CIFAR100.  Figure 1: Active Learning Performance on CIFAR10 (left) and CIFAR100 (right). Orange line denotes using only labeled training data to predict validation labels. Red line denotes results from Bengar et al. [33]. Green line denotes using both labeled training data and the unlabeled training data with LP generated pseudo-labels to predict validation labels.

Efficient Annotation with Active Learning
For this task, we perform simulation using both datasets and start with no training label. We simulate the human-in-the-loop annotation process by iteratively performing LP and randomly sampling data for oracle labeling. Following the observation in [33], we choose the random sampling as a simple and strong baseline. As shown in Figure 1, we achieve exponential gain in Top-1 Accuracy when annotating fewer than < 0.1% data in CIFAR10 and < 1% data in CIFAR100. We outperform previous work by Bengar et al. [33] substantially.
To assess the value of pseudo-labels generated by LP on the unlabeled training data, we perform an ablation study by performing LP only on the annotated data and the validation data. As shown in Figure 1, without pseudo-labels from the unlabeled training data, the orange curves show substantial drops in validation top-1 accuracy when using the same amount of annotation. Thus, having a reliable nearest neighbor graph allows us to effectively scale performance with the amount of unlabeled data by propagating information from labeled data across the data manifold.

Robust Classification with Noisy Labels
We showcase our second workflow by demonstrating the effectiveness of using LP to correct corrupted labels and maintain robust classification performance under noise.   Effect of LP on Noise Reduction We first examine the evidence for the consistency assumption [29] by simulating random noise in labels and then performing LP. If our nearest neighbor graph faithfully captures the similarity among data, LP will aggregate and smooth out the inconsistency from noisy neighbor labels so that the neighbors with the correct label can stand out. As illustrated in Figure 3, with a noise level below 0.8, LP quickly reduces the noise level in pseudo label with an increasing number of iterations. At extreme noise levels such as 0.9, the consistency assumption no longer holds due to highly corrupted labels, and performing LP hurts performance instead.
Classification under Noisy Label After obtaining smoothed pseudo labels via LP, we use the pseudo labels with the nearest neighbor graph to perform weighted k-NN classification. For comparison, we use weighted k-NN based on the corrupted labels as a baseline. As shown in Figure 2, pseudo labels obtained via LP offer more robust performance than directly using the corrupted labels on both CIFAR10 and CIFAR100.

Conclusion
In summary, leveraging the latest advances in self-supervised learning, we developed a nearest neighbor graph-based approach that can perform versatile downstream tasks and quickly incorporate new information in a semi-supervised manner, suitable for integration with real time human-inthe-loop systems. We demonstrated the effectiveness of our methods on a combination of datasets (CIFAR10, CIFAR100) and tasks (active learning, label error detection, and learning under noise) and achieved competitive performance. We hope our work can serve as a simple and strong baseline for the development of practical tools in industrial settings.