{'fraction_non_alphanumeric': 0.03796203796203796, 'fraction_numerical': 0.015926930212644498, 'mean_word_length': 4.878523489932886, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Interpolation-based Contrastive Learning for Few-Label Semi-Supervised Learning

Semi-supervised learning (SSL) has long been proved to be an effective technique to construct powerful models with limited labels. In the existing literature, consistency regularization-based methods, which force the perturbed samples to have similar predictions with the original ones have attracted much attention for their promising accuracy. However, we observe that, the performance of such methods decreases drastically when the labels get extremely limited, e.g., 2 or 3 labels for each category. Our empirical study finds that the main problem lies with the drifting of semantic information in the procedure of data augmentation. The problem can be alleviated when enough supervision is provided. However, when little guidance is available, the incorrect regularization would mislead the network and undermine the performance of the algorithm. To tackle the problem, we (1) propose an interpolation-based method to construct more reliable positive sample pairs; (2) design a novel contrastive loss to guide the embedding of the learned network to change linearly between samples so as to improve the discriminative capability of the network by enlarging the margin decision boundaries. Since no destructive regularization is introduced, the performance of our proposed algorithm is largely improved. Specifically, the proposed algorithm outperforms the second best algorithm (Comatch) with 5.3% by achieving 88.73% classification accuracy when only two labels are available for each class on the CIFAR-10 dataset. Moreover, we further prove the generality of the proposed method by improving the performance of the existing state-of-the-art algorithms considerably with our proposed strategy.

However, collecting labeled data can consume a lot of resources which is un-affordable to countless everyday learning demands in modern society. Therefore, deep learning algorithms which can achieve appropriate performance with tractable supervision have been a hot research spot in recent years. Specifically, deep semi-supervised learning (SSL) algorithms, which seek to improve the performance of deep learning models on datasets with only limited labeled data by leveraging large amounts of unlabeled data, are an important branch in this family. This has led to a plethora of SSL methods designed for various fields [17]- [23]. Among all the deep semi-supervised learning algorithms, consistency regularization based methods treat the original input and its augmented version as positive pairs, which is a form of contrastive learning [24]- [31]. These consistency regularization-based methods follow a common assumption that ever after data augmentation, the classifier could output the same class probability for an unlabeled sample, which means data augmentation will not change the semantic. The input image should be more similar to its augmented version than other images. Under this assumption, researchers perturb the input samples by conducting data augmentation to generate similar samples of the original data.
The mentioned algorithms have contributed remarkable performance improvement to improve the learning accuracy when only a few labeled data are available. However, we observe that, when the number of labeled data gets extremely small, e.g., 2 to 3 labels for each category, the performance of the existing algorithms would drop drastically. For example, to the CIFAR-10 dataset whose scale for training samples is 50, 000 and 10 categories, the performance of the state-of-theart algorithm MixMatch [28] can achieve the top-1 accuracy of 86.47% when 250 labeled data is available. Nevertheless, the performance of the same algorithm drops to 50.10% when only 30 labeled samples are available. The similar phenomenon happens to the Mean-Teacher [27] algorithm whose performance drop by more than a half when the label number decreases from 250 to 30. More experimental results can be found in Table I. According to our analysis, one of the main reasons that Table I: Classification accuracy of two state-of-the-art semisupervised algorithms, i.e., MixMatch [28] and Mean-Teacher [27], on CIFAR-10 dataset with 30, 40, 250, 500 and 1000 labels.  cause large performance decrease lies with the semantic information drift during data augmentation. Taking the samples in the MINIST dataset for example, when the vertical flip is applied to the samples, the labels of "6"s and "9"s, "2"s and "5"s can easily get changed. This would challenge the rationality of the information consistency assumption of existing methods. This problem could be alleviated when relatively abundant label information is available. However, when the label information is extremely lacked, the performance of the corresponding algorithms could decrease a lot. In this paper, to solve the problem of semantic information drift caused by data augmentation-based positive sample pair construction, we propose a novel interpolation-based positive sample pair construction fashion. Generally, our design roots from the observation that the margin of decision boundaries would get larger if the prediction of the network could change linearly [32], [33]. Under the circumstance of semi-supervised learning, when the label is extremely limited, we seek to improve the discriminative capability of the network by forcing the embedding of the network to change linearly. Specifically, given two unlabeled images, on the one hand, we embed the samples separately into the latent space. On the other hand, we conduct image-level interpolation for an integrated image and do the embedding with the same network. Then, by combining the embedding of the interpolated images with the interpolation of the embeddings, we construct a positive sample pair. In our setting, the negative sample pairs are the embedding pair of different samples. By forcing the positive sample pairs to be close to each other in the latent space and the negative sample pairs to get far away from each other, we enlarge the margin of decision boundaries, thus improving the performance of the algorithm. To achieve the goal, we further propose a novel contrastive learning-based loss function to guide the network for better learning. We name the resultant algorithm Interpolation Contrastive Learning Semi-Supervised Learning (ICL-SSL).
The main contributions of this paper are listed as follows: • We find that semantic information drift is one of the main problems that cause the performance of existing consistency regularization-based semi-supervised algorithms to decrease drastically when extremely limited labeled data is provided. • We propose an interpolation-based positive sample construction method and a novel contrastive loss function to solve the problem and improve the learning accuracy. • Our experimental results on the benchmark datasets verify the superior performance of the proposed algorithms against the state-of-the-art algorithms. We also show the generality of our proposed algorithm by enhancing the performance of the existing advanced algorithms steadily with our method.
II. RELATED WORK In this section, we first define the main notations and then review several semi-supervised learning (SSL) methods related to our method ICL-SSL.

A. Notations Definition
Given a dataset D = X ∪ U, where X = {(x 1 , y 1 ), · · · , (x m , y m )} is an labeled sub-dataset, U = u m+1 , · · · , u m+n } is a unlabeled sub-dataset, n m and y m is encoded by one-hot, we define a classification model as p(y|x; θ), which outputs a distribution over class labels y for an input x with parameters θ. For the model p(y|x; θ), it is concatenated by a encoder network f (·) and a classification head h(·) before softmax function. Meanwhile, after the encoder network f (·), we set a projection head g(·), outputting the normalized low-dimensional representation z = g(f (·)). To simplify, F (·) is defined as g(f (·)). For more detailed definitions, please refer to Table II. The numer of classes Label with C classes encoded by one-hot q ∈ R C Predicted category probability distribution f (·) The encoder network h(·) The classifier g(·) The projector head Normalized low-dimensional representation

B. Contrastive Learning
Thanks to leveraging unlabeled data for model training, contrastive learning attracts much attention of some researchers and becomes a hot spot recently [34]- [37]. It is a widely adopted form of self-supervised learning [34], [38]- [42], which can be used to optimize the task of instance discrimination. Instead of training a classification, contrastive learning is to maximize the similarities of positive pairs and minimize the similarities of negative pairs. It is important to learn the invariance with different views generated by data augmentations. The contrastive learning loss on unlabeled data can be described as follows: where T is a temperature parameter [31]. DA(·) denotes the stochastic data augmentation function. F (·) is the simplified presentation of the encoder network f (·) and the project head g(·). In recent methods, through designing a memory bank, MoCo [35] maintains the consistency of the negative sample pairs. SimCLR [34] calculates the pairwise similarity between two similar samples from the images in the same batch, which pushes the negative samples away while pulling the positive samples. Consistency regularization can be interpreted as a special form of contrastive learning, in which only positive samples are included.

C. Consistency Regularization
Consistency regularization utilizes the assumption that the classifier should output the same prediction for the unlabeled data even after it is augmented. Data augmentation is a frequent regularization technique in semi-supervised learning. Through various data augmentation methods, consistency regularization generates a copy of the sample regarded as a similar sample to the original data. In the simplest form, prior work [43] adds the following consistency regularization loss on unlabeled samples: where DA(·) is a stochastic data augmentation. With the use of an exponential moving average (EMA) model, Mean-Teacher [27] replaces one of the terms in Eq.2, which provides a more stable target. To maximally alter the output class distribution, Virtual Adversarial Training (VAT) [44] uses an adversarial transformation in place of DA(·). More recently, a form of consistency regularization is utilized in Mixmatch [28] by using random horizontal flips and crops for the input samples. Unsupervised data augmentation (UDA) [45], ReMixMatch [30] and FixMatch [29] have been proposed with the use of weak and strong data augmentations. Generally speaking, through a weakly-augmented unlabeled sample, they generate a pseudo label and enforce consistency against the stronglyaugmented version of the same input. The above consistency regularization models are based on data augmentation to generate positive samples. Although promising performance has been achieved, we observe that the discriminative capability of previews methods is limited since they would suffer from the semantic information drift issue. Therefore, the constructed samples are no longer similar. Instead of carefully designing data augmentations to utilize consistency regularization, we use an interpolation-based method to obtain positive pairs, which will avoid the semantic information drift caused by data augmentations.

D. The Interpolation-based Method
Mixup [32] is an effective data augmentation strategy for image classification in computer vision [46]- [49]. It linearly interpolates the input samples and their labels on the input data and label spaces.
where the α and β are the parameter of Beta distribution, λ ∈ [0, 1]. The interpolations of input samples should lead to interpolations of the associated labels. In this manner, Mixup could extend the training distribution. It is recently achieved state-of-the-art performance through different tasks and network architectures. In [50], the interpolations are performed in the input space. In order to improve model performance, [51] is proposed to measure the realism of latent space interpolations in unsupervised learning. [33] performs the interpolation between input and pseudo-labels. Although the above methods are verified to be effective, they will still change the construction method of consistency regularized positive sample pairs. Therefore, how to solve the semantic information drift in consistency regularization is an open question. Different from the above approaches, we propose an interpolation-based method term ICL-SSL to construct positive sample pairs. Without using data augmentation to construct positive sample pairs, ICL-SSL is performed between the input samples and the representations, thus avoiding semantic information drift.

III. METHOD
In this section, we introduce our proposed semi-supervised learning method. Firstly, we will explore the reason for the performance degradation under few labels via some experiments Figure 4: Illustration of Interpolation-based Contrastive Learning Semi-Supervised Learning (ICL-SSL) mechanism. Following the definition in [29], Augw denotes the weak augmentation of the original input image. Augs denotes strong augmentation. Specifically, given two unlabeled images u i and u j , we firstly embed the samples separately into the latent space. Then, we conduct image-level interpolation for an integrated image and do the embedding with the same network. z mix and mix z are the positive embeddings pair constructed by ICL-SSL. By combining these two positive embeddings, we design a novel contrastive loss L c and force these two embeddings to change linearly between samples, improving the discriminative capability of the network. Therefore, our network would be guided to learn more discriminative embeddings with the interpolation-based method.
on MINIST dataset. Through the exploratory experiment, we analyze that the semantic information of the input samples will be drifted after some inappropriate data augmentations, thus limiting the performance. After that, to address this issue, we introduce an interpolation-based method ICL-SSL under few labels to construct more reliable positive sample pairs. Finally, we will detail the designed contrastive loss of ICL-SSL.

A. Semantic information drift
Although promising performances have been achieved by the existing algorithms, we observe that when the number of labeled data gets extremely small, e.g. 2 to 3 labels for each category, the performance of the existing algorithms would decrease drastically. The detailed observation is shown in Table. I. Therefore, we conduct experiments to explore the reason to cause the performance dropping.
Consistency regularization is an essential piece for many state-of-the-art semi-supervised learning methods [28]- [31]. A common assumption of consistency regularization is that the classifier should output the same class probability of an unlabeled sample even if it is augmented.
In several SSL methods [28]- [30], when training data is not enough for generalization, data augmentation is a technique to apply consistency regularization. MixMatch [28] processes the input samples through random horizontal and random crops. The weak data augmentation method uses horizontal flips and vertical flips to process unlabeled samples in FixMatch [29].
Through experiments shown in Fig. 2, we find that some data augmentations will change the semantic information about the input samples, leading to a decrease in the semantic similarity of the constructed samples damaging the SSL training. We visualize the result of data augmentation. It can be found that the semantic information of the input samples has been changed. Fig. 2 shows that under one data augmentation (random vertical flip), the semantic information of "7"s and "2"s, "6"s and "9"s, "2"s and "5"s can easily get changed. As a result, the quality of the constructed positive samples decreases or the construction fails, which in turn affects the performance of the model. To further verify the effect of data augmentation, we implement experiments on the MINIST dataset.
As shown in Fig. 5(b), MINIST is a dataset composed of handwritten numbers, which is commonly used in deep learning research. MINIST consists of 60000 training data and 10000 test data. Aiming to reduce the influence of irrelevant factors (e.g. complex structure of training model) to the performance, we explore the semantic information drift problem caused by data augmentation with two-layer MLPs.
From the empirical analysis, we observe that the accuracy is decreased by 5.0% after the random horizontal flip argumentation on MINIST. As a consequence, after random vertical flips, the accuracy decreases by 4.0% shown in Fig. 3. Additionally, we also explore rotation, random re-cropping and random cropping, the result shows that those data augmentations will also limit the performance of the model.
The experiment on MINIST can illustrate that during SSL training, some inappropriate data augmentations will change the semantic information of the input samples. Therefore, the semantic correlation of positive sample pairs will be destroyed by inappropriate data augmentations. When the label information is lacking, the incorrect regularization caused by data augmentation would mislead the network and limit the algorithm performance.

B. ICL-SSL
To solve the semantic information drift problem, we proposed a novel interpolation contrastive learning Semisupervised learning method termed ICL-SSL. Specifically, ICL-SSL does not change the semantic information during the positive pair construction of consistency regularization. In the following, we first obtain the low-dimensional representation z of the unlabeled sample. Then, we describe the interpolationbased positive sample pairs construction method and loss function in detail.
In our ICL-SSL method, the representations are extracted by encoder network f (·). Concretely, for any two unlabeled samples u i , u j in a batch of unlabeled sub-dataset U B , we could obtain their normalized representations z i , z j with 2 -norm: where F (·) is defined as g(f (·)), a simple form of encoder network f (·) and project head g(·).
After that, we perform interpolation operations on the normalized low-dimensional feature representations z i and z j .
where mix z denotes the interpolated representation of z i and z j , λ is generated by Beta distribution. Simultaneously, unlike the above steps, we first perform an interpolation operation in the sample space (u i , u j ) and then get the normalized low-dimensional feature: where z mix is the representation of interpolated input data u i and u j . The constructed positive sample pair can be presented as follows: The framework of our proposed ICL-SSL is shown in Fig. 4. ICL-SSL is a semantic-agnostic positive sample construction method. Specifically, we generate one positive sample from the features z mix obtained by interpolating two inputs, and the other mix z from interpolating the two features of the input. By this setting, both of these positive samples contain the original semantic information of each input (u i , u j ). It has demonstrated that the interpolation operation has the effect to push the decision boundaries away from the class boundaries in [32], [33]. In this manner, with the utilization of our ICL-SSL, the margin decision boundaries would get larger, thus improving the discriminative capability of the network under few labels.

C. Loss function
The loss of ICL-SSL mainly consists of three parts: the supervised classification loss L x , the unsupervised classification loss L u and the contrastive loss L c .
In detail, L x is the supervised classification loss on the labeled data, which is defined as the cross-entropy between the ground-truth labels and the model's predictions: where x b denotes the labeled data in X B . p(y|x b ) is the output of the classifier. H is the cross-entropy between the two distributions y b and p(y|x b ). For the unlabeled data, its pseudo labelq b is generated by the classification head h(·) and the Sof tmax function. The formula can be described as: L u is defined as the cross-entropy between the pseudo-labels and the model's predictions. It can be calculated by: whereq b is the predicted probability of pseudo labels. is the function to calculate the loss. When the largest class probability is above the threshold τ , the loss will be calculated. Meanwhile, µ is used to count the number of valid unlabeled samples. H is the cross-entropy between q b and p(y|u b ).
Through the positive sample pairs [mix z , z mix ] constructed by interpolation strategy, the contrastive loss can be computed as: where T is a temperature parameter. Similar to SimCLR [34], we do not sample negative samples explicitly. Instead, we treat the other examples within a minibatch as negative samples. I k =i ∈0,1 is an indicator function. When k = i, the value of I is set to 1. The similarity between positive is measured by the inner product. This loss is calculated across all positive samples in a batch. The contrastive loss encourages the model to produce similar representations for positive samples and pushes the negative samples away. The relation of the embedding changes linearly due to the proposed positive sample pairs constructed method. By minimizing Eq.11, the margin decision boundaries will be enlarged, thus improving the discriminative of the network.
In summary, the loss function of ICL-SSL can be computed by: where L x represents the supervised loss and L u is the unsupervised loss. α is a trade-off hyper-parameter to control the weight of the total loss. The detailed learning procedure of ICL-SSL is shown in Algorithm 1.

IV. EXPERIMENT
We evaluate the effectiveness of ICL-SSL on several semisupervised learning benchmarks. We focus on the most challenging label-scare scenario where few labels are available, e.g., 2 or 3 labels for each category. At the same time, our ablation study teases apart the contribution of ICL-SSL components. In addition, we further verify the generality of the proposed method by improving the performance of the existing state-of-the-art algorithms considerably with our proposed strategy.
• The CIFAR-10 dataset consists of 60000 images of size 32 × 32. The training set of CIFAR-10 consists of 50000 images and the test set consists of 10000 images. The dataset includes ten classes, including images of natural objects such as horse, deer, fork, car and aircraft. • The CIFAR-100 dataset is similar to the CIFAR-10 dataset and contains 60000 images of the size 32 × 32. The 100 classes in the CIFAR-100 are grouped into 20 superclasses. Each class consists of 500 training images and 100 testing images.
• The SVHN dataset includes 73257 training data and 26032 test data of size 32 × 32. Besides, each example is a close-up image of house numbers from 0 to 9. Detailed dataset statistics are summarized in Table III. We use the accuracy metric to evaluate the classification performance.
2) Experiment Settings: All experiments are implemented with an NVIDIA 1080Ti GPU on PyTorch platform. Following SSL evaluation methods, we evaluate our method on standard SSL benchmarks with the "Wide-ResNet-28" model from [54]. Compared with other methods, our model focuses on the challenging label-scare scenario e.g., 2 or 3 labels for each category. For CIFAR-10 and SVHN datasets, we train them for 300 epochs until convergence, the batch size chosen by us is 64. Due to the limited computing resources, the batch size of the all comparison experiments on CIFAR-100 dataset is set to 16. The weight parameter α to control loss is set to 0.5, and the parameter µ of the batch size for the control of unlabeled data is set to 1. The learning rate is set to 0.03 for CIFAR-10, CIFAR-100 and SVHN. The threshold τ is set to 0.95. Besides, our network is trained using SGD optimizer. For our proposed method, we adopt the source data of CoMatch [31]. To alleviate the impact of randomness, we evaluate the models on 5 runs for each number of labeled points with different random seeds.
In Sub-Section "Transfer to other models", the algorithms are implemented with an NVIDIA 1080Ti GPU on PyTorch platform with 40, 250, 500, and 1000 labels on CIFAR-10 dataset. Three state-of-the-art algorithms are compared in our transferring experiments, including MixMatch [28], Mean-Teacher [27] and VAT [44]. For those algorithms, we reproduce results by adopting their source code with the original settings. The code for the compared algorithms can be downloaded from the authors' website: MixMatch 1 , Mean-Teacher 2 , VAT 3 . Specifically, the training epoch is set as 300. The learning rate of the optimizer is set as 0.002 for MixMatch, 0.003 for Mean-Teacher, and 0.01 for VAT.

B. Comparison with the State-of-the-Art Algorithms
In this section, six state-of-the-art semi-supervised algorithms are compared to verify the effectiveness of ICL-SSL. The information for the compared algorithms is listed as follows: (1) CoMatch [31]: The class probabilities and lowdimensional embeddings are jointly learned in CoMatch. Through imposing a smoothness constraint to the class probabilities, the quality of pseudo labels could be improved. Overall, CoMatch combines the pseudo-based model, the contrastloss-based model and the graph-based model to improve the model performance in the case of few labels.
(2) FixMatch [29]: For the labeled image FixMatch utilize weak-augmentation to generate the pseudo label. Additionally, for the unlabeled image, the pseudo label is obtained by the high-confidence prediction. And then, the network is trained to predict the pseudo label with the strongly augmented version of the same image.
(3) MixMatch [28]: MixMatch jointly optimizes two losses: the supervised loss and unsupervised loss. In detail, crossentropy is chosen for the supervised loss. The unsupervised loss is the mean square error (MSE) between predictions and generated pseudo labels. MixMatch constructs pseudo labels by data augmentation. With the use of the sharpen function Sharpen(·), MixMatch could improve the quality of pseudo labels. In addition, Mixup is added in the training process, which can construct virtual samples through interpolation.
(4) Virtual Adversarial Training(VAT) [44]: VAT is based on data perturbation. It replaces data augmentation with adversarial transformations. The adversarial transformation can lead to a lower classification error.
(5) π-model [26]: For the same image, data augmentation is used to apply consistency regularization. The loss of πmodel contains the supervised loss and the unsupervised loss. Specifically, the supervised loss is defined as the cross-entropy loss, and the unsupervised loss is the unsupervised consistency loss.
(6) Mean-Teacher [27]: Mean-Teacher is a student-teacherapproach for SSL. The teacher model is based on the average weights of a student model in each update step. In Mean-Teacher, the mean square error loss (MSE) is used as its consistency loss between two predictions. Besides, it uses the exponential moving average (EMA) to update, because the EMA is only updated once per epoch, which can control the model update speed.
C. Performance Comparison 1) CIFAR-10: To demonstrate the superiority of ICL-SSL, we conduct performance comparison experiments for our proposed ICL-SSL and 4 baselines, including Mean-Teacher [27], MixMatch [28], FixMath [29] and CoMatch [31]. For CIFAR-10 dataset, we evaluate the accuracy of above methods with a varying number of labeled data from 20 to 40. The results are reported in Table. IV. For fairness, we create 5 runs for each number of labeled points with different random seeds to alleviate the influence of randomness. We can observe that our method ICL-SSL outperforms all other methods by a significant margin, taking the result on only 2 labeled data in each class for example, ICL-SSL could reach an accuracy of 88.73%. For comparison, at 20 labels the second best algorithm (CoMatch [31]) achieves an accuracy 83.43%, which is 5.30% lower than ICL-SSL. ICL-SSL can achieve higher accuracy by using fewer labels.
2) SVHN: Moreover, we implement comparison experiments on SVHN dataset. The comparison algorithms contains π model [26], Mean-Teacher [27], MixMatch [28], FixMatch [29]. The quantity of labels is 250 to 1000. The results can be seen in TableV. With different random seeds, we evaluate the models on 5 runs for each number of labeled data. We could observe that ICL-SSL outperforms all compared methods SVHN with 250, 500, and 1000 labeled data. For example, ICL-SSL exceeds MixMatch by 3.46% with 250 labels.  3) CIFAR-100: To further investigate the effectiveness of our proposed model, we conduct experiments on CIFAR-100 dataset. Table. V reports the performance of the four methods with 200, 400, and 1000 labels. From those results, we can observe that, our proposed ICL-SSL could achieve better performance compared with other state-of-the-art algorithms. Taking the result with 200 lables for example, ICL-SSL exceeds FixMatch [29] by 4.75%.
Through the above experiments, our method outperforms all the existing methods in the case of few labels. The reason is that other methods use data augmentation to generate positive sample pairs, easily leading to incorrect regularization. Different from them, our ICL-SSL aims to improve the discriminative capability from two aspects. Firstly, we proposed an interpolation-based method to construct more reliable positive sample pairs, thus alleviating the incorrect regularization. Additionally, we design a contrastive loss to guide the embedding to change linearly in samples, which could enlarge the margin decision boundaries. In summary, we proposed ICL-SSL that could improve the discriminative capability of the network and Table IV: Accuracy comparison with other state-of-the-art methods on five different folds, including Mean-Teacher [27], MixMatch [28], FixMatch [29] and CoMatch [31] on CIFAR-10 dataset. The red and blue values indicate the best and the runner-up results. achieves the top-level performance on CIFRA-10, SVHN, and CIFAR-100 dataset.

D. Time Cost
As shown in Table VIII, we compare the training and the inference time of ICL-SSL and other state-of-the-art algorithms, including MixMatch [28], FixMatch [29], and CoMatch [31]. The results are the average training time for 300 epochs with 20 labels on CIFAR-10 dataset. We observe that the training and the inference time of ICL-SLL are 193.81 seconds and 1.08 seconds, respectively. From the Table.VIII we find that the computational efficiency of the proposed algorithm is comparable to the MixMatch and FixMatch and is much faster than that of CoMatch.

E. Ablation Study
In this section, we implement extensive ablation studies to examine the effect of different components in ICL-SSL. Due to the number of experiments in our ablation study, we perform the study with 20 and 40 labels split from CIFAR-10 dataset. The parameter settings are kept the same with comparison experiments, and the results are shown in Table. VII.

Effective of contrastive loss
To further investigate the superiority of the proposed contrastive loss, we experimentally compare our method. Here, we denote the FixMatch [29] as the baseline. With the experimental results, in the case of few labels, the model performance achieves better performance than that of baselines. Taking the result on CIFAR-10 with 20 labels for example, the accuracy exceeds the baseline by 16.1% performance increment. From the empirical analysis, it benefits from the contrastive loss to guide the embedding of the network to change linearly between samples to improve the discriminative capability of the network.
Effective of interpolation-based positive samples construction method.
Additionally, we verify the effectiveness of the interpolation-based positive samples construction method. As shown in TableVII, we can observe that the accuracy would decrease from 88.73% to 56.91%. The above experiments

F. Sensitivity Analysis
Further, we investigate the effect of hyper-parameters α. As shown in Fig. 8, we observe that the classification accuracy will not fluctuate greatly when the α is varying. This demonstrates that our model ICL-SSL is insensitive to the variation of the hyper-parameter α.

G. Transferring to other models
To verify the generality of our proposed ICL-SSL, we transfer our method to the existing state-of-the-art algorithms. We implement our method into other semi-supervised learning models (MixMatch [28], VAT [44], Mean-Teacher [27]). All the experiments are implemented with CIFAR-10 dataset. Experiments are carried out on the number of labeled data from 40, 250, 500 and 1000. Here, we denote the baseline and the baseline with our method ICL-SSL as "B" and "B+O", respectively.
From Fig. 7, we have observed as follows: 1) The models could achieve better performance with our method. 2) As shown in Table VI, taking the results in MixMatch [28] for example, our method could improve the classification accuracy by 4.02% on 40 labeled data and 2.14% on 500 labels on CIFAR-10 dataset, respectively. In conclusion, the experiment results show that ICL-SSL can improve the model performance in other semi-supervised models. Moreover, in Fig. 6, we further show that other state-of-the-art methods could obtain higher accuracy with our proposed strategy during the training process.

V. CONCLUSION
In this work, we propose an interpolation-based method termed ICL-SSL to construct reliable positive sample pairs, thus alleviating the semantic information drift with extreme labels (e.g., 2 or 3 labels for each class). Specifically, ICL-SSL is a semantic-agnostic method. We interpolate the input images and their representations in image-level and latent space, respectively. Besides, the designed contrastive loss will Table V: Accuracy comparison with π model [26], Mean-Teacher [27], MixMatch [28], and FixMatch [29] on CIFAR-100 and SVHN dataset. The red and blue values indicate the best and the runner-up results. The average and std values of the five fold cross validation are reported.    guide the embeddings changing linearly between samples and thus get a larger margin decision boundary. Benefiting from this mechanism, the discriminative capability of the network can be improved with extreme labels. Extensive experiments demonstrate the effectiveness and generality of our ICL-SSL.
In the future, we will try to extend ICL-SSL to other fields (e.g. graph semi-supervised node classification). Besides, as we analyzed in section IV-D, although our proposed algorithm is as efficient as other state-of-the-art contrastive algorithms, its efficiency still needs to be improved to suit even larger scale datasets. Therefore, how to reduce the training time is also a future work direction.