{'fraction_non_alphanumeric': 0.03067112055876101, 'fraction_numerical': 0.01662617673853629, 'mean_word_length': 5.059337626494941, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Few-shot Text Classification with Dual Contrastive Consistency

In this paper, we explore how to utilize pre-trained language model to perform few-shot text classification where only a few annotated examples are given for each class. Since using traditional cross-entropy loss to fine-tune language model under this scenario causes serious overfitting and leads to sub-optimal generalization of model, we adopt supervised contrastive learning on few labeled data and consistency-regularization on vast unlabeled data. Moreover, we propose a novel contrastive consistency to further boost model performance and refine sentence representation. After conducting extensive experiments on four datasets, we demonstrate that our model (FTCC) can outperform state-of-the-art methods and has better robustness.


Introduction
Text classification is a fundamental task in natural language processing with various applications such as question answering [1], spam detection [2] and sentiment analysis [3]. With the advancement of deep learning, fine-tuning pre-trained language model [4,5] achieved significant success. However, it still requires a large amount of labeled data to reach optimal generalization of model. Thus, researcher gradually focus on semi-supervised text classification where only a few annotated data is provided. The success of semi-supervised methods results from the usage of abundant unlabeled data: Unlabeled documents in training dataset provide natural consistency regularization by constraining the model predictions to be invariant to small noises in text input [6,7,8]. Despite mitigating the annotation burden, these methods are highly unstable in different runs and can still easily overfit on the very limited labeled data.
Inspired by the success of supervised contrastive learning under few-shot settings [9,10], we hypothesize that the learned contrastive representation under this scenario can help us tackle aforementioned high variance issue and impose additional constraints to the model. Label information and feature structure can simultaneously be propagated from labeled examples to unlabeled ones. Thus, we devised a novel contrastive consistency schema to further boost model performance. To validate the effectiveness and robustness of FTCC, we conduct extensive experiments on four datasets. The result of our experiments confirms that FTCC can be leveraged to improve the performance of the few-shot text classification.
Based on this motivation, the contributions of this paper are as follows: • We integrate supervised contrastive learning objective into consistency-regularized semi-supervised framework to perform text classification under few-shot scenario.
• We devised a novel contrastive consistency schema to propagate feature structure from labeled data to unlabeled data dynamically.
• We demonstrated our model's superiority over state-of-the-art semi-supervised methods and analyze the contribution of each component of FTCC through ablation study and also visualize the learned instance representations, showing the necessity of each loss and the advantage of FTCC on representation learning over BERT fine-tuning with cross-entropy. Increasing attention are paid to semi-supervised text classification, since abundant unlabeled data are easier to obtain compared to labeled data. For instance, [7] adds perturbations to word embedding from bidirectional LSTM [11] and employs adversarial and virtual adversarial training in the text domain. [6] enforces consistency regularization between unlabeled data and augmented data after back-translation. [8] introduces an interpolation-based augmentation and regularization technique to extend consistency training work and better leverage vast unlabeled data.

Contrastive Learning
Contrastive Learning recently achieves remarkable success in self-supervised representation learning on computer vision [12,13] and natural language processing [14,15]. The basic idea of unsupervised contrastive learning is that after generating different views of the sample example, the model adopts a loss function that can pull an anchor and a positive view closer and push the anchor away from other negative examples in the embedding space. Its effectiveness on downstream linear classification is justified through alignment and uniformity [16,17]. Learning good generic representation is widely investigated. For example, in natural language field, [14,18] adopt data augmentations such as word substitution, back translation, word reordering to generate different views of samples. [19,15] adjust dropout ratio to generate positive pairs. Subsequently, researcher proposes supervised contrastive learning [20] loss that enforces representations of examples from the same class to be similar and the ones from different classes to be distinct. [9] adopts this loss to fune-tune pre-trained language model and brings astonishing improvement in low-resource scenario.
[10] introduces label-aware data augmentation and utilizes supervised contrastive learning to simultaneously obtain discriminative feature representations of input examples and corresponding classifiers in the same space.

Problem Formulation
The task of our model is to train a text classifier that efficiently utilizes both annotated data and unannotated data in semisupervised learning. We denote that

Supervised Contrastive Learning
Unlike traditional fine-tuning pre-trained language model, we additionally includes a supervised contrastive learning term to fully leverage the supervised signals [9,20] . This mechanism takes the samples from the same class as positive samples and the samples from different classes as negative samples. The contrastive loss and cross-entropy loss is defined as follows: where we work with a batch of training examples of size N, x ∈ R d is l 2 normalized embedding on the [CLS] token from an encoder to represent an example. N yi is the total number of examples in the batch that have the same label as y i ; τ scl > 0 is an adjustable scalar temperature; the · symbol denotes the dot product; y i,c denotes the label andŷ i,c denotes the model output for the probability of the i th example belonging to the class c.

Consistency Training
Back-translations [21,22] is common data augmentation technique and can generate diverse paraphrases while preserving the semantics of the original sentences. We utilize back translations to paraphrase the unlabeled data to get augmented noisy data. Following [6], we minimize the Kullback-Leibler divergence between the augmented view of the example and the original view of example to encourage their consistency and enforce the smoothness of the model. FTCC adopts the loss function as follows: where a is back-translation transformation, pθ(y|x u ) is an original example's label distribution, p θ (y|a(x u )) is the back-translated example's label distribution.θ denotes the stop gradient of θ, as suggested by VAT [23].

Contrastive Consistency
Inspired by [6,24], we conjecture that consistency regularization not only can propagate label information from label data to unlabeled data, but also can propagate feature structure in their latent space. Thus, we propose to encourage the consistency between the feature pattern of original examples and their corresponding augmented examples, which further refines deep clustering and classification accuracy on augmented data. We randomly sample a batch of N examples and perform back-translation on every example, resulting in 2N data points. Given one example and its corresponding back-translated example, we treat the other 2(N − 1) examples as negatives. Then we denote the similarity between the example x u and the negatives n i (i ∈ {1, ...., 2(N − 1)}) in one batch as: Likewise, the similarity between the corresponding augmented example a(x u ) and the negatives n i (i ∈ {1, ...., 2(N − 1)}) in one batch as: where τ con is a temperature hyperparameter different from τ scl and every embedding is l 2 normalized. Then we can impose the consistency between the probability distributions P and Q by using KL Divergence as the measure of disagreement:

Overall training objective
Above all, the overall training objective is: L = L ce + λ 1 L scl + λ 2 L con + αλ 3 L cc (7) Motivated by [25], We also dynamically release contrastive consistency signal. During the first half of training epochs, α is set to 0 to avoid false feature structure propagation. After half of the training epochs finish, the model becomes more stable on representing feature, and we gradually increase α as the epoch grows: α = 2t−T T , where T is the total number of epochs and t is the current epoch number.

Set up
We conduct our experiments on following four benchmark text classification datasets. SST2 [26] is a sentiment classification dataset of movie reviews. SUBJ [27] is a review dataset with sentences labelled as subjective or objective. PC [28] is a binary sentiment classification dataset that includes Pros and Cons data. IMDB [29] is a sentiment classification dataset from IMDB movie reviews. The dataset statistics are shown in Table 1. All datasets are in English language. In few-shot learning scenario, we set the number of training labeled examples K = 10 per class. We follow [30] to randomly split training dataset into labeled data, unlabeled data and development data in 3 different samples. During splitting, we take label distribution and the size of dataset into account and do not use all unlabeled data. Three different models are trained on these splits. Then we evaluate the average performance of these three models on given test set. We choose German as the intermediate language to perform data augmentation with Fairseq toolkit [31]. We use pre-train language model BERT as our backbone to encode all examples and use the input format "[CLS] sentence [SEP]" for all models. For three datasets SST2, SUBJ, PC, the max length are set to be the first 128, 128, and 64 tokens. In IMDB dataset, we remained the last 256 tokens, as suggested by [6,24]. We use Adam [32] as the optimizer with linear decaying schedule. For simplicity, we set λ 1 , λ 2 , λ 3 to 1. For all datasets, the batch size of labeled data is 8 and the batch size of unlabeled data is 32,24,32,32. Detailed information on experimental hyperparameter settings can be found in Table 3. We run experiments on one NVIDIA RTX A6000 GPU.  SST2  2  7447  1821  3000  1000  SUBJ  2  9000  1000  3600  1000  IMDB  2  25000 25000  10000  4000  PC  2  32097 13759  12000  4000  Table 1: Statistics of four text datasets. Unlabeled and Dev dataset have even label distribution.

Baseline
We compare our FTCC to five baselines: (1) BERT-FT: fine-tuning the pre-trained BERT-based-uncased model on the labeled texts directly. (2) BERT-SCL [9]: fine-tuning BERT with cross-entropy loss and supervised contrastive loss   Table 2: Performance (test accuracy(%)) comparison with baselines. We use 10 labeled examples for each class. The results are averaged after three random seeds to show the significance [33]. ± denotes the Standard Error of the Mean.

Main Result
The classification accuracy of all methods is shown in Table 2 [34], Training Signal Annealing) and uses supervised contrastive learning and contrastive consistency to prevent overfit and achieve better and more stable performance. Compared to MixText, FTCC does not redesign pre-train language model between different layers but adopts a simply but effective method to fine-tune it, which demonstrates that learning quality feature representation for data is more effective and efficient in low-resources scenario and FTCC is more robust and capable of fully leveraging labeled and unlabeled data. To explore how FTCC improves the quality of learning representations, we uses t-SNE [35] to visualize learned CLS embeddings on four test dataset. We shows the results in Figure 2. We can find that only using 10 examples per class to fine-tune BERT with cross-entropy loss has unstructured feature representation. However, our model can clearly separate examples from different classes and pull examples from same class closer. It demonstrates that the learned representations are more discriminative and enhance our model's robustness and generalization.   Table  4. We can observe that every removal causes worse performance than original model, confirming the necessity of each loss term; Moreover, simply using SCL or CON has highly variance result while jointly incorporating them in our model design successfully alleviates this issue; CC component during the second half of training process further refines deep clustering and boosts FTCC's performance.

Conclusion
In this paper, we propose FTCC that can effectively utilize unlabeled data under few-shot scenario and simultaneously enforce more compact clustering of sentence embeddings with similar semantic meanings. Moreover, we propose a contrastive consistency schema that can improve the quality of feature representation and the ability of model generalization. Our model achieves competitive performance on four text classification datasets, outperforming previous methods.