{'fraction_non_alphanumeric': 0.03711987443594271, 'fraction_numerical': 0.018206788306847164, 'mean_word_length': 4.391580283477893, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Fine-tuning (FT) pre-trained sentence embedding models on small datasets has been shown to have limitations. In this paper we show that concatenating the embeddings from the pre-trained model with those from a simple sentence embedding model trained only on the target data, can improve over the performance of FT for few-sample tasks. To this end, a linear classifier is trained on the combined embeddings, either by freezing the embedding model weights or training the classifier and embedding models end-to-end. We perform evaluation on seven small datasets from NLP tasks and show that our approach with end-to-end training outperforms FT with negligible computational overhead. Further, we also show that sophisticated combination techniques like CCA and KCCA do not work as well in practice as concatenation. We provide theoretical analysis to explain this empirical observation.


Introduction
Fine-tuning (FT) powerful pre-trained sentence embedding models like BERT (Devlin et al., 2018) has recently become the de-facto standard for downstream NLP tasks. Typically, FT entails jointly learning a classifier over the pre-trained model while tuning the weights of the latter. While FT has been shown to improve performance on tasks like GLUE (Wang et al., 2018) having large datasets (QQP, MNLI, QNLI), similar trends have not been observed on small datasets, where one would expect the maximum benefits of using a pre-trained model. Several works (Phang et al., 2018;Garg et al., 2019;Dodge et al., 2020;Lee et al., 2020) have demonstrated that FT with a few target domain samples is unstable with high variance, thereby often leading to sub-par gains. Furthermore, this * Equal contribution by authors † Work completed at the University of Wisconsin-Madison issue has also been well documented in practice 1 . Learning with low resources has recently become an active research area in NLP, and arguably one of the most interesting scenarios for which pre-trained models are useful (e.g., (Cherry et al., 2019)). Many practical applications have small datasets (e.g., in social science, medical studies, etc), which are different from large-scale academic benchmarks having hundreds of thousands of training samples (e.g, DBpedia (Lehmann et al., 2015), Sogou News (Wang et al., 2008), etc). This necessitates effective transfer learning approaches using pre-trained sentence embedding models for fewsample tasks.
In this work, we show that concatenating sentence embeddings from a pre-trained model and those from a smaller model trained solely on the target data, can improve over the performance of FT. Specifically, we first learn a simple sentence embedding model on the target data. Then we concatenate(CAT) the embeddings from this model with those from a pre-trained model, and train a linear classifier on the combined representation. The latter can be done by either freezing the embedding model weights or training the whole network (classifier plus the two embedding models) end-to-end.
We evaluate our approach on seven small datasets from NLP tasks. Our results show that our approach with end-to-end training can significantly improve the prediction performance of FT, with less than a 10% increase in the run time. Furthermore, our approach with frozen embedding models performs better than FT for very small datasets while reducing the run time by 30%−50%, and without the requirement of large memory GPUs.
We also conduct evaluations of multiple techniques for combining the pre-trained and domainspecific embeddings, comparing concatenation to 461 CCA and KCCA. We observe that the simplest approach of concatenation works best in practice. Moreover, we provide theoretical analysis to explain this empirical observation.
Finally, our results also have implications on the semantics learning ability of small domainspecific models compared to large pre-trained models. While intuition dictates that a large pre-trained model should capture the entire semantics learned by a small domain-specific model, our results show that there exist semantic features captured solely by the latter and not by the former, in spite of pretraining on billions of words. Hence combining the embeddings can improve the performance of directly FT the pre-trained model. Related Work Recently, several pre-trained models have been studied, of which some provide explicit sentence embeddings (Conneau et al., 2017;Subramanian et al., 2018), while others provide implicit ones (Howard and Ruder, 2018;Radford et al., 2018). Peters et al. (2019) compare the performance of feature extraction (by freezing the pre-trained weights) and FT. There exists other more sophisticated transferring methods, but they are typically much more expensive or complicated. Our focus is to propose alternatives to FT with similar simplicity and computational efficiency, and study conditions where it has significant advantages. While the idea of concatenating multiple embeddings has been previously used (Peters et al., 2018), we use it for transfer learning in a low resource target domain.

Methodology
We are given a set of labeled training sentences S={(s i , y i )} m i=1 from a target domain and a pretrained sentence embedding model f 1 . Denote the embedding of s from f 1 by v 1s =f 1 (s)∈R d 1 . Here f 1 is assumed to be a large and powerful embedding model such as BERT. Our goal is to transfer f 1 effectively to the target domain using S. We propose to use a second sentence embedding model f 2 , which is different from and typically much smaller than f 1 , which has been trained solely on S. The small size of f 2 is necessary for efficient learning on the small target dataset. Let v 2s =f 2 (s)∈R d 2 denote the embedding for s obtained from f 2 .
Our method CAT concatenates v 1s and v 2s to get an adaptive representationv s =[v 1s , αv 2s ] for s. Here α>0 is a hyper-parameter to modify emphasis on v 1s and v 2s . It then trains a linear classifier c(v s ) using S in the following two ways: (a) Frozen Embedding Models Only training the classifier c while fixing the weights of embedding models f 1 and f 2 . This approach is computationally cheaper than FT f 1 since only c is trained. We denote this by CAT (Locked f 1 , f 2 weights).

(b) Trainable Embedding Models
Jointly training classifier c, and embedding models f 1 , f 2 in an end-to-end fashion. We refer to this as CAT .
The inspiration for combining embeddings from two different models f 1 , f 2 stems from the impressive empirical gains of ensembling (Dietterich, 2000) in machine learning. While typical ensembling techniques like bagging and boosting aggregate predictions from individual models, CAT and CAT aggregate the embeddings from individual models and train a classifier using S to get the predictions. Note that CAT keeps the model weights of f 1 , f 2 frozen, while CAT initializes the weights of f 2 after initially training on S 2 .
One of the benefits of CAT and CAT is that they treat f 1 as a black box and do not access its internal architecture like other variants of FT (Houlsby et al., 2019). Additionally, we can theoretically guarantee that the concatenated embedding will generalize well to the target domain under assumptions on the loss function and embedding models.

Theoretical Analysis
Assume there exists a "ground-truth" embedding vector v * s for each sentence s with label y s , and a "ground-truth" linear classifier f * (s)= w * , v * s with a small loss L(f * )=E s [ (f * (s), y s )] w.r.t. some loss function (such as cross-entropy), where E s denotes the expectation over the true data distribution. The superior performance of CAT in practice (see Section 3) suggests that there exists a linear relationship between the embeddings v 1s , v 2s and v * s . Thus we assume a theoretical model: v 1s = P 1 v * s + 1 ; v 2s = P 2 v * s + 2 where i 's are noises independent of v * s with variances σ 2 i 's. If we denote P =[P 1 , P 2 ] and =[ 1 , 2 ], then the concatenationv s =[v 1s , v 2s ] isv s =P v * s + . Let σ= σ 2 1 + σ 2 2 . We present the following theorem which guarantees the existence of a "good" classifierf overv s : Theorem 1. If the loss function L is λ-Lipschitz for the first parameter, and P has full column rank, then there exists a linear classifierf overv s such that Then the difference in the losses is given by More intuitively, if the SVD of P =U ΣV , then (P † ) w * 2 = (Σ † ) V w * 2 . So if the top right singular vectors in V align well with w * , then (P † ) w * 2 will be small in magnitude. This means that if P 1 and P 2 together cover the direction w * , they can capture information important for classification. And thus there exists a good classifierf onv s . Additional explanation is presented in Appendix A.1.

Do Other Combination Methods Work?
There are several sophisticated techniques to combine v 1s and v 2s other than concatenation. Since v 1s and v 2s may be in different dimensions, a dimension reduction technique which projects them on the same dimensional space might work better at capturing the general and domain specific information. We consider two popular techniques: CCA Canonical Correlation Analysis (Hotelling, 1936) learns linear projections Φ 1 and Φ 2 into dimension d to maximize the correlations between Analysis (Schölkopf et al., 1998) first applies nonlinear projections g 1 and g 2 and then CCA on . We empirically evaluate CCA and KCCA and our results (see Section 3) show that the former two perform worse than CAT . Further, CCA performs even worse than the individual embedding models. This is a very interesting negative observation, and below we provide an explanation for this.
We argue that even when v 1s and v 2s contain information important for classification, CCA of the two embeddings can eliminate this and just retain the noise in the embeddings, thereby leading to inferior prediction performance. Theorem 2 constructs such an example.
Theorem 2. Letv s denote the embedding for sentence s obtained by concatenation, andṽ s denote that obtained by CCA. There exists a setting of the data and w * , P, such that there exists a linear classifierf onv s with the same loss as f * , while CCA achieves the maximum correlation but any classifier onṽ s is at best random guessing.
Let the linear classifierf have weights [1, 0, 0, 0, 1, 0] where 0 is the zero vector of d dimensions. Clearly,f (s)=f * (s) for any s, so it has the same loss as f * .
For CCA, since the coordinates of v * s are independent Gaussians, v 1s and v 2s only have correlation in the last d dimensions. Solving the CCA optimization, the projection matrices for both embeddings are the same φ = diag(0, 0, 1, . . . , 1) which achieves the maximum correlation. Then the CCA embedding isṽ s = [0, 0, (v * s ) 3:(d+2) ] where (v * s ) 3:(d+2) are the last d dimensions of v * s , which contains no information about the label. Therefore, any classifier onṽ s is at best random guessing.
The intuition for this is that v 1s and v 2s share com-mon information while each has some special information for the classification. If the two sets of special information are uncorrelated, then they will be eliminated by CCA. Now, if the common information is irrelevant to the labels, then the best any classifier can do with the CCA embeddings is just random guessing. This is a fundamental drawback of the unsupervised CCA technique, clearly demonstrated by the extreme example in the theorem. In practice, the common information can contain some relevant information, so CCA embeddings are worse than concatenation but better than random guessing. KCCA can be viewed as CCA on a nonlinear transformation of v 1s and v 2s where the special information gets mixed non-linearly and cannot be separated out and eliminated by CCA. This explains why the poor performance of CCA is not observed for KCCA in Table 2. We present additional empirical verification of Theorem 2 in Appendix A.2.

Experiments
Datasets We evaluate our approach on seven low resource datasets from NLP text classification tasks like sentiment classification, question type classification, opinion polarity detection, subjectivity classification, etc. We group these datasets into 2 categories: the first having a few hundred training samples (which we term as very small datasets for the remainder of the paper), and the second having a few thousand training samples (which we term as small datasets). We consider the following 3 very small datasets: Amazon (product reviews), IMDB (movie reviews) and Yelp (food article reviews); and the following 4 small datasets: MR (movie reviews), MPQA (opinion polarity), TREC (question-type classification) and SUBJ (subjectivity classification). We present the statistics of the datasets in Table 1 (Li and Roth, 2002) 6 5952 9592 500 SUBJ (Pang and Lee, 2004) 2 10000 21323 1000

Models for Evaluation
We use the BERT (Devlin et al., 2018) base uncased model as the pre-trained model f 1 . We choose a Text-CNN (Kim, 2014) model as the domain specific model f 2 with 3 approaches to initialize the word embeddings: randomly initialized (CNN-R), static GloVe (Pennington et al., 2014) vectors (CNN-S) and trainable GloVe vectors (CNN-NS). We use a regularized logistic regression as the classifier c. We present the model and training details along with the chosen hyperparameters in Appendix B.2-B.3. We also present results with two other popular pre-trained models: GenSen and InferSent in Appendix C.2. We consider two baselines: (i) BERT finetuning (denoted by BERT FT) and (ii) learning c over frozen pre-trained BERT weights (denoted by BERT No-FT). We also present the Adapter (Houlsby et al., 2019) approach as a baseline, which injects new adapters in BERT followed by selectively training the adapters while freezing the BERT weights, to compare with CAT since neither fine-tunes the BERT parameters.
Results on Very Small Datasets On the 3 very small datasets, we present results averaged over 10 runs in  in the number of parameters due to the text-CNN is a function of the vocabulary size of the dataset as it includes the word embeddings which are fed as input to the text-CNN. For a dataset having a larger vocabulary size like SUBJ 3 , Adapter might be more parameter efficient than CAT .

Effect of Dataset Size
We study the effect of size of data on the performance of our method by varying the training data of the MR dataset via random sub-sampling. From Figure 1, we observe that CAT gets the best results across all training data sizes, significantly improving over BERT FT. CAT gets performance comparable to BERT FT on a wide range of data sizes, from 500 points on. We present qualitative analysis and complete results with error bounds in Appendix C.

Conclusion
In this paper we have proposed a simple method for transferring a pre-trained sentence embedding model for text classification tasks. We empirically show that concatenating pre-trained and domain specific sentence embeddings, learned on the target dataset, with or without fine-tuning can improve the classification performance of pre-trained models like BERT on small datasets. We have also provided theoretical analysis identifying the conditions when this method is successful and to explain the experimental results.

Appendix A Theorems: Additional Explanation
A.1 Concatenation Theorem 1. If the loss function L is λ-Lipschitz for the first parameter, and P has full column rank, then there exists a linear classifierf overv s such that L(f ) ≤ L(f * ) + λσ (P † ) w * 2 where P † is the pseudo-inverse of P .

Justification of Assumptions
The assumption of Lipschitz-ness of the loss means that the loss changes smoothly with the prediction, which is a standard assumption in machine learning. The assumption on P having full column rank means that v 1s , v 2s contain the information of v * s and ensures that P † exists. 4 Explanation For intuition about the term (P † ) w * 2 , consider the following simple example. Suppose v * s has 4 dimensions, and w * = [1, 1, 0, 0] , i.e., only the first two dimensions are useful for classification. Suppose P 1 = diag(c, 0, 1, 0) is a diagonal matrix, so that v 1s captures the first dimension with scaling factor c > 0 and the third dimension with factor 1, and P 2 = diag(0, c, 0, 1) so that v 2s captures the other two dimensions. Hence we have (P † ) w * = [1/c, 1/c, 0, 0] , and thus Thus the quality of the classifier is determined by the noise-signal ratio σ/c. If c is small, meaning that v 1s and v 2s mostly contain nuisance, then the loss is large. If c is large, meaning that v 1s and v 2s mostly capture the information along with some nuisance while the noise is relatively small, then the loss is close to that of f * . Note thatf can be much better than any classifier using only v 1s or v 2s that has only part of the features determining the class labels.
A.2 CCA Theorem 2. Letv s denote the embedding for sentence s obtained by concatenation, andṽ s denote that obtained by CCA. There exists a setting of the data and w * , P, such that there exists a linear classifierf onv s with the same loss as f * , while CCA achieves the maximum correlation but any classifier onṽ s is at best random guessing.
Empirical Verification One important insight from Theorem 2 is that when the two sets of embeddings have special information that is not shared with each other but is important for classification, then CCA will eliminate such information and have bad prediction performance. Let r 2s = v 2s − Φ 2 Φ 2 v 2s be the residue vector for the projection Φ 2 learned by CCA for the special domain, and similarly define r 1s . Then the analysis suggests that the residues r 1s and r 2s contain information important for prediction. We conduct experiments for BERT+CNN-non-static on Amazon reviews, and find that a classifier on the concatenation of r 1s and r 2s has accuracy 96.4%. This is much better than 81.3% on the combined embeddings via CCA. These observations provide positive support for our analysis.

B.1 Datasets
In addition to Table 1, here we provide details on the tasks of the datasets and links to download them for reproducibility of results.
• Amazon: A sentiment classification dataset on Amazon product reviews where reviews are classified as 'Positive' or 'Negative'. 5 .
• IMDB: A sentiment classification dataset of movie reviews on IMDB where reviews are classified as 'Positive' or 'Negative' 3 .
• Yelp: A sentiment classification dataset of restaurant reviews from Yelp where reviews are classified as 'Positive' or 'Negative' 3 .
• MR: A sentiment classification dataset of movie reviews based on sentiment polarity and subjective rating (Pang and Lee, 2005)  • TREC: A question type classification dataset with 6 classes for questions about a person, location, numeric information, etc. (Li and Roth, 2002) 8 .
• SUBJ: A dataset for classifying a sentence as having subjective or objective opinions (Pang and Lee, 2004).
The Amazon, Yelp and IMDB review datasets have previously been used for research on few-sample learning by Sarma et al. (2018) and capture sentiment information from target domains very different from the general text corpora of the pre-trained models.

B.2.1 Domain Specific f 2
We use the text-CNN model (Kim, 2014) for domain specific embeddings f 2 the details of which are provided below.
Text-CNN The model restricts the maximum sequence length of the input sentence to 128 tokens, and uses convolutional filter windows of sizes 3, 4, 5 with 100 feature maps for each size. A maxovertime pooling operation (Collobert et al., 2011) is used over the feature maps to get a 384 dimensional sentence embeddings (128 dimensions corresponding to each filter size). We train the model using the Cross Entropy loss with an 2 norm penalty on the classifier weights similar to Kim (2014). We use a dropout rate of 0.5 while training. For each dataset, we create a vocabulary specific to the dataset which includes any token present in the train/dev/test split. The input word embeddings can be chosen in the following three ways: • CNN-R : Randomly initialized 300-dimensional word embeddings trained together with the text-CNN.
• CNN-S : Initialised with GloVe (Pennington et al., 2014) pre-trained word embeddings and made static during training the text-CNN.
• CNN-NS : Initialised with GloVe (Pennington et al., 2014) pre-trained word embeddings and made trainable during training the text-CNN.
For very small datasets we additionally compare with sentence embeddings obtained using the Bag of Words approach.

B.2.2 Pre-Trained f 1
We use the following three models for pre-trained embeddings f 1 : BERT We use the BERT 9 -base uncased model with WordPiece tokenizer having 12 transformer layers. We obtain 768 dimensional sentence embeddings corresponding to the [CLS] token from the final layer. We perform fine-tuning for 20 epochs with early stopping by choosing the best performing model on the validation data. The additional fine-tuning epochs (20 compared to the typical 3) allows for a better performance of the fine-tuning baseline since we use early stopping.
InferSent We use the pre-trained InferSent model (Conneau et al., 2017) to obtain 4096 dimensional sentence embeddings using the implementation provided in the SentEval 10 repository. We use InferSent v1 for all our experiments.

GenSen
We use the pre-trained GenSen model (Subramanian et al., 2018) implemented in the SentEval repository to obtain 4096 dimensional sentence embeddings.

B.3 Training Details
We train domain specific embeddings on the training data and extract the embeddings. We combine these with the embeddings from the pre-trained models and train a regularized logistic regression classifier on top. This classifier is learned on the training data, while using the dev data for hyperparameter tuning the regularizer penalty on the weights. The classifier can be trained either by freezing the weights of the embedding models or training the whole network end-to-end. The performance is tested on the test set. We use test accuracy as the performance metric and report all results averaged over 10 experiments unless mentioned otherwise. The experiments are performed on an NVIDIA Titan Xp 12 GB GPU.

B.3.1 Hyperparameters
We use an Adam optimizer with a learning rate of 2e −5 as per the standard fine-tuning practice. For CCA , we used a regularized CCA implementation and tune the regularization parameter via grid search in [0.00001, 10] in multiplicative steps of 10 over the validation data. For KCCA , we use a Gaussian kernel with a regularized KCCA implementation where the Gaussian sigma and the regularization parameter are tuned via grid search in [0.05, 10] and [0.00001, 10] respectively in multiplicative steps of 10 over the validation data. For CAT and CAT , the weighting parameter α is tuned via grid search in the range [0.002, 500] in multiplicative steps of 10 over the validation data.

C.1 Qualitative Analysis
We present some qualitative examples from the Amazon, IMDB and Yelp datasets on which BERT and CNN-NS are unable to provide the correct class predictions, while CAT or KCCA can successfully provide the correct class predictions in Table 4. 10 https://github.com/facebookresearch/SentEval Correctly classified by KCCA However-the ringtones are not the best, and neither are the games. This is cool because most cases are just open there allowing the screen to get all scratched up. Correctly classified by CAT TNot nearly as good looking as the amazon picture makes it look . Magical Help .

(a) Amazon
Correctly classified by KCCA I would have casted her in that role after ready the script . Predictable , but not a bad watch . Correctly classified by CAT I would have casted her in that role after ready the script . Predictable , but not a bad watch .

(b) IMDB
Correctly classified by KCCA The lighting is just dark enough to set the mood . I went to Bachi Burger on a friend's recommendation and was not disappointed . dont go here . I found this place by accident and I could not be happier .

Correctly classified by CAT
The lighting is just dark enough to set the mood . I went to Bachi Burger on a friend's recommendation and was not disappointed . dont go here . I found this place by accident and I could not be happier .  We observe that these are either short sentences or ones where the content is tied to the specific reviewing context as well as the involved structure to be parsed with general knowledge. Such input sentences thus require combining both the general semantics of BERT and the domain specific semantics of CNN-NS to predict the correct class labels.

C.2 Complete Results with Error Bounds
We present a comprehensive set of results along with error bounds on very small datasets (Amazon, IMDB and Yelp reviews) in Table 2, where we evaluate three popularly used pre-trained sentence embedding models, namely BERT, GenSen and In-ferSent. We present the error bounds on the results for small datasets in Table 3. For small datasets, we additionally present results from using CCA (We omit KCCA here due to high computational memory requirements).   Table 6: Test accuracy (± std dev) for MR, MPQA, SUBJ and TREC datasets. Best results on the datasets are highlighted in boldface. The domain specific embedding model used is CNN-non-static, and the pre-trained model used is BERT.