{'fraction_non_alphanumeric': 0.03863623296728569, 'fraction_numerical': 0.013913643419766572, 'mean_word_length': 4.594724991958829, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On Measuring the Intrinsic Few-Shot Hardness of Datasets

While advances in pre-training have led to dramatic improvements in few-shot learning of NLP tasks, there is limited understanding of what drives successful few-shot adaptation in datasets. In particular, given a new dataset and a pre-trained model, what properties of the dataset make it \emph{few-shot learnable} and are these properties independent of the specific adaptation techniques used? We consider an extensive set of recent few-shot learning methods, and show that their performance across a large number of datasets is highly correlated, showing that few-shot hardness may be intrinsic to datasets, for a given pre-trained model. To estimate intrinsic few-shot hardness, we then propose a simple and lightweight metric called"Spread"that captures the intuition that few-shot learning is made possible by exploiting feature-space invariances between training and test samples. Our metric better accounts for few-shot hardness compared to existing notions of hardness, and is ~8-100x faster to compute.


Introduction
A growing body of recent work has shown impressive advances in few-shot adaptation of pretrained transformers (Radford et al., 2019;Schick and Schütze, 2020;Brown et al., 2020;Karimi Mahabadi et al., 2021;Liu et al., 2021a, among others). Despite this progress, there is no concrete understanding of when and why few-shot learning may be successful for a given pre-trained model. Indeed, no free lunch style arguments necessitate the existence of tasks that are not few-shot learnable by a given pre-trained model, regardless of the adaptation method, and in practice, very similar datasets exhibit varying levels of success when state-of-the-art few-shot adaptation methods are applied (Fig. 1a) Figure 1: (a) Fine-tuning a model on D 1 (D 2 ) on the entire dataset leads to a +45.3 (+35) point improvement over a random baseline. However, most methods are successful at few-shot adaptation on D 1 with an average improvement of~45 points over a random baseline, while few-shot adaptation is unsuccessful on D 2 . (b) We observe that features of test inputs are closer to training set inputs for D 1 than D 2 , motivating Spread as a metric for evaluating few-shot hardness.
This work advances understanding of few-shot learnability in two ways. First, we find that, given a dataset, few-shot performance of various adaptation methods is highly correlated. This suggests the existence of adaptation-method independent factors behind few-shot learnability of a dataset, for a given pre-trained model. Next, we propose a simple and lightweight metric to estimate this intrinsic few-shot learnability. Concretely, we consider an extensive set of recently proposed adaptation methods and find that on a wide range of datasets, these methods have highly correlated behaviors i.e., the degree to which a few-shot adaptation method succeeds on a dataset is correlated across methods.
Next, we consider two recently proposed methods that may be used for assessing intrinsic dataset hardness-Rissanen Data Analysis (RDA, Perez et al. (2021)) and Sensitivity Analysis (SA, Hahn et al. (2021)). RDA computes dataset hardness as the area under the curve of the test loss as a function of number of training samples, while SA computes hardness by examining how perturbations in the input features cause a model to change its predicted label. From experiments, we show that SA is poorly correlated with few-shot hardness and RDA, while well correlated, is very expensive to compute. In response, we propose a new metric that measures the ability of a model to exploit feature-space invariances between the train and test set to make few-shot generalizations. For instance, consider the datasets in Fig. 1b, where D 1 has a test set that "looks similar" to the training set while D 2 has a test set that looks dissimilar to the training set. We capture this intuition into a lightweight metric called Spread and show that Spread correlates as well or better than existing methods (Spearman correlation of 0.467 vs 0.356) while being~8-100x more computationally efficient.

Background
Consider a labeled dataset D = {z (1) , z (2) , . . .}, split into a training set D tr and a test set D ts , where each example z (k) is a tuple consisting of an input x (k) and a label y (k) . Typically, in k-way l-shot learning (l is typically less than 128), D tr consists of l examples of each of the k labels. Given some pre-trained model f , a few-shot adaptation method m uses D tr to modify f , outputting an "adapted model" that can make predictions on D ts .
State-of-the-art approaches for such adaptation typically involve either recasting the task into the pre-training objective of the model (Gao et al., 2021a;Tam et al., 2021), or using lightweight / parameter efficient finetuning (Houlsby et al., 2019;Logan IV et al., 2022;Li and Liang, 2021;Liu et al., 2021b;Ben Zaken et al., 2022). For this work, we experiment with an extensive set of recently proposed few-shot adaptation methods that we further categorize into prompt-based methods which includes LMBFF (Gao et al., 2021a), AdaPET (Tam et al., 2021), Null Prompts (Logan IV et al., 2022) and Prompt-Bitfit (Ben Zaken et al., 2022) 1 , and Light-weight finetuning methods which includes 1 As demonstrated by (Logan IV et al., 2022), we use trigger prompts in BitFit to improve few-shot performance. "prompt-fs" refers to prompt-based methods while "lw-fs" refers to lightweight finetuning methods. We note a large correlation of 0.85 and 0.77. We also observe a higher correlation among methods from the same category.

Intrinsic Few-Shot Hardness
For a fixed method and dataset, we define "method specific few-shot hardness" (MFH) as a function that takes a dataset D and some adaptation method m, and outputs a real number that captures fewshot hardness of D with respect to f and m. Concretely, for this work, we define MFH(D, f, m) as the classification accuracy of the adapted model, normalized against the classification accuracy of the majority baseline, on D ts .
Datasets. In FS-GLUE, we consider 11 tasks (details in Appendix A.1) from the GLUE  and SuperGLUE  benchmarks, covering a wide range of task formats and objectives. Additionally, since these tasks have differing textual formats as well as performance metrics (F1 / accuracy / Pearson's correlation), we curate FS-NLI where every dataset is recast into 2 way NLI, and the performance metric is standardized as classification accuracy. To do this, we take datasets from White et al.

Experiments
We compute MFH values for all adaptation methods on FS-GLUE and FS-NLI. For each dataset, we sample a fixed training set consisting of 64 examples per label. Finally, we control for hyperparameter tuning across methods (details in Appendix A.2). We report the average spearman correlation between MFH values for all pairs of adaptation methods for every dataset in Fig. 2.
Results. We observe an extremely high average correlation between hardness measurements for different methods-0.85 and 0.77 for FS-GLUE and FS-NLI respectively. The average correlation is higher among methods within the same category than among methods from different categories e.g., on FS-GLUE, the average correlation between prompt-based and lightweight finetuning methods is 0.8, while the average correlation among all prompt-based methods is 0.89. We conclude that few-shot hardness may be intrinsic to datasets, i.e., for a given pre-trained model, a dataset may be "easy" or "hard" regardless of the adaptation method used.

Automatic Metrics For Intrinsic Few-Shot Hardness
Noting that the experiments of Section 3 suggest that few-shot hardness may be intrinsic to datasets, we define the "intrinsic few-shot hardness" (IFH) of a dataset as the mean MFH values across a collection of adaptation method M = {m 1 , m 2 , . . .}, Clearly, computing IFH values for some dataset is computationally intensive since it requires running multiple few-shot adaptation methods on f . Thus, to automatically estimate IFH, an automatic hardness metric h takes f and D and outputs a (real-valued) score h(f, D) that is well-correlated with IFH(f, D) across some collection of datasets {D 1 , D 2 , . . .}.

Existing Metrics
We consider two recently proposed dataset hardness metrics that have been applied in the context of pre-trained models. h sensitivity (Hahn et al., 2021) measures how many tokens of an input need to be  (Perez et al., 2021) approximates hardness as the area under the curve of the test loss obtained by finetuning f on successively larger slices of the training data-a large area implies that a large number of training samples are required to achieve low test set loss, implying that the dataset is hard for f .

Our Approach
In the few-shot regime, one of the factors affecting dataset hardness is the degree to which input features for a given label in the test set differ from the training set (Fig. 1b). Based on this intuition, we propose a simple few-shot specific hardness metric called Spread that computes the average euclidean distance of test input features to the closest training set input. For some input z = (x, y) ∈ D, let f (z) denote the vector valued features of the input x produced by the model. We define the distance between some example z (k) to the training set D tr as, (2) subject to: z = (x, y) ∈ D tr y = y (k) .
Then, h Spread (f , D) can be computed as

Experiments
We experiment with RDA, SA and Spread as our few-shot hardness metrics. To obtain input features for computing Spread, we use SimCSE (Gao et al., 2021b) features of the input with a RoBERTa-large base. Given a collection of datasets and methods,  Table 2: Correlation between hardness measurements from using the same adaptation method with different pre-trained models. We observe a high correlation among hardness measurements from using different base pre-trained models.
we measure the correlation between metric outputs and IFH values for each dataset, and report the average across all datasets. We also report time taken for computing the metric on a single 1080ti GPU for each dataset, and report the average compute time across all datasets.
Results. We report results on FS-NLI to ensure uniformity of task formats and performance metrics. From Table 1, we observe that h sensitivity is poorly correlated with IFH. Next, we note that while h RDA produces better hardness judgements, these come at the cost of increased computation time. Finally, h Spread is able to best account for intrinsic few-shot hardness while being~100x computationally lightweight compared to h RDA .

Measuring IFH across pre-trained models
While we define IFH of a dataset with respect to a fixed pre-trained model, certain datasets or tasks might be few-shot hard for a wider range of pretrained models. To investigate this further, we experiment with Electra-large (Clark et al., 2020) as the base pre-trained model. We obtain hardness measurements on FS-NLI and compute the correlation between methods with different base pretrained models (RoBERTa-large vs Electra-large). From average correlation across datasets in Table 2, we find that hardness measurements from the same method with different pre-trained models are wellcorrelated. We conclude that there may even be "pre-trained model independent" factors behind intrinsic few-shot hardness, and we leave further analysis of this to future work.

Decreasing Few-Shot hardness via dataset decomposition
We conclude experiments with a simple application of Spread by proposing a training set sampling strategy that minimizes Spread values for  Table 3: Comparing a Spread-inspired training set sampling strategy to random sampling. We observe an average boost of 13.9 accuracy points compared to the control where the~3 point boost is due to ensembling.
improved few-shot performance. Our strategy is based on performing a "clustering based decomposition" of a dataset, similar to Murty et al. (2021).
In particular, we run k-means clustering on input features from the pre-trained model to create P clusters. Then, we train P distinct models on fewshot training sets sampled from each of the clusters. At test time, examples are classified into one of the P clusters, and the corresponding model is used to make predictions. To account for any effects due to ensembling, we compare with an approach where we randomly sample a model to make predictions, instead of using the model corresponding to the cluster.
Results. We experiment with the sampling strategy described above on a named entity classification based dataset from FS-NLI. From Table 3, we note an improvement of~14 accuracy points, while the control increases by only a~3 accuracy points.

Related Work and Discussion
Most notions of what makes datasets challenging are either model agnostic-example length (Spitkovsky et al., 2010), order sensitivity (Nie et al., 2019) etc, or indirect (Agarwal and Hooker, 2020). On the other hand, intrinsic hardness as defined in this work, is both model-centric and directly measures the ability of a specific model family to make good generalizations from a training set. While measuring hardness of datasets has seen some recent traction (Hahn et al., 2021;Perez et al., 2021), to the best of our knowledge, we are the first to study hardness in the context of few-shot adaptation. We show that few-shot hardness of datasets (as measured by test set performance normalized against a random baseline) among an extensive set of recently proposed methods is highly correlated, suggesting that few-shot hardness may be a property intrinsic to datasets. We then propose a simple hardness metric called Spread, based on the intuition that a test set with input features close to the few-shot training set is easy, since it allows a model to exploit feature-space invariances. Compared to other metrics, Spread provides hardness judgements that are much better correlated with intrinsic few-shot hardness while being 8-100x faster to compute, compared to prior hardness metrics. Metrics for predicting fewshot hardness have several applications. For the NLP practitioner, Spread could be used as a simple plug-and-play estimator of whether few-shot adaptation of a pretrained model might be successful for a given use case. For the dataset developer, Spread could be used to adversarially curate harder few-shot benchmarks. Finally, a model developer can use Spread to inform better training set sampling strategies to improve test set performance in the few-shot training regime, as well as for model selection by selecting models with lower Spread.

Acknowledgements
SM was funded by a gift from Apple Inc. We are grateful to John Hewitt, Eric Mitchell, Roy Schwartz and the anonymous reviewers for helpful comments.

Limitations
Instance-level Analysis. We focus on discovering and measuring intrinsic few-shot hardness at the dataset level and do not study instance-level hardness quantitatively or qualitatively. Understanding hardness at the instance level can further help understand the recent successes behind fewshot learning in NLP.
Base model selection. We compare few-shot performance between a variety of methods with two base models that have very different pre-training objectives, yet have correlated few-shot behaviors across tasks and adaptation methods. Of course, there exist a much wider range of pre-trained models with very diverse pre-training data and objectives. While beyond the scope of this work, we believe that studying factors such as the relationship between few-shot performance and pre-training data / objectives is worth further investigation.    Table 4.

A.2 Implementation Details
Hyper-parameter Details: For grid search, we choose a learning rate from the set 5e-6, 1e-5, 5e-5 and train for 30 epochs. For each method, we train for 1000 steps, evaluating every 100 steps. All adaptation methods use RoBERTa-large as the base model. All other hyperparameters are as used in the original works. We follow the original work to generate and select prompts for LMBFF. For AdaPET, we use the same prompts as we used in LMBFF.

A.3 Intrinsic few-shot hardness
In Fig. 2, we show averaged spearman correlations of methods clustered according to their type. In Fig. 3 and Fig. 4, we further show detailed correlations between every pair of methods. We generally observe high correlations across methods for both FS-GLUE and FS-NLI tasks. In Fig. 4, we further add h Spread , h RDA , and h sensitivity as references.

A.4 Decreasing Few-Shot hardness via dataset decomposition (Addendum)
We present the strategy of clustering based decomposition on named entity classification task from FS-NLI and demonstrates its effectiveness in Table 3. Since there naturally exist heuristics based cluster (e..g., questions about location, organiza-  Table 5: Jaccard index between each pair of clusters, where 1 to 6 denote the 6 clusters generated by k-means clustering over the representation space and person, entity, and etc denote the heuristics based clusters. tion, and person) in NEC, we here show the relevance between model-generated clusters and these heuristics based cluster by computing the Jaccard index. From Table 5, we can observe that modelbased clusters are not decomposing the dataset in a similar way as the heuristics.