{'fraction_non_alphanumeric': 0.029092662541187815, 'fraction_numerical': 0.011291489190709636, 'mean_word_length': 5.020077406869859, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models

There is growing evidence that pretrained language models improve task-specific fine-tuning not just for the languages seen in pretraining, but also for new languages and even non-linguistic data. What is the nature of this surprising cross-domain transfer? We offer a partial answer via a systematic exploration of how much transfer occurs when models are denied any information about word identity via random scrambling. In four classification tasks and two sequence labeling tasks, we evaluate baseline models, LSTMs using GloVe embeddings, and BERT. We find that only BERT shows high rates of transfer into our scrambled domains, and for classification but not sequence labeling tasks. Our analyses seek to explain why transfer succeeds for some tasks but not others, to isolate the separate contributions of pretraining versus fine-tuning, and to quantify the role of word frequency. These findings help explain where and why cross-domain transfer occurs, which can guide future studies and practical fine-tuning efforts.


Introduction
Fine-tuning pretrained language models has proven to be highly effective across a wide range of NLP tasks; the leaderboards for standard benchmarks are currently dominated by models that adopt this general strategy (Rajpurkar et al., 2016(Rajpurkar et al., , 2018Wang et al., 2018;Yang et al., 2018;Wang et al., 2019). Recent work has extended these findings in even more surprising ways: Artetxe et al. (2020), Karthikeyan et al. (2019), and Tran (2020) find evidence of transfer between natural languages, and Papadimitriou and Jurafsky (2020) show that pretraining language models on non-linguistic data such as music and computer code can improve test performance on natural language. Starting with a model (e.g., pretrained BERT, GloVeinitialized LSTM, etc.), we copy it and fine-tune it on the regular and scrambled train set using a scrambling function F. The model is then evaluated on regular and scrambled test sets. Our paper explores different options for F and a number of variants of our models to try to quantity the amount of transfer and identify its sources.
Why does pretraining help even across what appear to be fundamentally different symbolic domains, and what are the limits of such cross-domain transfer? In this work, we seek to inform these questions via a systematic exploration of how much cross-domain transfer we see when the model is denied any information about word identity. Figure 1 gives an overview of our core experimental paradigm: starting with two identical copies of a single pretrained model for English, we finetune one on English examples and the other on scrambled English sentences, using a scrambling function F (Section 3), and then we evaluate the resulting models. We apply this paradigm to four classification tasks and two sequence modeling tasks, and we evaluate bag-of-words baselines, LSTMs with GloVe initialization and rich attention mechanisms, and BERT. Our central finding is that only BERT is able to achieve robust crossdomain transfer, and for classification tasks but not sequence labeling ones.
To try to understand why such transfer is successful for some tasks but not others, we pursue a number of hypotheses. First, we consider whether using a scrambling function F that matches word frequencies is required for transfer, and we find that such matching plays a small role, but not enough to account for the observed performance (Section 7.1). Second, we assess whether frequency matching might actually be inserting semantic consistency into the scrambling process by, for example, systematically creating substitution pairs like good/great and professor/teacher (Section 7.2). However, we find no evidence of such semantic consistency. Third, we try to isolate the contribution of pretraining versus fine-tuning by fine-tuning randomly initialized models of different sizes (Section 7.3) and by freezing the BERT parameters, such that only task-specific parameters are updated (Section 7.4). These variations lead to a substantial drop in transfer, suggesting that fine-tuning is vital, although our LSTM results show that the BERT pretrained starting point is also an essential component. While these findings do not fully account for the transfer we observe, they offer a partial explanation which should help guide future studies of this issue and which can help with practical fine-tuning work in general.

Related work 2.1 Evidence for Transfer
Transferability across domains is often used to benchmark large pretrained models such as BERT (Devlin et al., 2019a), RoBERTa (Liu et al., 2019b), ELECTRA (Clark et al., 2019), and XL-Net (Yang et al., 2019). To assess transferability, pretrained models are fine-tuned for diverse downstream tasks (Wang et al., 2018(Wang et al., , 2019. Recently, pretrained Transformer-based models (Vaswani et al., 2017) have even surpassed estimates of human performance on GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019). While the benefits of pretraining are reduced when there is a large train set (Hernandez et al., 2021), there is little doubt that this pretraining process helps in many scenarios.

Studies of Why Transfer Happens
There are diverse efforts underway to more deeply understand why transfer occurs. Probing tests often involve fitting supervised models on internal representations in an effort to determine what they encode. Such work suggests that BERT representations encode non-trivial information about morphosyntax and semantics (Tenney et al., 2019;Liu et al., 2019a;Hewitt and Manning, 2019;Manning et al., 2020) and perhaps weakly encode world knowledge such as relations between entities (Da and Kasai, 2019;Petroni et al., 2019), but that they contain relatively little about pragmatics or rolebased event knowledge (Ettinger, 2020). Newer feature attribution methods (Zeiler and Fergus, 2014;Springenberg et al., 2015;Shrikumar et al., 2017;Binder et al., 2016;Sundararajan et al., 2017) and intervention methods (McCoy et al., 2019;Vig et al., 2020;Geiger et al., 2020) are corroborating these findings while also yielding a picture of the internal causal dynamics of these models.
Another set of strategies for understanding transfer involves modifying network inputs or internal representations and studying the effects of such changes on task performance. For instance, Tamkin et al. (2020) show that BERT's performance on downstream GLUE tasks suffers only marginally even if some layers are reinitialized before finetuning, and Gauthier and Levy (2019), Pham et al. (2020), and Sinha et al. (2021) show that BERTlike models are largely insensitive to word order changes.

Extreme Cross-Domain Transfer
Cross-domain transfer is not limited to monolingual cases (Karthikeyan et al., 2019). With modifications to its tokenizer, English-pretrained BERT improves performance on downstream multilingual NLU tasks (Artetxe et al., 2020;Tran, 2020). Papadimitriou and Jurafsky (2020) show that pretraining language models on structured non-linguistic data (e.g., MIDI music or Java code) improves test performance on natural language. Our work advances these efforts along two dimensions. First, we challenge models with extremely ambitious cross-domain settings and find that BERT shows a high degree of transfer, and we conduct a large set of follow-up experiments to help identify the sources and limitations of such transfer.

Scrambling Method Sentence
Original English (No Scrambling) "the worst titles in recent cinematic history" Similar Frequency "a engaging semi is everyone dull dark" Random "kitsch theatrically tranquil andys loaf shorty lauper" Table 1: An example from the SST-3 dataset and its two scrambled variants.

Experimental Paradigm
We now describe the evaluation paradigm summarized in Figure 1 (Section 3.1), with special attention to the scrambling functions F that we consider (Sections 3.2-3.3).

Evaluation Pipeline
Figure 1 shows our main evaluation paradigm for testing the transferability of a model without word identity information. On the left side, we show the classic fine-tuning pipeline (i.e., we fine-tune on the original English training set and evaluate on the original English test set). On the right side, we show our new evaluation pipeline: starting from a single model, we (1) fine-tune it with a corrupted training split where regular English word identities are removed and then (2) evaluate the model on a version of the evaluation set that is corrupted in the same manner. The paradigm applies equally to models without any pretraining and with varying degrees of pretraining for their model parameters.

Scrambling with Similar Frequency
To remove word identities, we scrambled each sentence in each dataset by substituting each word w with a new word w in the vocabulary of the dataset. For Scrambling with Similar Frequency, we use the following rules: 1. w and w must have the same sub-token length according to the BERT tokenizer; and 2. w and w must have similar frequency.
The first rule is motivated by the concern that subtoken length may correlate with word frequency, given that rarer and longer words may be tokenized into longer sub-tokens. The second rule is the core of the procedure. The guiding idea is that word frequency is often reflected in learned embeddings (Gong et al., 2018), so this scrambling procedure might preserve useful information and thus help to identify the source of transfer. Table 5 shows an example, and Appendix C provides details about the matching algorithm and additional examples of scrambled sentences.

Random Scrambling
To better understand the role of frequency in domain transfer, we also consider a word scrambling method that does not seek to match word frequencies. For this, we simply shuffle the vocabulary and match each word with another random word in the vocabulary without replacement. We include the distributions of the difference in frequency for every matched word pair in Appendix C to make sure a word is paired with a new word with drastically different frequency in the dataset. We also tried to pair words by the reverse order of frequencies, which yielded similar results, so we report only random scrambling results here.

Models
In this section, we describe the models we evaluated within our paradigm. Appendix B provides additional details about how the models were designed. Dummy Model We include a random classifier that generates predictions randomly proportional to the class distribution of the training set. We use this model to further contextualize our results.

Tasks
We consider six sequence classification and sequence labeling tasks ( We select two datasets covering distinct tasks: part-of-speech detection (POS) and named entity recognition (NER). We used Universal Dependencies English Web Treebank (EN-EWT) (Silveira et al., 2014) for POS, and CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003) for NER. For sequence labeling tasks, we used Micro-F1 (i.e., accuracy with full labels) for POS and F1 scores for NER.

Results
In this section, we analyze the fine-tuning performance of BERT on scrambled datasets. Table 3 shows performance results. We focus for now on the results for Scrambling with Similar Frequency. Additionally, we also include baseline models trained with original sentences for comparison purposes. When training models on each task, we select models based on performance on the dev split during fine-tuning. We average performance results with multiple random seeds to get stabilized results. See Appendix B for additional details on our training and evaluation procedures.

Sequence Classification
Comparing the second column (BERT model that is trained and tested on English) with the sixth column (BERT model that is trained and tested on  Scrambled English with Similar Frequency Scrambling) in Table 3, we see that BERT maintains strong performance for all sequence classification tasks even when the datasets are scrambled. More importantly, we find that BERT fine-tuned with a scrambled dataset performs significantly better than the LSTM model (with GloVe embeddings) trained and evaluated on standard English data For example, on the MRPC task, BERT evaluated with scrambled data experiences a less than 5% performance drop, and shows significantly better performance (a 13.9% improvement) than the best LSTM model. BERT evaluated with scrambled QNLI experiences the biggest drop (a 9.89% decrease). However, this still surpasses the best LSTM performance by a large margin (a 20.6% improvement). Table 3 also presents performance results for other baseline models, which can be used to assess the intrinsic difficulty of each task. Our results suggest that BERT models fine-tuned with scrambled tasks remain very strong across the board, and they remain stronger than best LSTM baseline models (i.e., trained and tested on regular English) in all the classification tasks.
The overall performance of the LSTM models is worth further attention. The LSTMs are far less successful at our tasks than the BERT models. However, it seems noteworthy that scrambling does not lead to catastrophic failure for these models. Rather, they maintain approximately the same performance in the scrambled and unscrambled conditions. This might seem at first like evidence of some degree of transfer. However, as we discuss in Section 7.3, the more likely explanation is that the LSTM is simply being retrained more or less from scratch in the two conditions.

Sequence Labeling
For a more complex setting, we fine-tuned BERT with sequence labeling tasks, and evaluated its transferability without word identities (i.e., using datasets that are scrambled in the same way as in our sequence classification tasks). The second set (bottom set) of Table 3 shows performance results for sequence labeling tasks where the goal of the BERT model is to classify every token correctly. As shown in Table 3, BERT experiences a significant drop when evaluated with a scrambled dataset for a sequence labeling task. For LSTMs trained with scrambled sequence labeling tasks, we also observe bigger drops compared with sequence classification tasks. For CoNLL-2003, LSTM with GloVe embeddings drops (a 18.7% decrease) from its baseline counterpart. Our results suggest that transfer learning without word identities is much harder for sequence labeling tasks. One intuition is that sequence labeling tasks are more likely to rely on word identities given the fact that classification (i.e., labeling) is at the token-level.

Frequency Effects
Preserving word frequencies during scrambling may lead to higher performance when training and evaluating on scrambled datasets. To assess how much of the observed transfer relates to this factor, we can compare Scrambling with Similar Frequency (SSF) with Random Scrambling (RS), as described in Section 3. As shown in Table 3, performance drops slightly if we use RS. For sequence classification tasks, RS experiences 1-5% drops in performance compared with SSF. For sequence labeling tasks, the difference is slightly larger: about 2-6%. This suggests that word frequency is indeed one of the factors that affects transferability, though the differences are relatively small, indicating that this is not the only contributing factor. This is consistent with similar findings due to Karthikeyan et al. 2019 for multilingual BERT.

Does Scrambling Preserve Meaning?
Another explanation is that our scrambling methods tend to swap words that are predictive of the same labels. For example, when we are substituting words with similar frequencies in SST-3, "good" may be swapped with "great" since they may have similar frequencies in a sentiment analysis dataset.
To rule this out, we conducted zero-shot evaluation experiments with our BoW model on sequence classification tasks. The rationale here is that, to the extent that our swapping preserved the underlying connection between features and class labels, this should show up directly in the performance of the BoW model. For example, just swapping of "good" for "great" would hardly affect the final scores for each class. If there are a great many such invariances, then it would explain the apparent transfer. Figure 2 shows the zero-shot evaluation results of our BoW model on all sequence classification datasets. Our results suggest that both scrambling methods result in significant performance drops, which suggests that word identities are indeed destroyed by our procedure, which again shines the spotlight on BERT as the only model in our experiments to find and take advantage of transferable information.

Transfer or Simple Retraining?
Our results on classification tasks show that English-pretrained BERT can achieve high performance when fine-tuned and evaluated on scrambled data. Is this high performance uniquely enabled by transfer from BERT's pretrained representations, or is BERT simply re-learning the token identities from its scrambled fine-tuning data?
To distinguish between these two hypotheses, we first examine whether randomly-initialized BERT models can also achieve high performance when fine-tuned and evaluated on scrambled data. We study models of varying capacity by modulating the number of BERT Transformer layers. See Appendix B for details about the training procedure for these randomly-initialized models.
We compare these varying-depth randomlyinitialized models against BERT models pretrained on English. To modulate the capacity of these pretrained models, we progressively discard the later Transformer layers (i.e., we make predictions from intermediate layers). Comparing these models is a step toward disentangling the performance gains of pretraining from the performance gains relating to model capacity. Figure 3 summarizes these experiments. The red line represents our fine-tuning results, across different model sizes. The shaded area represents the performance gain from pretraining when training and testing on scrambled data. Pretraining yields consistent gains across models of differing depths, with deeper models seeing greater gains.
For sequence labeling tasks, the patterns are drastically different: the areas between the two lines are small. Since the random-initialized and pretrained models achieve similar performance when fine-tuned and tested on scrambled data, pretraining is not beneficial. This suggests that BERT hardly transfers knowledge when fine-tuned for sequence labeling with scrambled data. Figure 3: Performance results when fine-tuning end-to-end for different number of Transformer layers. Annotated numbers are the differences between the red lines and the green lines in percentages. Scoring for each task is defined in Section 5. Table 4 shows our results when training LSTMs without any pretrained embeddings. Unlike with BERT, GloVe initialization (a pretraining step) hardly impacts model performance across all tasks. Our leading hypothesis here is that the LSTMs may actually relearn all weights without taking advantage of pretraining. All of our LSTM models have parameter sizes around 1M, whereas the smallest BERT model (i.e., with a single Transformer layer) is around 3.2M parameters. Larger models may be able to rely more on pretraining.
Overall, these results show that we do see transfer of knowledge, at least for classification tasks, but that there is variation between tasks on how much transfer actually happens.

Assessing Transfer with Frozen BERT Parameters
We can further distinguish the contributions of pretraining versus fine-tuning by freezing the BERT parameters and seeing what effect this has on crossdomain transfer. Ethayarajh (2019) provides evidence that early layers are better than later ones for classifier fine-tuning, so we explore the effects of this freezing for all the layers in our BERT model. As shown in Figure 4, performance scores drop significantly if we only fine-tune the classifier head and freeze the rest of the layers in BERT across three of our tasks. However, we find that performance scores change significantly depending on which layer we append the classifier head to. Consistent with Ethayarajh's findings, contextualized embeddings in lower layers tend to be more predictive. For example, if we freeze BERT weights and use the contextualized embeddings from the 2nd layer for SST-3, the model reaches peak performance compared with contextualized embeddings from other layers. More importantly, the trend of the green line follows the red line in Figure 4, especially for SST-3 and QNLI. The only exception is MRPC, where the red line plateaus but the green line keeps increasing. This could be an artifact of the size of the dataset, since MRPC only contains around 3.7K training examples. Our results suggest that pretrained weights in successive selfattention layers provide a good initial point for the fine-tuning process.

Conclusion
In this paper, we propose an evaluation pipeline for pretrained models by testing their transferability without word identity information. Specifically, we take an English pretrained BERT off-the-shelf and fine-tune it with a scrambled English dataset. We conduct analyses across six tasks covering both classification and sequence labeling. By evaluating performance against multiple baselines, we aim to assess where BERT can transfer knowledge even without word identities. We find considerable transfer for BERT as compared to even powerful baselines, by only for classification tasks.
What is the source of successful cross-domain transfer with BERT? We find that word frequency Figure 4: Performance results when fine-tuning only the classifier head by freezing all proceeding layers in BERT (red line) vs. fine-tuning end-to-end, which includes the classifier head and all proceeding layers in BERT (green line). Numbers are scores for the red lines. Scoring for each task is defined in Section 5.
contributes, but only to a limited extent: scrambling with matched word frequencies consistently outperforms scrambling with unmatched word frequencies, but transfer still occurs robustly even with random scrambling. We are also able to determine that both pretraining and fine-tuning are important and interacting factors in this transfer; freezing BERT weights during task-specific training leads to much less transfer, but too much task-specific training erodes the benefits of pretraining and in turn reduces the amount of transfer observed.
These analyses begin to piece together a full account of these surprising transfer results for BERT, but they do not fully explain our experimental results. Recent literature suggests at least two new promising avenues to explore. First, Sinha et al. (2021) seek to help characterize the rich distributional prior that models like BERT may be learning, which suggests that higher-order notions of frequency play a significant role in transfer. Second, the findings of Ethayarajh (2019) may be instructive: through successful layers, BERT seems to perform specific kinds of dimensionality reduction that help with low-dimensional classification tasks. Our results concerning layer-wise variation are consistent with this. And there may be other paths forward. The more we can learn about the extent of cross-domain transfer, the more effectively we can train and fine-tune these models on challenging tasks.

C Frequency Matching
To study the effect of word frequencies on the transferability of BERT, we control word frequencies when scrambling sentences. Figure 5 shows the differences in frequencies of matched pairs. Our results show that the difference in frequency for a frequency-matched pair is significantly smaller than a randomly matched pair.
To match word frequency during scrambling, we first preprocess sentences by lower-casing and separating by spaces and punctuation. We then use the original BERT WordPiece tokenizer to determine the sub-token length for each word, where the sub-token length is the number of word pieces a word contains. To randomly match words with similar frequencies, we first bucket words by their sub-token length. Then, we iterate through words within each bucket in the order of word frequencies.
For each word, we use the round-robin method to find the closest neighbor with the closest frequency.
A perfect match is not always possible as not every word can be paired with another word with an identical word frequency. We include the distributions of the difference in frequency for every matched word pair in Appendix C to illustrate word frequencies are preserved.

D Scrambed Sentence
In Table 5 and Table 6, we provide one example sentence from each dataset destructed by our 4 scrambling methods. We also include the original English sentence (OR) at the top for each dataset.