{'fraction_non_alphanumeric': 0.02982860909690178, 'fraction_numerical': 0.0115771259063942, 'mean_word_length': 4.910153396639883, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Switching Contexts: Transportability Measures for NLP

This paper explores the topic of transportability, as a sub-area of generalisability. By proposing the utilisation of metrics based on well-established statistics, we are able to estimate the change in performance of NLP models in new contexts. Defining a new measure for transportability may allow for better estimation of NLP system performance in new domains, and is crucial when assessing the performance of NLP systems in new tasks and domains. Through several instances of increasing complexity, we demonstrate how lightweight domain similarity measures can be used as estimators for the transportability in NLP applications. The proposed transportability measures are evaluated in the context of Named Entity Recognition and Natural Language Inference tasks.


Introduction
The empirical evaluation of the quality of NLP models under a specific task is a fundamental part of the scientific method of the NLP community. However, commonly, many proposed models are found to perform well in the specific context in which they are evaluated and state-of-the-art claims are usually found not transportable to similar but different settings. The current evaluation metrics may only indicate which algorithm or setup performs best: they are unable to estimate performance in a new context, to demonstrate internal validity, or to verify causality. To offset this, statistical significance testing is sometimes applied in conjunction with performance measures (e.g. F1-score, BLEU) to attempt to establish validity. However, statistical significance testing has been shown to be lacking. Dror et al. (2018) reviewed NLP papers from ACL17 and TACL17 and found that only a third of these papers use significance * equal contribution testing. Further, many papers did not specify the type of test used, and some even employed an inappropriate statistical test.
Performance is measured in NLP tasks primarily through F1 score or task-specific metrics such as BLEU. The limited scope of these as performance evaluation techniques has been shown to have issues. Søgaard et al. (2014) highlights the data selection bias in NLP system performance. Gorman and Bedrick (2019) show issues of using standard splits, as opposed to random splits. We support their statement that "practitioners who wish to firmly establish that a new system is truly state-of-the-art augment their evaluations with Bonferroni-corrected random split hypothesis testing". In an NLI task, using SNLI and MultiNLI datasets with a set of different models, it has been shown that permutations of training data leads to substantial changes in performance (Schluter and Varab, 2018).
Further, the lack of transportability for NLP tasks has been raised by specialists in applied domains. For example, healthcare experts have expressed their frustration in the limitations of algorithms built in research settings for practical applications (Demner-Fushman and Elhadad, 2016) and the reduction of performance "outside of their development frame" (Maddox and Matheny, 2015). More generally, "machine learning researchers have noted current systems lack the ability to recognize or react to new circumstances they have not been specifically programmed or trained for" (Pearl, 2019).
The advantages of "more transportable" approaches, such as BERT, in terms of their performance in multiple different domains, is currently not expressed (other than the prevalence of such architectures across a range of state-of-the-art tasks and domains). To support analysis and investigation into the insight that could be gained by examination of these properties, we suggest metrics and a method for measuring the transportability of models to new domains. This has immediate relevance for domain experts, wishing to implement existing solutions on novel datasets, as well as for NLP researchers wishing to assemble new dataset, design new models, or evaluate approaches.
To support this, we propose feature gradient, and show it to have promise as a way to gain lexical or semantic insight into factors influencing the performance of different architectures in new domains. This differs from data complexity, being a comparative measure between two datasets. We aim to start a conversation about evaluation of systems in a broader setting, and to encourage the creation and utilisation of new datasets.
This paper focuses on the design and evaluation of a lightweight transportability measure in the context of the empirical evaluation of NLP models. A further aim is to provide a category of measures which can be used to estimate the stability of the performance of a system across different domains. An initial transportability measure is built by formalising properties of performance stability and variation under a statistical framework. The proposed model is evaluated in the context of Named Entity Recognition tasks (NER) and Natural Language Inference (NLI) tasks across different domains.
Our contribution is to present a measure that evaluates the transportability and robustness of an NLP model, to evaluate domain similarity measures to understand and anticipate the transportability of an NLP model, and to compare state of the art models across different datasets for NER and NLI.
2 Relevant background and related work

Terminology
To quote Campbell and Stanley (2015), "External validity asks the question of generalizability: To what populations, settings, treatment variables, and measurement variables can this effect be generalized?". For Pearl and Bareinboim (2014), transportability is how generalisable an experimentally identified causal effect is to a new population where only observational studies can be conducted. "However, there is an important difference, not often distinguished, between what might be called the potential (or generic) transferability of a study and its actual (or specific) transferability to another policy or practice decision context at another time and place." (Walker et al., 2010) Bareinboim and Pearl (2013)  They postulate that degradation differences were due to the differing prevalence of entities in the transported training data. Another term from this domain is "portability", in the sense that a model could be successfully used with consideration of implementation issues such as different data formats and target NLP vocabularies (Carroll et al., 2012). Blitzer et al. (2007) created a multi-domain dataset for sentiment analysis, and propose a measure of domain similarity for sentiment analysis based on the distance between the probability distributions in terms of characteristic functions of linear classifiers. In image processing, domain transfer is an active area of research.  propose transfer component analysis as a method to learn subspaces which have similar data properties and data distributions in different domains. They state that domain adaptation is "a special setting of transfer learning which aims at transferring shared knowledge across different but related tasks or domains". In computer vision, Peng et al. (2019) combine multiple datasets into a larger dataset DomainNet, and consider multi-source domain adaptation, formalising for binary classification. They demonstrate multisource training improves model accuracy, and publish baselines for state of the art methods.

Generalisability
The language used in literature is not consistent. Bareinboim and Pearl (2013) highlights that generalisability goes under different "rubrics" such as external validity, meta-analysis, overgeneralisation, quasi-experiments and heterogeneity. Boulenger et al. (2005) disambiguate terms in the context of healthcare economics (such as generalisability, external validity, and transferability), and created a self-reporting checklist to attempt to quantify transferability. They define generalisability as "the degree to which the results of a study hold true in other settings", and "the data, methods and results of a given study are transferable if (a) potential users can assess their applicability to their setting and (b) they are applicable to that setting". They advocate a user-centric view of transferability, considering specific usability aspects such as explicit currency conversion rates. Antonanzas et al. (2009) create a transferability index at general, specific and global levels. Their "general index" is comprised of "critical factors", which utilise Boulenger et al.'s factors, adding subjective dimensions.

Definitions
To support a rigorous discussion, notational conventions are introduced. Extending the choices of Pearl and Bareinboim (2011), we denote a domain D with population Π, governed by feature probability distribution P , which is data taken from a particular domain. We denote the source with a 0 subscript. Definition 1. Generalisability: A system Ψ has performance p for solving task T 0 in domain D 0 . Generalisability is how the system Ψ performs for solving task T i in domain D j , relative to the original task, without retraining.
Special cases, such as transportability or transference, have some i, j = 0 in the definition above.
Definition 2. Transportability: A system Ψ has performance p for solving task T 0 in domain D 0 . Transportability is the performance of system Ψ for solving task T 0 in a new domain D i , relative to the original task, without retraining.
Across multiple D i , we have relative performance τ p (D 0 , D i ), from which we can establish statistical measures for transportability performance and variation.  Transfer learning is a specific way of achieving transportability (between populations or domains) or generalisability (including between tasks). Singleton et al. (2014) state that "transport encompasses transfer learning in attempting to use statistical evidence from a source on a target, but differs by incorporating causal assumptions derived from a combination of empirical source data and outside domain knowledge.". Note that this is different to generalisation in the Machine Learning sense, which is akin to internal validity (Marsland, 2011). Figure 1 shows the definitions associated with transportability discussed in this paper. Table 1 summarises terminology, of how the target differs from source (Ψ 0 , T 0 , D 0 (Π 0 )).

Term
Ψ T D Π Cross-validation 0 0 0 i New modeling i 0 0 0 Transportability 0 0 i i Transferability 0 0,i i i Generalisability 0 0,i 0,i 0,i Chance, bias and confounding are the three broad categories of "threat to validity". Broadly, chance and bias can be assessed by cross-validity, as it applies a model to the same task in the same domain on different data population. Confounding, error in interpretation of what is being measured, is more difficult to assess. Transportability is concerned with the transfer of learned information, with particular advances in the transport of causal knowledge.
Generalisability is the catch-all term for how externally valid a result or model is. Any combination of task, domain and data can be used.

Transportability performance
We define transportability performance τ p as the gradient of the change in the performance metric's score from one domain to another. This measure does not take into account the underlying probability distributions, only the change in resulting performance measure.
The measure uses a ratio in order to allow comparison between different systems. To generalise this measure across different settings, we can take an average to give Equation 2. Note that this is the average percentage change in performance, not an aggregated performance measure.
An analogous definition holds for different tasks over the same domain, τ p (T ).

Performance variation
Performance variation reflects how stable performance is across different contexts and can include, for example, to what extent the sampling method from the source data effects the performance metric of the algorithm. Part of this is data representativeness, the extent to which the source data representation also represents the target data.
More formally, performance variation τ s (Ψ, T, D) is the change in performance of (Ψ, T, D) across different contexts. This is useful in order to gain specific insight into external validity and generalisability. Indeed, we can assess the change in performance between source context D 0 and target context D i . The source context has a privileged position, in that it is this space which the "learning" takes place, and the proposed metric for performance variation to multiple different domains is based on τ p to reflect this. Through repeated measurement in different contexts, we can go further.
Definition 3. Performance Variation: For a model trained on domain D 0 and applied on n new domains D i , we define the performance variation as the coefficient of variation of performance across this set of domains so that: The 1 + 1 4n term corrects for bias. In order to be meaningful, the target contexts must to have a good coverage of different domains. Enumerating these would be a task of ontological proportions, but can be pragmatically approximated by using the available Gold Standard datasets.
We can also assess ability to generalise not just over different domains, but also different tasks, provided they can be meaningfully assessed by the same performance measure. We can consider n different domain-task combinations, and with τ p = n i,j=0 τ p (Ψ, T i , D j )/n, this gives a more general form for Equation 3, with n large: In the case where different tasks cannot be assessed by the same measure, we are still able to compare different systems by looking at how the respective measures change.

Performance variation properties
For a purely random system, the transportability should be related to how similar the distributions of "answers" in the test dataset are. A random system should really be transportable by our measures. Similarly, we can consider trivial systems, such as identity and constant functions, which are necessarily entirely transportable. That is, for a system that is an identity function Ψ = I, τ p = f (P ), and τ var (I, T, D i ) = τ var (I, T, D j ) = 0, ∀i, j. Note that we would not expect the same performance of these functions on different tasks.
A stable system will have τ var (Ψ, T, D 0 ) ≈ τ var (Ψ, T, D i )∀i, reflecting that it is resilient to the domain on which it is trained.

Factors influencing performance variation
Through repeated measurement, we can quantify how F 1 -score changes with respect to different measures A (e.g. dataset complexity), ∂F 1 ∂A , with other properties held constant.
NLP system performance is dependent on A. This list may include gold standard feature distribution (in terms of representativeness of the semantic or linguistic phenomena), and task difficulty or sensitivity.
Users of NLP systems would benefit from being able to estimate the performance of an existing NLP system on a new domain, without performing the full implementation. Important for the performance of an NLP system, especially for few or zero shot learning, is having a common set of features (or phenomena) across domains. We proceed to propose three measures of increasing complexity, in order to attempt to understand how "similar" two domains are.
Lexical feature difference: A measure grounded on lexical features (i.e. bag of words). The intuition behind this measure is for treating the set of lexical features as a representation. Linguistic space is observed as materialised tokens, which in turn are in some higher-dimensional semantic space, which enable interpretation. The measure considers the overlap of these linguistic spaces, and indeed the extent to which the linguistic space is covered by the data. Due to the simplicity of this measure, correlation between this and actual transportability performance is likely to be weaker than other measures but is simpler to calculate.
Where |D i | is the number of features in the target domain D i , and |D i ∩ D 0 | is the number of features overlapping. This measure is then the proportion of unseen features in the new dataset. If all features of D i are found in D 0 , then the feature difference is 0. If no features of D i are found in D 0 , then the feature difference is 1. The feature overlap is task specific, and therefore appropriate to consider for transportability, but not generalisability.
In the simplest case, the transported performance of a bag of words model should be precisely the lexical feature difference combined with distributions of the source and target domains. The feature set can range from binary lexical features to latent vector spaces. For different models, which target different aspects of semantic phenomena, different semantic and syntactic features will matter more. For this reason, considering a set of measures for domain complexity is warranted. In the context of this work, two measures are used over more complex feature spaces. Cosine distance: Specifically, we use Doc2Vec (Le and Mikolov, 2014) to embed the documents from each domain in a 300-dimensional featurevector space, normalise, and calculate cosine distance to compare source and target domains.
Kullback-Leibler divergence: Considering each domain as a distribution of features, we can use relative entropy to understand the difference between the source and target domains. Similar to cosine distance, we convert the corpus to a vector using Doc2Vec and normalize. We treat these values as discrete probability distributions to calculate the KL divergence.
The usefulness of any of these domain similarity measures depends on the semantic phenomena and supporting corpora underlying the system, for example if the system requires a large training dataset, it may be more appropriate to use a measure which considers the underlying probability distributions in each feature. In this case, we can restrict to the case of the same task in order to keep the essential features reasonably consistent across domains. This makes this a measure of transportability (rather than generalisability).
There are additional dimensions of transportability potentially worthy of further investigation and quantification: (i) domain similarity (e.g. missing features), (ii) data efficiency (redundant/repeated features), (iii) data preparation (initial setup and formatting) and (iv) data manipulation required (data pipeline).

Setup
The experiments aim to evaluate the consistency of the proposed transportability measures in the context of two standard tasks: named entity recognition and natural language inference. For reproducibility purposes the code and supporting data are available online 1 .
We calculated the F1 score of multiple models on   (Figure 2). Note that in general the applicability of the proposed transportability measures are not limited to the use of F1 score, but this is simpler as the same measure applies for both tasks. All models and datasets are standard. For NER, the datasets were chosen as they have the consistent tags: Location, Person and Organisation. Stanford NER (Finkel et al., 2005) is a CRF classifier, SpaCy v2 is a CNN, ELMo (Peters et al., 2018) is a vector embedding model which outperforms GloVe and word2vec. Each of the three models used are trained on the CoNLL-2003 dataset (Sang and De Meulder, 2003). We evaluated these models on CoNLL-2003, Wikipedia NER (Ghaddar and Langlais, 2017) (Wiki) and WNUT datasets (Baldwin et al., 2015) for NER in twitter microposts.
For NLI, we chose to use standard datasets. SNLI (Bowman et al., 2015) is well established with a limited range of NLI statements, MultiNLI (Williams et al., 2018) is multigenre with a more diverse range of texts, and SciTail (Khot et al., 2018) is based on scientific exam questions. We applied BERT (Devlin et al., 2018), a state of the art embedding model, to these datasets.

Results
NER: Table 2 shows results for the NER task, trained on CoNLL. Unsurprisingly, all models performed better when the target was in the CoNLL domain. The reduced performance on Wiki was more extreme than expected, particularly for ELMo, which was expected to be resilient to domain change (i.e. transportable). Table 6 and Table 4 illustrate the transportability and domain similarity scores for different NER models respectively. NLI: Table 3 shows results for the NLI task, using BERT. We find that, despite the vast training data, BERT's performance is substantially higher when it has been trained on data from that domain. BERT trained on SciTail performs poorly when transported to SNLI or MultiNLI. Table 7 and Table 5 illustrates the transportability and domain similarity scores for different NLI corpora.

Analysis
Every model had τ p 1, meaning they performed worse on the new domain. This is as expected, though this would not be true in general.
NER: Examining the F1 scores (88.11 vs. 88.78) of SpaCy and Stanford they appear almost comparable. However, the latter transports much more effectively, with τ p score difference (0.671 Vs 0.524 when transporting to Wiki) (refer Table 6).
ELMo is one of the state of the art approaches for NER, as evidenced by the high F1 scores for the source corpus. However, Stanford NER transports equally well, and when transported outperforms ELMo for twitter domain. While the absolute F1 score difference between them is 5, the τ p scores are almost identical, with a difference of 0.003. In terms of transportability, it is notable that an approach that employs CRF tagger with linguistic features outperforms significantly the CNN-based SpaCy approach and stands in comparison to a computationally expensive model like ELMo.
Stanford NER also has the lowest τ var . This indicates this to be the most robust model out of the three. This conclusion was facilitated by the τ p and τ var measures.
NER for English is assumed to be an accomplished task as supported by the traditional F1 scores. By using τ p we argue that there is a need for more robust models, with better transportability performance. Figure 3a and Figure 3b illustrates the decrease in F1 scores as cosine distance and KL divergence increase. A simple 3 parameter non-linear regression model on KL Divergence and Cosine distance is able to predict the F1 score with an mean error of 3.33 and 2.66 respectively. Considering the lexical difference has similar results (Table 4). This implies that by using these measures we may be able to anticipate the accuracy of a model in a new domain based on easy to compute domain similarity, which is straightforward to compute.   The average τ p is 0.612 over transported domains, despite these being standard corpora from the domains. We found MultiNLI(Train) to be more transportable than the others, since its performance in new domains is not much worse than new data from the same domain. This is as expected, since MultiNLI has been built to have good domain coverage. Specifically, MultiNLI has τ p = 0.744 and τ var = 8.582, whilst SNLI has τ p = 0.646 and τ var = 15.22 and SciTail has τ p = 0.446 and τ var = 3.921. SciTail transports poorly, and does so reliably! SNLI transports in between, but variably, being quite "hit or miss" with different samples of SciTail. These results suggest a threshold for τ p of perhaps 0.8 as being "appropriate" for transportability performance. A threshold for τ var is more difficult to establish and would benefit from further investigation. Clearly, these measures depend on the domains chosen. As with NER, we found lexical difference indicative of transported performance, and that for NLI, accuracy scores decrease with increasing lexical difference, cosine distance and KL divergence (Tables 3 and 5, and Figures 4a and 4b). A simple 3 parameter non-linear regression model on KL Divergence and Cosine distance is able to predict the accuracy score with an mean error of 3.98 and 1.95 respectively.

Discussion
τ p and τ var as complementary to traditional measures. We are not breaking new ground in terms of evaluation methodology, but the experiments demonstrate that traditional F1 and accuracy measures do not capture a complete picture. Transportability measure are not only simple enough to calculate and convey but also evaluates a model