{'fraction_non_alphanumeric': 0.03836948116507714, 'fraction_numerical': 0.014250469492211953, 'mean_word_length': 4.729535864978903, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension

The creation of large-scale open domain reading comprehension data sets in recent years has enabled the development of end-to-end neural comprehension models with promising results. To use these models for domains with limited training data, one of the most effective approach is to first pretrain them on large out-of-domain source data and then fine-tune them with the limited target data. The caveat of this is that after fine-tuning the comprehension models tend to perform poorly in the source domain, a phenomenon known as catastrophic forgetting. In this paper, we explore methods that overcome catastrophic forgetting during fine-tuning without assuming access to data from the source domain. We introduce new auxiliary penalty terms and observe the best performance when a combination of auxiliary penalty terms is used to regularise the fine-tuning process for adapting comprehension models. To test our methods, we develop and release 6 narrow domain data sets that could potentially be used as reading comprehension benchmarks.


Introduction
Reading comprehension (RC) is the task of answering a question given a context passage. Related to Question-Answering (QA), RC is seen as a module in the full QA pipeline, where it assumes a related context passage has been extracted and the goal is to produce an answer based on the context. In recent years, the creation of large-scale open domain comprehension data sets (Yang, Yih, and Meek 2015;Nguyen et al. 2016;Rajpurkar et al. 2016;Joshi et al. 2017;Rajpurkar, Jia, and Liang 2018;Kocisky et al. 2018) has spurred the development of a host of end-to-end neural comprehension systems with promising results.
In spite of these successes, it is difficult to train these modern comprehension systems on narrow domain data (e.g. biomedical), as these models often have a large number of parameters. A better approach is to transfer knowledge via fine-tuning, i.e. by first pre-training the model using data from a large source domain and continue training it with examples from the small target domain. It is an effective strategy, although a fine-tuned model often performs poorly when it is re-applied to the source domain, a phenomenon known as catastrophic forgetting (French 1999; Copyright c 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Wiese, Weissenborn, and Neves 2017;Kirkpatrick et al. 2017;Riemer et al. 2018). This is generally not an issue if the goal is to optimise purely for the target domain, but in real-word applications where model robustness is an important quality, over-optimising for a development set often leads to unexpected poor performance when applied to test cases in the wild.
In this paper, we explore strategies to reduce forgetting for comprehension systems during domain adaption. Our goal is to preserve the source domain's performance as much as possible, while keeping target domain's performance optimal and assuming no access to the source data. We experiment with a number of auxiliary penalty terms to regularise the fine-tuning process for three modern RC models: QANet (Yu et al. 2018), decaNLP (McCann et al. 2018 and BERT (Devlin et al. 2018). We observe that combining different auxiliary penalty terms results in the best performance, outperforming benchmark methods that require source data.
Technically speaking, the methods we propose are not limited to domain transfer for reading comprehension. We also show that the methodology can be used for transferring to entirely different tasks. With that said, we focus on comprehension here because it is a practical problem in real world applications, where the target domain often has a small number of QA pairs and over-fitting occurs easily when we fine-tune based on a small development set. In this scenario, it is as important to develop a robust model as achieving optimal development performance.
To demonstrate the applicability of our approach, we apply topic modelling to MSMARCO (Nguyen et al. 2016)a comprehension data set based on internet search queries -and collect examples that belong to a number of salient topics, producing 6 small to medium sized RC data sets for the following domains: biomedical, computing, film, finance, law and music. We focus on extractive RC, where the answer is a continuous sub-span in the context passage. 1 Scripts to generate the data sets are available at: https://github.com/ibm-aur-nlp/domain-specific-QA.

Related Work
Most large comprehension data sets are open-domain because non-experts can be readily recruited via crowdsourcing platforms to collect annotations. Development of domain-specific RC data sets, on the other hand, is costly due to the need of subject matter experts and as such the size of these data sets is typically limited. Examples include BIOASQ (Tsatsaronis et al. 2015) in the biomedical domain, which has less than 3k QA pairs -orders of magnitude smaller compared to most large-scale open-domain data sets (Nguyen et al. 2016;Rajpurkar et al. 2016;Joshi et al. 2017;Kocisky et al. 2018). (Wiese, Weissenborn, and Neves 2017) explore supervised domain adaptation for reading comprehension, by pretraining their model first on large open-domain comprehension data and fine-tuning it further on biomedical data. This approach improves the biomedical domain's performance substantially compared to training the model from scratch. At the same time, its performance on source domain decreases dramatically due to catastrophic forgetting (French 1999;McCloskey and Cohen 1989;Riemer, Khabiri, and Goodwin 2017).
This issue of catastrophic forgetting is less of a problem when data from multiple domains or tasks are present during training. For example in (McCann et al. 2018), their model decaNLP is trained on 10 tasks simultaneouslyall casted as a QA problem -and forgetting is minimal. For multi-domain adaptation, (Daume III 2007) and (Kim, Stratos, and Sarikaya 2016) propose using a K+1 model to capture domain-general pattern that is shared by K domains, resulting in a more robust model. Using multi-task learning to tackle catastrophic forgetting is effective and generates robust models. The drawback, however, is that when training for each new domain/task, data from the previous domains/tasks has to be available.
Several studies present methods to reduce forgetting with limited or no access to previous data (Riemer et al. 2017;Lopez-Paz and others 2017;Kirkpatrick et al. 2017;Serra et al. 2018;Riemer et al. 2018). Inspired by synaptic consolidation, (Kirkpatrick et al. 2017) propose to selectively penalise parameter change during fine-tuning. Significant updates to parameters which are deemed important to the source task incur a large penalty. (Lopez-Paz and others 2017) introduce a gradient episodic memory (gem) to allow beneficial transfer of knowledge from previous tasks. More specifically, a subset of data from previous tasks are stored in an episodic memory, against which reference gradient vectors are calculated and the angles with the gradient vectors for the current task is constrained to be between −90 • and 90 • . (Riemer et al. 2018) suggest combining gem with optimisation based meta-learning to overcome forgetting. Among these three methods, only that of (Kirkpatrick et al. 2017) assumes zero access to previous data. In comparison, the latter two rely on access to a memory storing data from previous tasks, which is not always feasible in realworld applications (e.g. due to data privacy concerns).

Data Set
We use SQUAD v1.1 (Rajpurkar et al. 2016) as the source domain data for pre-training the comprehension model. It contains over 100K extractive (context, question, answer) triples with only answerable questions.
To create the target domain data, we leverage MSMARCO (Nguyen et al. 2016), a large RC data set where questions are sampled from Bing TM search queries and answers are manually generated by users based on passages in web documents. We apply LDA topic model (Blei, Ng, and Jordan 2003) to passages in MSMARCO and learn 100 topics. 2 Given the topics, we label them and select 6 salient domains: biomedical (MS-BM), computing (MS-CP), film (MS-FM), finance (MS-FN), law (MS-LW) and music (MS-MS). A QA pair is categorised into one of these domains if its passage's top-topic belongs to them. We create multiple (context, question, answer) training examples if a QA pair has multiple contexts, 3 and filter them to keep only extractive examples. 4 In addition to the MSMARCO data sets, we also experiment with a real biomedical comprehension data set: BIOASQ (Tsatsaronis et al. 2012). Each question in BIOASQ is associated with a set of snippets as context, and the snippets are single sentences extracted from a scientific publication's abstract/title in PubMed Central TM . There are four types of questions: factoid, list, yes/no, and summary. As our focus is on extractive RC, we use only the extractive factoid questions from BIOASQ. As before, we create multiple 2 When collecting the passages, we include only those being selected as useful for answering a query (i.e. is selected = 1). We tokenise the passages with Stanford CoreNLP (Manning et al. 2014) and use MALLET (McCallum 2002) for topic modelling. 3 We only consider context passages that are marked as being useful by annotators in the original data (i.e. is selected = 1).
training examples for QA pairs with multiple contexts.
For each target domain, we split the examples into 70%/15%/15% training/development/test partitions. 5 We present some statistics for the data sets in Table 1.

Methodology
We first pre-train a general domain RC model on SQUAD, our source domain. Given the pre-trained model, we then perform fine-tuning (finetune) on the MSMARCO and BIOASQ data sets: 7 target domains in total. By fine-tuning we mean taking the pre-trained model parameters as initial parameters and update them accordingly based on data from the new domain. To reduce forgetting on the source domain (SQUAD), we experiment with incorporating auxiliary penalty terms (e.g. L2 between new and old parameters) to the standard cross entropy loss to regularise the fine-tuning process.
We explore 3 modern RC models in our experiments: QANet The objective of the model is to predict the position of the starting and ending indices of the answer words in the context. decaNLP is a recurrent networkbased comprehension model trained on ten NLP tasks simultaneously, all casted as a question-answer problem. Much of decaNLP's flexibility is due to its pointer-generator network, which allows it to generate words by extracting them from the question or context passages, or by drawing them from a vocabulary. BERT is a deep bi-directional encoder model based on Transformers. It is pre-trained on a large corpus in an unsupervised fashion using a masked language model and next-sentence prediction objective. To apply BERT to a specific task, the standard practice is to add additional output layers on top of the pre-trained BERT and fine-tune the whole model for the task. In our case for RC, 2 output layers are added: one for predicting the start index and another the end index. (Devlin et al. 2018) demonstrates that this transfer learning strategy produces state-ofthe-art performance on a range of NLP tasks. For RC specifically, BERT (BERT-Large) achieved an F1 score of 93.2 on SQUAD, outperforming human performance by 2 points.
Note that BERT and QANet RC models are extractive models (goal is to predict 2 indices), while decaNLP is a generative model (goal is to generate the correct word sequence). Also, unlike QANet and decaNLP, BERT is not designed specifically for RC. It represents a growing trend in the literature where large models are pre-trained on big corpora and further adapted to downstream tasks.
To reduce the forgetting of source domain knowledge, we introduce auxiliary penalty terms to regularise the finetuning process. We favour this approach as it does not require storing data samples from the source domain. In general, there are two types of penalty: selective and nonselective. The former penalises the model when certain pa-rameters diverge significantly from the source model, while the latter uses a pre-defined distance function to measure the change of all parameters.
For selective penalty, we use elastic weight consolidation (EWC: (Kirkpatrick et al. 2017)), which weighs the importance of a parameter based on its gradient when training the source model. For non-selective penalty, we explore L2 (Wiese, Weissenborn, and Neves 2017) and cosine distance. We detail the methods below.
Given a source and target domain, we pre-train the model first on the source domain and fine-tune it further on the target domain. We denote the optimised parameters of the source model as θ * and that of the target model as θ. For vanilla fine-tuning (finetune), the loss function is: where L ce is the cross-entropy loss.
For non-selective penalty, we measure the change of parameters based on a distance function (treating all parameters as equally important), and add it as a loss term in addition to the cross-entropy loss. One distance function we test is the L2 distance: where λ l2 is a scaling hyper-parameter to weigh the contribution of the penalty. Henceforth all scaling hyperparameters are denoted using λ.
We also experiment with cosine distance, based on the idea that we want to encourage the parameters to be in the same direction after fine-tuning. In this case, we group parameters by the variables they are defined in, and measure the cosine distance between variables: where θ v denotes the vector of parameters belonging to variable v.
For selective penalty, EWC uses the Fisher matrix F to measure the importance of parameter i in the source domain. Unlike non-selective penalty where all parameters are considered equally important, EWC provides a mechanism to weigh the update of individual parameters: where ∂Lce(f θ * ,(x,y)) ∂θ * is the gradient of parameter update in the source domain, with f θ * representing the model and x/y the data/label from the source domain.
In preliminary experiments, we notice that EWC tends to assign most of the weights to a small subset of parameters. We present Figure 1a, a plot of mean Fisher values for all variables in QANet after it was trained on SQUAD, the source domain. We see that only the last two variables have some significant weights (and a tiny amount for the rest of the variables). We therefore propose a new variation of EWC, normalised EWC, by normalising the weights within each variable via min-max normalisation, which brings up the weights for parameters in other variables (Figure 1b): where {F } vi denotes the set of parameters for variable v where parameter i belongs. Among the four auxiliary penalty terms, L2 and EWC are proposed in previous work while cosine distance and normalised EWC are novel penalty terms. Observing that EWC and normalised EWC are essentially weighted l1 distances and L2 is based on l2 distance while cosine distance focuses on the angle between variables (and ignores the magnitude), we propose combining them altogether as these different distance metrics may complement each other in regularising the fine-tuning process:

Experiments
We test 3 comprehension models: QANet, decaNLP and BERT. To pre-process the data, we use the the models' original tokenisation methods. 6 For BERT, we use the smaller pre-trained model with 110M parameters (BERT-Base).

Fine-Tuning with Auxiliary Penalty
We first pre-train QANet and decaNLP on SQUAD, tuning their hyper-parameters based on its development partition. 7 For BERT, we fine-tune the released pre-trained model on SQUAD by adding 2 additional output layers to predict the start/end indices (we made no changes to the hyper-parameters). We initialise word vectors of QANet and decaNLP with pre-trained GloVe embeddings (Pennington, Socher, and Manning 2014) and keep them fixed during training. We also freeze the input embeddings for BERT. 8 To measure performance, we use the standard macro-averaged F1 as the evaluation metric, which measures the average overlap of word tokens between prediction and ground truth answer. 9 Our pre-trained QANet, decaNLP and BERT achieve an F1 score of 80.47, 75.50 and 87.62 respectively on the development partition of SQUAD. Note that the test partition of SQUAD is not released publicly, and so all reported SQUAD performance in the paper is on the development set. Given the pre-trained SQUAD models, we fine-tune them on the MSMARCO and BIOASQ domains. We test vanilla finetuning (finetune) and 5 variants of fine-tuning with auxiliary penalty terms: (1) EWC (+ewc); normalised EWC (+ewcn); cosine distance (+cd); L2 (+l2); and combined normalised EWC, cosine distance and L2 (+all). As a benchmark, we also perform fine-tuning with gradient episodic memory (gem), noting that this approach uses the first m examples from SQUAD (m = 256 in our experiments).
To find the best hyper-parameter configuration, we tune it based on the development partition for each target domain. For a given domain, finetune and its variants (+ewc, +ewcn, +cd, +l2 and +all) all share the same hyper-parameter configuration. 10 Detailed hyper-parameter settings are given in the supplementary material.
As a baseline, we train QANet, decaNLP and BERT from scratch (scratch) using the target domain data. As before, we tune their hyper-parameters based on development performance. We present the full results in Table 2.
For each target domain, we display two F1 scores: the source SQUAD development performance ("SQUAD"); and the target domain's test performance ("Test"). We first compare the performance between scratch and finetune. Across all domains for QANet, decaNLP and BERT, finetune substantially improves the target domain's performance compared to scratch. The largest improvement is seen in BIOASQ for QANet, where its F1 improves two-fold (from 29.83 to 65.81). Among the three RC models, BERT has the best performance for both scratch and finetune in most target domains (with a few exceptions such as MS-FN and MS-LW). Between QANet and decaNLP, we see that decaNLP tends to have better scratch performance but the pattern is reversed in finetune, where QANet produces higher F1 than decaNLP in all domains except for MS-LW.
In terms of SQUAD performance, we see that finetune  We now turn to the fine-tuning results with auxiliary penalties (+ewc, +ewcn, +cd and +l2). Between +ewc and +ewcn, the normalised versions consistently produces better recovery for the source domain (one exception is MS-MS for decaNLP), demonstrating that normalisation helps. Between +ewcn, +cd and +l2, performance among the three models vary depending on the domain and there's no clear winner. Combining all of these losses (+all) however, produces the best SQUAD performance for all models across most domains. The average recovery (+allfinetune) of SQUAD performance is 4.54, 3.93 and 8.77 F1 points for QANet, decaNLP and BERT respectively, implying that BERT benefits from these auxiliary penalties more than decaNLP and QANet.
When compared to gem, +all preserves SQUAD performance substantially better, on average 2.86 points more for QANet and 5.57 points more BERT. For decaNLP, the improvement is minute (0.02); generally gem has the upper hand for most domains but the advantage is cancelled out by its poor performance in one domain (MS-FN). As gem requires storing training data from the source domain (SQUAD training examples in this case), the auxiliary penalty tech- niques are more favourable for real world applications. Does adding these penalty terms harm target performance? Looking at the "Test" performance between finetune and +all, we see that they are generally comparable. We found that the average performance difference (+all-finetune) is 0.23, −0.42 and 0.34 for QANet, decaNLP and BERT respectively, implying that it does not (in fact, it has a small positive net impact for QANet and BERT). In some cases it improves target performance substantially, e.g. in BIOASQ for BERT, the target performance is improved from 71.62 to 76.93, when +all is applied.
Based on these observations, we see benefits for incorporating these penalties when adapting comprehension models, as it produces a more robust model that preserves its source performance (to a certain extent) without trading off its target performance. In some cases, it can even improve the target performance.

Continuous Learning
In previous experiments, we fine-tune a pre-trained model to each domain independently. With continuous learning, we seek to investigate the performance of finetune and its four variants (+l2, +cd, +ewcn and +all) when they are applied to a series of fine-tuning on multiple domains. For the remainder of experiments in the paper, we test only with decaNLP.
When computing the penalties, we consider the last trained model as the source model. 11 Figure 2 demonstrates the performance of the models on the development set of SQUAD and test sets of MS-BM and MS-CP when they are adapted to MS-BM, MS-CP, MS-FN, MS-MS, MS-FM and MS-LW in sequence. 12 We exclude plots for the latter domains as they are similar to that of MS-CP. Including the pre-training on SQUAD, all models are trained for a total of 170K iterations: SQUAD from 0-44K, MS-BM from 45K-65K, MS-CP from 66K-86K, MS  We first look at the recovery for SQUAD in Figure 2a. +all (black line; legend in Figure 2c) trails well above all other models after a series of fine-tuning, followed by +ewcn and +cd, while finetune produces the most forgetting. At the end of the continuous learning, +all recovers more than 5 F1 points compared to finetune. We see a similar trend for MS-BM (Figure 2b), although the difference is less pronounced. The largest gap between finetune and +all occurs when we fine-tune for MS-FM (iteration 129K-149K). Note that we are not trading off target performance when we first tune for MS-BM (iteration 45K-65K), where finetune and +all produces comparable F1.

Task Transfer
In decaNLP, curriculum learning was used to train models for different NLP tasks. More specifically, decaNLP was first pre-trained on SQUAD and then fine-tuned on 10 tasks (including SQUAD) jointly. During the training process, each minibatch consists of examples from a particular task, and they are sampled in an alternating fashion among different tasks. In situations where we do not have access to training data from previous tasks, catastrophic forgetting occurs when we adapt the model for a new task. In this section, we test our methods for task transfer (as opposed to domain transfer in previous sections). To this end, we experiment with decaNLP and monitor its SQUAD performance when we fine-tune it for other tasks, including semantic role labelling (SRL), summarisation (SUM), semantic parsing (SP), machine translation (MT), and sentiment analysis (SA). Note that we are not doing joint or continuous learning here: we are taking the pre-trained model (on SQUAD) and adapting it to the new tasks independently. Description of these tasks are detailed in (McCann et al. 2018).
A core novelty of decaNLP is that its design allows it to generate words by extracting them from the question, context or its vocabulary, and this decision is made by the pointer-generator network. Based on the pointer-generator analysis in (McCann et al. 2018), we know that the pointergenerator network favours generating words using: (1) context for SRL, SUM, and SP; (2) question for SA; and (3) vocabulary for MT.
As before, finetune serves as our baseline, and we have 5 variants with auxiliary penalty terms. Table 3 displays the F1 performance on SQUAD and the target task; the table shares the same format as Table 2.
In terms of target task performance ("Test"), we see similar performances for all models. This is a similar observation we saw in previously, and it shows that the incorporation of the auxiliary penalty terms does not harm target task or domain performance.
For the source task SQUAD, +all produces substantial recovery for SUM, SRL, SP and SA, but not for MT. We hy-pothesise that this is due to the difference in nature between the target task and the source task: i.e. for SUM, SRL and SP, the output is generated by selecting words from context, which is similar to SQUAD; MT, on the other hand, generate using words from the vocabulary and question, and so it is likely to be difficult to find an optimal model that performs well for both tasks.

Discussion
Observing that the model tends to focus on optimising for the target domain/task in early iterations (as the penalty term has a very small value), we explore using a dynamic λ scale that starts at a larger value that decays over time. With just simple linear decay, we found substantial improvement in +ewc for recovering SQUAD's performance, although the results are mixed for other penalties (particularly for +ewcn). We therefore only report results that are based on static λ values in this paper. With that said, we contend that this might be an interesting avenue for further research, e.g. by exploring more complex decay functions.
To validate the assumption made by gem (Lopez-Paz and others 2017), we conduct gradient analysis for the auxiliary penalty terms. During fine-tuning, at each step t, we calculate the gradient cosine similarity sim(g t , g t ), where g t = ∂L(f θ t ,M ) ∂θt , g t = ∂L(f θ t ,(x,y)) ∂θt , M is a memory containing SQUAD examples, and x/y is training data/label from the current domain. We smooth the scores by averaging over every 1K steps, resulting in 20 cosine similarity values for 20K steps. Figure 3 plots the gradient cosine similarity for our models in MS-FN. Curiously, our best performing model +all produces the lowest cosine similarity at most steps (the only exception is between 0-1K steps). finetune, on the other hand, maintains relatively high similarity throughout. Similar trends are found for other domains. These observations imply that the inspiration gem draw on -i.e. catastrophic forgetting can be reduced by constraining a positive dot product between g t and g t -is perhaps not as empirically effective as intuition might tell us, and that our auxiliary penalty methods represent an alternative (and very different) direction to preserving source performance.

Conclusion
To reduce catastrophic forgetting when adapting comprehension models, we explore several auxiliary penalty terms to regularise the fine-tuning process. We experiment with selective and non-selective penalties, and found that a combination of them consistently produces the best recovery for the source domain without harming its performance in the target domain. We also found similar observations when we apply our approach for adaptation to other tasks, demonstrating its general applicability. To test our approach, we develop and release six narrow domain reading comprehension data sets for the research community.