{'fraction_non_alphanumeric': 0.04749166088708346, 'fraction_numerical': 0.012814161630172727, 'mean_word_length': 4.31694468832309, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Ask Question First for Enhancing Lifelong Language Learning

Lifelong language learning aims to stream learning NLP tasks while retaining knowledge of previous tasks. Previous works based on the language model and following data-free constraint approaches have explored formatting all data as “begin token (B) + context (C) + question (Q) + answer (A)” for different tasks. However, they still suffer from catastrophic forgetting and are exacerbated when the previous task’s pseudo data is insufficient for the following reasons: (1) The model has difficulty generating task-corresponding pseudo data, and (2) A is prone to error when A and C are separated by Q because the information of the C is diminished before generating A. Therefore, we propose the Ask Question First and Replay Question (AQF-RQ), including a novel data format “BQCA” and a new training task to train pseudo questions of previous tasks. Experimental results demonstrate that AQF-RQ makes it easier for the model to generate more pseudo data that match corresponding tasks, and is more robust to both sufficient and insufficient pseudo-data when the task boundary is both clear and unclear. AQF-RQ can achieve only 0.36% lower performance than multi-task learning.


Introduction
Lifelong learning is the capacity of human beings to acquire, reconstruct, strengthen, and transfer knowledge (Ring, 1997). Human beings can learn new knowledge while consolidating old knowledge by first detecting and learning the distinctions between old and new knowledge and then simplifying the old and new knowledge based on leveraging the common points. This concept is critical for encouraging machines to learn NLP tasks in a similar way that people do. In the application of NLP, new data are continuously acquired and categorized as either new data for existing tasks or new data for new tasks. For new data of existing tasks, the traditional method, known as isolated learning (Chen and Liu, 2018), is to retrain the model with the old data appended with new data. For data from new tasks, multi-task learning (MTL) integrates data from previous and new tasks to retrain the model. Both of these methods are limited to assuming that all new and old data can be obtained during training. However, the reality is that tasks are acquired and trained in the stream, which makes the model suffer from catastrophic forgetting (Ring, 1997;McCloskey and Cohen, 1989;French, 1999) (i.e. forgetting previously learned tasks/knowledge). Lifelong language learning (LLL), which this paper focuses on, aims to learn a stream of NLP tasks with lifelong learning. LAMOL (Sun et al., 2019) has recently proposed implementing a language model for LLL by formatting all data as QA-style and generating pseudo data instead of real data for previous tasks in order to prevent catastrophic forgetting. Many works (Chuang et al., 2020;Sun et al., 2020;Kanwatchara et al., 2021) have investigated how to improve it through methods that require more computing resources or additional parameters (e.g. knowledge distillation or adding sub-networks). In this paper, we refer to these works as LAMOL-based methods. In LAMOLbased methods, each example is consist of four segments: task-specific or task-independent token [TASK]/[GEN] (B), context (C), question (Q), and answer (A). Then each example is formatted as "B+C+Q+A" ("BCQA"). LAMOL has a gap between MTL that is regarded as the upper bound of LLL. Other LAMOL-based methods require more computing resources or additional parameters but still have a gap between MTL especially pseudo samples is unsufficient. Therefore, the "BCQA" format is useful but not train-efficient enough for reasons below: (1) generating task-corresponding pesudo data is hard because the similar C can be found in distinct tasks; (2) the C's information is diminished when A is generated due to the fact that A and C are separated by Q.
To generate more task-corresponding pseudo data for previous tasks and tighten the relationship between the C and the A, we proposed the Ask Question First (AQF) and Replay Question (RQ).
The AQF formats all data into a novel format: "B+Q+C+A" ("BQCA"), which is consistent with human reading comprehension behavior. Questions are more than just questions; they direct our learning. As is shown in Fig.1, when people do reading comprehension, they usually read the questions first, then read the articles with the questions, and pay attention to what can answer the questions while reading. This is an efficient reading and learning method, which known as metacognition (Flavell, 1979), has been researched in the field of education and psychology. There are two benefits to applying the "BQCA" format: (1) It's easier to generate pseudo-data that matches the corresponding task because the Q has different but limited types for each task.
(2) The model is more stable when there isn't enough pseudo data because the A is right next to the C. Furthermore, the model can pay more attention to important information in the C with the help of the Q.
The RQ introduce a novel training task to help model generate more task-corresponding pseudo data. In the "BQCA" format, generating the correct Q is crucial for generating task-corresponding pseudo-data. However, the Q of the previous may be covered by the new task since we cannot predict the number and type of Q of the new task. In order to strengthen the generation of the Q of the previous task, we generate pseudo-problems of the previous task to train the model. The contributions of our paper are listed below: (1) We proposed the Ask Question First and Re-play Question (AQF-RQ 1 ) to alleviate catastrophic forgetting when the pseudo data is sufficient and insufficient without additional computation resources and parameters. (2) We proposed a novel data format "BQCA" to make data train-efficient and generating corresponding pseudo data easier. (3) We proposed a novel training task to help the model generate correct questions and then generate more task-corresponding pseudo data.

Related Work
Lifelong language learning (LLL) is an essential step in promoting the realization of general artificial intelligence in the field of NLP. (Liu et al., 2019;Mi et al., 2020;Huang et al., 2021) have studied LLL on a single type of NLP task by regularization or replaying real data. Recently, LAMOL (Sun et al., 2019) uses a language model (LM) to learn various kinds of NLP tasks in QA-style. In LAMOL, the pseudo-data generated by the model is trained together with the new task to alleviate catastrophic forgetting. Many works explored the enhancement of LAMOL with additional computation resources or adding sub-networks. L2KD (Chuang et al., 2020), DnR (Sun et al., 2020) and DFSD (Wang et al., 2022) distilled parts or all layers of the model to improve LAMOL. ARPER (Mi et al., 2020) applied regualarization on parameters with prioritized exemplar replay. Rational-LAMOL (Kanwatchara et al., 2021) applied critical freezing guided by rationale information which is obtained by human or unsupervised rationale generation (Chang et al., 2020). (Madotto et al., 2021) applied Adapter (Houlsby et al., 2019) to plug into pretrain language model. Those are effective but require more computational resources (e.g., distillation and regularization) or new parameters (such as adding sub-networks for new tasks). We aspire to improve without increasing our resources. Consequently, based on the metacognition (Flavell, 1979;Bowler, 2010;Braithwaite and Sprague, 2021) proposed in education and psychology, we propose a more training-efficient and robust data format: first ask questions, then observe the context, finally answer the questions. In addition, we introduce replay questions to strengthen the model's attention to questions.  Figure 2: Illustration of QA/LM/RQ task with the "BQCA" format.

Methodology
In this section, we first introduce LAMOL in Section 3.1. Then, our proposed Ask Question First (AQF) is detailed in Section 3.2. Thirdly, we introduce the Replay Questions in Section 3.3. Finally, we summarize the training objectives in our paper in Section 3.4.

LAMOL
LAMOL (Sun et al., 2019) proposed using a single language model for lifelong language learning. A stream of various NLP tasks is learned by GPT-2 (Radford et al., 2019) with joint training of the language model (LM) task and the questionanswering (QA) task. In LAMOL, all tasks are formatted QA-style. Each example can consist of four segments: task-specific or task-independent token [TASK]/[GEN] (B), context (C), question (Q), and answer (A). In this way, the LM task is to generate "C+Q+A" by inputting [TASK]/[GEN], and the QA task is to generate "A" by inputting "C+Q". The loss of LAMOL is calculated as below: where λ is the weight of the LM task.
With the help of the LM task, the [TASK]/[GEN] can be used as the first token to input the model and generate the pseudo sample by greedy decoding. From the second task on, pseudo samples of previous tasks are generated in this way and jointly trained with the new task dataset to alleviate catastrophic forgetting. Let denote the set of pseudo samples as P i = {ps j |i, j ∈ N + , 1 ≤ j < i}, where ps j represents the pseudo data of j-th task.
The number of pseudo samples of previous tasks is determined by the size of the new task and a hyperparameter sampling ratio γ ∈ (0, 1]. Assuming that T = {T 1 , T 2 , . . . , T N } represents N tasks to be learned. Let |D t |, t ∈ [2, N ] denote the size of t-th task. LAMOL generates |ps i | = γ t−1 |D t | pseudosamples for each previously learned task. Then, D t is joined with P t to train the model. However, the performance of LAMOL still has a gap between Case ID Content Cor. Table 1: The cases of the pseudo sample after the model learning two tasks in turn with "BCQA" format. Cor. is the abbreviation of "corresponding". multi-task learning (MTL), which is regarded as the upper bound of lifelong learning, since not each pseudo sample is tied to [TASK]/ [GEN]. In this paper, we propose Ask Question First (AQF) and replaying questions to make data train-efficient and shorten the gap between MTL.

Ask Question First
Ask Question First (AQF) is a novel data format that makes the model imitate the process of human reading comprehension to learn knowledge from the training set efficiently. LAMOL proposed a data format, "B+C+Q+A" ("BCQA"), which is useful but not train-efficient enough. samples not corresponding to their tasks when more tasks are learned. These non-corresponding pseudo-samples jointly trained with the new task will aggravate the catastrophic forgetting of the model. Otherwise, the C's information is diminished when A is generated due to the fact that A and C are separated by Q.
To generate more corresponding pseudo samples, we propose AQF format all data as "B+Q+C+A" ("BQCA") which is simple yet efficient to make the model learn knowledge from the training set efficiently. We choose the task-specific token to analyze why our proposed "BQCA" is better than the naive "BCQA". Assuming that the model has learned two tasks T 1 and T 2 , pseudo samples of T 1 and T 2 needed to be generated before training the new task. Two types of task-specific tokens B 1 and B 2 are inputted to the model to generate pseudo samples with greedy decoding. There are four main kinds of pseudo samples that can be obtained as shown in Table 1. Case 1 and Case 3 are nice pseudo samples that can alleviate catastrophic forgetting, but Case 2 and Case 4 are terrible ones that can aggravate catastrophic forgetting. Ideally, pseudo samples, which we denote as taskcorresponding pseudo-samples P c , like Case 1 and Case 3 are what we expect to generate. However, in fact, due to catastrophic forgetting, Case 2 and Case 4 also appear. These pseudo samples like case 2 and case 4 are denoted as the not task-corresponding samples P nc . The probability of generating P nc is positively correlated with the severity of catastrophic forgetting. Therefore, the model will form a vicious circle: forgetting old tasks and generating P nc with greater probability. For pseudo samples start with task-specific token, the ps i consists of corresponding pseudo data ps c i and not corresponding pseudo data ps nc i (i.e. ps i = {ps c i , ps nc i }). In LAMOL-based methods, they used GPT-2 (Radford et al., 2019) for experiments. GPT-2 is an autoregressive model that can only obtain information before the current position, and then predict the words. Therefore, the probability of our proposed "BQCA" format and naive "BCQA" format can be calculated by Eq.(2) and Eq.(3), respectively: (2) = P (A|BQC)P (C|BQ)P (Q|B)P (B) P naive = P (BCQA) (3) = P (A|BCQ)P (Q|BC)P (C|B)P (B) As shown in Eq.(2) and Eq. (3), P (B) is a constant because it is given as the first token. Next, P (Q|B) and P (C|B) affect the quality of the generated pseudo data. To obtain P c , the Q or C should be generated to be corresponding to the given B, then the remaining generation has a greater probability correspond to B.
Then we first consider P (Q|B) and P (C|B). Let Θ N denote the parameters of the model after learning N tasks in turn. Assuming two tasks have been learned on the condition of sufficient pseudo samples (i.e. large γ) in turn, P (Q|B) and P (C|B) can be written as, P (Q|B, Θ 2 ) and P (C|B, Θ 2 ). In the dataset of a task, there are thousands of kinds of C, but there are only limited kinds of Q. For a simplest classification task, only one fixed Q is needed. For example, in the SST (Radford et al., 2017), all Q is "is review is positive or negative?". For complex tasks, such as SRL (He et al., 2017), it is mainly based on limited types of Q such as what, where, why, when, how etc, but it is still smaller than that of C. Therefore, P (Q|B, Θ 2 ) and P (C|B, Θ 2 ) are inversely proportional to the vocabulary sizes of Q and C, } denote the vocabulary of Q and C, respectively. w Q i and w C i denotes the word in the vocabulary of Q and C, respectively. According to the above analysis, it can be obtained that |V Q | ≪ |V C |. For most words, since their probability is inversely proportional to the size of the vocabulary, we can conclude that P (w Q ) ≫ P (w C ). Slight noise perturbations, such as shifts from the data distribution of the new task, can make P (w C ) more susceptible than P (w Q ). On the other hand, different tasks may have similar C due to a large number of similar common phrases, but the Q is almost different in what the questions pay attention to. Therefore, we can conclude that: where the subscript B 1 or B 2 means the probability on the condition of B 1 or B 2 . Then, we can conclude that: It can be concluded that generating Q first is more stable than generating C. On the condition that the Q or C is corresponding to the B, we analysis the P (C|BQ, Θ 2 ) and P (Q|BC, Θ 2 ). With the help of GPT-2, a Transformer decoder-based model, the current token can pay more attention to the information before the current position. When the previous adjacent content contains enough information, the generated token strongly correlates with the previous content. Therefore, Finally, P (A|BQC) and the P (A|BCQ) are trained with the QA task. It can be ignored from the perspective of generating task-corresponding pseudo-samples. However, correct "A" is easier to obtain on the condition of correct "BQC" than correct "BCQ" because the "C" is right next to "A" in "BQC". In "BQC", "Q" can help the model pay more attention to the important information in "C" and then have a larger probability of generating the correct "A".
From the above analysis, we can conclude that P (Q|B) and P (C|B) play main role on generating task-corresponding pseudo-samples P c according to P AQF and P naive , respectively. As shown in Eq.(6) and Eq. (7), P (Q|B) are better than P (C|B) for generating P c after sufficient pseudo samples being joint trained.

Replay Questions
Replay Questions (RQ) is proposed to strengthen the probability of generating task-corresponding Q when given B. Based on Eq.(4), P B 1 (w T 1 Q ) greater than or equal to P B 1 (w T 2 Q ) on the condition of sufficient pseudo samples being joint trained. However, the situation P B 1 (w T 2 Q ) > P B 1 (w T 1 Q ) may happen when the amount of pseudo samples is much less than that of the new task (i.e. |D 2 | ≫ γ|D 2 |). Since P (Q|B) plays a main role in P AQF , we propose to replay questions to make P (Q i |B i ) larger than P (Q j̸ =i |B i ).
As shown in Fig.2, RQ is to generate the questions of pseudo samples that start with [TASK]/[GEN] but have no end token. Since the object of RQ is to make the model generate taskcorresponding pseudo-samples, RQ should not supply the information about the end of the sentence.

Training
In summary, our proposed AQF is applied to both the LM task and the QA task. Otherwise, we expand the training objective with our proposed RQ. The final training objective is shown as Eq. (8).
where λ and η denotes the weight of the LM task and the RQ task.

Datasets
To be comparable with previous works, we chose four different kinds of tasks from DecaNLP (Mc-Cann et al., 2018) and five sequence-generation tasks (Chuang et al., 2020) from different domains. Details are summarized in the Table 7 in Appendix A. The SQuAD is a question-answering dataset with 12 main types of questions and other countless types of questions. SST is a sentiment analysis dataset with one question. SRL is a semantic role labeling dataset with 6 types of questions.

Implementation Detail
For fairness comparation. we implement experiments following LAMOL 2 . GPT-2 with 12 layers is selected as the language model. All experiments are run on a single Tesla P100 (12GB). Each task is trained for 9 epochs. The pseudo data is generated by greedy deocde. The other hyperparameter settings are the same as for LAMOL and are detailed in the Table 6 in the Appendix B.

Experiments
In this section, we experiment our proposed Ask Question First and Replay Question (AQF-RQ) an on three settings: (1) single task learning in Section 5.1; (2) different types of tasks in Section 5.2; (3) the same types of tasks in different domains in Section 5.3. Finally, we explore the effectiveness of our proposed AQF in Section 5.4.

Single Task
To validate our proposed AQF that format all task into new data format "BQCA", we experiment on each dataset independently. Each task is only trained by the QA task. As shown in Table 2, our proposed "BQCA" format beats "BCQA" format which is applied in previous LAMOL-based methods on each task. It exhibits that our "BQCA" format has the ability to increase the performance of a variety of different kinds of tasks, particularly those that were originally question-answering types (e.g. SQuAD, SRL). The SQuAD is improved significantly with the "BQCA" format scoring 8.8 percentage points higher than the "BCQA" format. For SRL, 3.5 percentage points of improvement come from the "BQCA" format compared with the "BCQA" format. For SST and WOZ, the "BQCA" format can improve by 1.3 and 1.8 percentage points respectively. For five generation tasks, "BQCA" format is slightly better than "BCQA". There are two reasons why our proposed "BQCA" format can improve the aforementioned tasks to a certain extent: (1) Due to the characteristics of autoregression and the mask attention mechanism of the Transformers Decoder, first Q and then C can enable the model to obtain more accurate attention information from C based on Q.
(2) The A is right next to the C, which reduces the information loss caused by the excessive length of the historical text in comparison to "BCQA" where the A and the C are gapped by the Q.

Different Types of Tasks
For a fair comparison, we conduct experiments on three different tasks in DecaNLP following LAMOL (Sun et al., 2019). To observe the performance in the case of sufficient and insufficient pseudo data, we select the sampling ratio γ = 0.2 and γ = 0.05 for experiments. Meanwhile, we conduct experiments on the condition of task-specific token [TASK] (denoted by the subscript T ) and task-independent token [GEN] (denoted by the subscript G) to verify if our proposed AQF-RQ is robust when the task boundary is clear or unclear.

Performance
As shown in Table 3, Finetune, which is a baseline for other methods, suffers serious catastrophic forgetting and has large gap bewteen multitask learning. Let's first observe the situation where the pseudo data is sufficient (i.e. γ = 0.2). LAMOL 0.2 G and LAMOL 0.2 T have good performance and are similar, indicating that in this case, the "BCQA" format is robust to clear and unclear task boundaries, but still 3 percentage points lower than MTL. However, AQF-RQ does better than LAMOL by 2.7 percentage points when the boundary is clear and by 2.3 percentage points when the boundary is not clear. AQF-RQ is only 0.3-0.6% worse than MTL. It is worth noting that AQF-RQ outperforms multi-task learning in the three task orders: SST-SRL-WOZ, SST-WOZ-SRL, and WOZ-SRL-SST. This indicates that AQF-RQ can not only better alleviate catastrophic forgetting but also further strengthen forward transfer between tasks.
When the pseudo data is insufficient (i.e. γ = 0.05), the performance of LAMOL drops significantly, falling 6.5-7.6 percentage points lower than that of multitask learning. Furthermore, LAMOL 0.05 T differs from LAMOL 0.05 G , indicating that LAMOL is not robust enough for clear and unclear task boundaries when the amount of pseudo data is insufficient. Although LAMOL's job boundary is apparent, our proposed AQF-RQ can improve by 4.7 and 5.5 percentage points compared to LAMOL. This illustrates that even when the amount of pseudo data is insufficient, AQF-RQ is still robust to both clear and unclear task boundaries, as evidenced by AQF-RQ 0.05 T and AQF-RQ 0.05 G . It is also worth noting that AQF-RQ 0.05 T , which means that AQF-RQ is more data-efficient and still performs better than LAMOL with a 75% reduction in the amount of pseudo data.

Distribution of Pseudo Data
The another main objective of our proposed AQF-RQ is to generate more corresponding-task pseudo data. Ideally, the ratio of pseudo data for each task is 1:1, which means |ps i | = |ps j |, i ̸ = j. We can    Table 4: The results of pseudo data distribution after learning two tasks on [SST, SRL, and WOZ]. ps i represents the pseudo data of i-th task. The subscripts c and nc indicate whether the pseudo data is correspond to the task or not.
determine which task the pseudo data belongs to according to the Q because the Q for various tasks is distinct (see Table 7). For task-specific token [TASK], a pseudo sample is task-corresponding if the Q is correspond to the [TASK]. For taskindepedent token [GEN], we can not judge whether a pseudo sample corresponds to the task because of unclear task boundary. In this paper, we applied this Q-based judgment method to count the distribution of pseudo data after learning the second task.
As shown in Table 4, AQF-RQ can generate more pseudo data of the first task for most task orders when applying task-indepedent token [GEN]. Looking at the Table 3 together, it can be observed that the performance will be better if the ratio of the pseudo data of different tasks is closer to 1:1. For SRL-SST-WOZ, SRL-WOZ-SST, WOZ-SST-SRL, and WOZ-SRL-SST, AQF-RQ generates more pseudo data than LAMOL for the first task; hence, AQF-RQ outperforms LAMOL significantly. When γ = 0.05, AQF-RQ is 5.3-10.6 percentage points higher than LAMOL. When γ = 0.05, AQF-RQ is 1.3-3.9 percentage points higher than LAMOL. This is due to the fact that, under the AQF-proposed "BQCA" format, Q can be used to help the model pay more attention to the useful details in C, and then A being immediately next to C can further generate A more accurately.
The benefit of AQF-RQ to produce pseudo data is also significant when the sentence starts with the task-specific token [TASK]. As shown in the Table 4, the pseudo samples generated by AQF-RQ are task-corresponding in the majority of task orders, except SRL-SST-WOZ and WOZ-SRL-SST, where one or two pseudo samples are not task-corresponding. The amount of not taskcorresponding pseudo samples in AQF-RQ is substantially smaller than in LAMOL, giving AQF-RQ a 1.3-3.9 percentage point advantage over LAMOL.  The above experimental results demonstrate that not only does AQF-RQ generate better pseudo-data, but the "BQCA" format is also more data-efficient and robust for lifelong language learning.

Generation for Different Domains
We conducted experiments on five sequence generation tasks for different domains by comparing LAMOL T , LAMOL G , and L2KD with AQF-RQ. The result is similar to Section 5.2.1 and is detailed in Appendix C. AQF-RQ is not only better than LAMOL-based baselines on performance but can also forward/backward transfer more knowledge among the same task but different domains.

Effectiveness of the Ask Question First
To verify the effectiveness of our proposed AQF, we experimentally apply AQF to each baseline: LAMOL T , LAMOL G , LAMOL R , and L2KD. In Section 3.2, we stated that AQF is used better when the pseudo data is sufficient. Therefore, we set γ = 0.2 for experiments. This value is also the best setting for those baselines.
As shown in the Table 5, applying AQF to each baseline resulted in varying degrees of improvement. LAMOL T has the most noticeable improvement, with a 2 percentage point increase. This demonstrates that having the A close to the C using the "BQCA" format proposed by AQF can reduce information attenuation caused by the A being far away from the C. For example, the C and the A are separated by the Q in the "BCQA" format. This benefit is more obvious when the task boundary is clear (i.e. the sentence starts with [TASK]). [TASK] can tighten the bond between "BQC" and "A".
Applying AQF improves LAMOL G by 1.2 percentage points, which is less than LAMOL T . It is because the generated pseudo data is still biased toward new tasks due to the unclear task boundary (the sentence begins with the task-independent token [GEN]). However, thanks to the A being next to the C in the "BQCA" format, it is still capable of learning from a small amount of pseudo data.
For LAMOL R , AQF can enhance the model by 1.4 percentage points, which is only 0.1% behind multi-task learning. LAMOL R uses the real data of the old task instead of the generated pseudo data when learning new tasks, so this further demonstrates that the "BQCA"format proposed by AQF is more conducive to lifelong language learning: it can use Q to help the model pay more attention to the important information in C, then further makes the generation of A more accurate.
For L2KD, the application of AQF improves the model by one percentage point, demonstrating that AQF is still applicable to the previous LAMOLbased enhanced approaches and is capable of effectively applied to other LAMOL-based studies.
For MTL, AQF can bring an improvement of 1 percentage point. At the same time, the improvement of each baseline demonstrates that the improvement brought by AQF is comprehensive: it is not only conducive to alleviating catastrophic forgetting but also improves the upper bound of the model. Therefore, we believe that AQF has the potential to apply LLL to real-world scenarios.

Effectiveness of the Replay Questions
Since the Replay QuestionsRQ is proposed based on AQF, we verify and analyze the effectiveness of the RQ based on Table 3 and 5. The AQF-RQ performance in Table 3 subtracts the corresponding LAMOL+AQF performance in Table  5 is the gains obtained by AQF-RQ come from the RQ step. Therefore, RQ 0.2 G = AQF-RQ 0.2 G -LAMOL+AQF 0.2 G = 1.1, and RQ 0.2 T = AQF-RQ 0.2 T -LAMOL+AQF 0.2 T = 0.7. It demonstrates that the RQ step based on the AQF can help the model generate more task-corresponding pseudo samples by making P (Q i |B i ) > P (Q j̸ =i |B i ).

Conclusion and Future Work
This work proposed AQF-RQ which is a simple yet efficient and robust lifelong language learning method. We propose a new question-first data format that is train-efficient without additional computational resources and new parameters. In AQF's "BQCA" format, generating Q first makes it easier to generate task-corresponding pseudo-data, and A is more accurate because A is next to C. In addition, RQ can strengthen the model's attention to the problem, so that the model has a greater probability of generating task-corresponding questions, and further generates more task-corresponding pseudodata. AQF-RQ effectively alleviates catastrophic forgetting, only 0.36% lower than multi-task learning. Due to a lack of computing resources, we did not experiment on larger datasets and longer task orders, which we leave as future work. In addition, we will investigate ways to enhance performance when the task boundary is unclear.

A Datasets
As we can not obtain test set of SQuAD since it is hidden from the host, we use development set for testing. For other tasks, we use the corresponding test set. A normalized F1 (nF1) metric that lower text and remove punctuation and articles, is used to evaluate SQuAD and SRL. The exact match (EM) is used to evaluate SST. The turn-based dialogue state EM (dsEM) is used for WOZ. The ROUGE is used to evaluate E2ENLG and RNNLG (rest/hotel/tv/laptop)The size of each dataset is detailed in Table 7. As shown in Table 7, SST, WOZ, E2ENLG and RNNLG (rest/hotel/tv/laptop) only have one question. The SQuAD has mainly 12 types of questions and other countless types of questions. SRL has 6 types of questions.

B Hyperparameter
The main hyperparameters in our experiments are detailed in Table 6.

C Sequence Generation for Different Domains
We conducted experiments on five sequence generation tasks for different domains by comparing LAMOL T , LAMOL G , and L2KD with AQF-RQ. Experiments are conducted with sufficient pseudo data (i.e. γ=0.2). Following L2KD, we select the task order from hard to easy: E2ENLG-RNNLG (rest) -RNNLG (hotel) -RNNLG (tv) -RNNLG (laptop). As shown in the Fig.3, we can observe that all models perform well except for Finetune and LAMOL G . However, AQF-RQ can still outperform other baselines. When learning the same