{'fraction_non_alphanumeric': 0.02956081081081081, 'fraction_numerical': 0.012749907441688264, 'mean_word_length': 4.815313193837045, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Continual learning: A comparative study on how to defy forgetting in classification tasks

Artificial neural networks thrive in solving the classification problem for a particular rigid task, where the network resembles a static entity of knowledge, acquired through generalized learning behaviour from a distinct training phase. However, endeavours to extend this knowledge without targeting the original task usually result in a catastrophic forgetting of this task. Continual learning shifts this paradigm towards a network that can continually accumulate knowledge over different tasks without the need for retraining from scratch, with methods in particular aiming to alleviate forgetting. We focus on task-incremental classification, where tasks arrive in a batch-like fashion, and are delineated by clear boundaries. Our main contributions concern 1) a taxonomy and extensive overview of the state-of-the-art, 2) a novel framework to continually determine stability-plasticity trade-off of the continual learner, 3) a comprehensive experimental comparison of 10 state-of-the-art continual learning methods and 4 baselines. We empirically scrutinize which method performs best, both on balanced Tiny Imagenet and a large-scale unbalanced iNaturalist datasets. We study the influence of model capacity, weight decay and dropout regularization, and the order in which the tasks are presented, and qualitatively compare methods in terms of required memory, computation time and storage.


INTRODUCTION
In recent years, machine learning models have been reported to exhibit or even surpass human level performance on individual tasks, such as Atari games [1] or object recognition [2]. While these results are impressive, they are obtained with static models incapable of adapting or expanding their behavior over time. As such, the training process needs to be restarted each time new data becomes available and the model needs an update. In a dynamic world like ours, such a practice becomes intractable when moving to real scenarios where the data is streaming or may only be available temporarily due to storage constraints or privacy issues. This calls for systems that adapt continually and keep on learning over time.
An illuminating example are natural cognitive systems. While humans may gradually forget some old information, a complete loss of previous knowledge is rarely attested [3]. Humans tend to learn concepts sequentially. During this process, revisiting of old concepts (i.e., observing examples of already learned concepts) may occur but is not essential to preserve the knowledge of the old concepts.
Without special measures, artificial neural networks, trained with stochastic gradient descent, cannot learn in this manner: they suffer from catastrophic forgetting of old concepts as new ones are learned [3]. To circumvent this problem, research on artificial neural networks has focused mostly on static tasks, where the data is usually shuffled to ensure i.i.d. conditions, and performance largely increases with repeated revisiting of the training data over multiple epochs.
Interestingly, catastrophic forgetting has also been observed in biological systems [4]: when learning two time events sequentially in rats, a complete wipe out of the first one occurs once the second is learned. This is exactly the catastrophic forgetting observed also when a simple neural network is used to model events presented sequentially. Catastrophic interference is a direct result of a more general problem in neural networks, the so-called "stability-plasticity" dilemma [5]. While plasticity refers to the ability of integrating new knowledge, stability indicates the preservation of previous knowledge while new data is encoded. This stability-plasticity trade-off is an essential aspect in both artificial and biological neural intelligent systems.
Continual Learning studies the problem of learning from an infinite stream of data, with the goal of gradually extending the acquired knowledge and using it for future learning [6]. The data can stem from changing input domains (e.g. varying imaging conditions) or be associated with different tasks (e.g. different fine-grained classification problems). Continual learning is also referred to as lifelong learning [6], [7], [8], [9], [10], [11], sequential learning [12], [13], [14] or incremental learning [15], [16], [17], [18], [19], [20], [21]. The main criterion is the sequential nature of the learning process where only a small portion of input data from one or a few tasks is available at once. The main challenge is to learn without catastrophic forgetting, that is: the performance on a previously learned task or domain should not (significantly) degrade over time as new tasks or domains are added.
To keep the focus, in this work we limit the scope of our study in three ways. First, we focus on classification problems only. Classification is, arguably, one of the most established tasks for artificial neural networks, with good performance using relatively simple, standard and well understood network architectures. Second, we only consider the task-incremental setting, where data arrives in batches and one batch corresponds to one task (i.e. a new set of categories to be learned). In other words, we assume that for a given task, all data becomes available simultaneously. This way, the task can be learned in an offline manner, with multiple epochs over all the training data belonging to the task and shuffling of the task data to ensure i.i.d. conditions during training. Importantly, this only applies to the current task data: data from previous or future tasks cannot be accessed. However, optimizing for a new task will result in catastrophic forgetting, with significant drops in performance for the old tasks, unless special measures are taken. The effectiveness of those measures, under different circumstances, is exactly what this paper is about. The third limitation in the scope concerns a multi-head configuration, where each task is allocated an exclusive output layer or head. This is in contrast to the even more challenging setup where all tasks share a single head, which may introduce additional interference in learning, and increases the amount of output nodes to chose from at test time. Instead, we assume it is known which task a given test sample belongs to. The setup is described in more detail in Section 2. In Section 7 we discuss the open issues towards tackling a more general setting.
Early research works [22], [23], [24] developed several strategies to mitigate the forgetting under the condition of not storing the training data, mostly at a small scale of just a few examples and considering shallow networks. More recently, the catastrophic forgetting problem and the continual learning paradigm received increased attention, especially in the task-incremental learning setting as studied here. There is, however, limited consensus on evaluation protocols and datasets to be used. While all papers provide some evidence that there is at least one setting (combination of tasks, model, hyperparameters, etc.) under which the proposed method reduces the forgetting and outperforms a finetuning baseline and possibly some alternative methods, there is no comprehensive experimental comparison performed to date.
"Which method really works best?" This question can only be answered if we can set up a fair comparison, which leads to another question: "How can the trade-off between stability and plasticity be set in a consistent manner, using only data from the current task (Section 4)?" We propose a principled framework to do so and, within this framework, study whether particular settings are more advantageous for specific methods. In particular, we seek to answer the following inquiries with our experiments (Section 6): "What is the effect on forgetting when altering the model size, and thus the capacity? Is it better to go for a wider or a deeper model? Are the proposed methods only effective in a vanilla setting or can they be combined with typical regularization schemes such as weight decay or dropout? How do methods cope with larger scale and unbalanced tasks? Does the order in which the classification tasks are presented matter? How do they compare in terms of memory, storage and computational requirements?" These are the questions we aim to tackle in this survey. Related work. Continual learning has been the subject of several recent surveys [9], [25], [26], [27]. In [9], Parisi et al. describe a wide range of methods and approaches, yet without an empirical evaluation or comparison. In the same vein, [25] descriptively surveys and formalizes continual learning, but with an emphasis on dynamic environments for robotics. Pfülb and Gepperth [26] perform an empirical study on catastrophic forgetting and develop a protocol for setting hyperparameters and method evaluation. However, they only consider two methods, namely Elastic Weight Consolidation (EWC) [28] and Incremental Moment Matching (IMM) [29]. Also, their evaluation is limited to small datasets. Farquhar and Gal [27] survey continual learning evaluation strategies and highlight the shortcomings of the common practice of using one dataset with different pixel permutations (typically permuted MNIST [30]). In addition, they show that incremental task learning under multi-head settings hides a big part of the true difficulty of the problem. While we agree with this statement, we still opted for this setting for our survey, as it allows us to compare existing methods without major modifications to the proposed algorithms. They also propose a couple of desiderata for evaluating continual learning methods. However, their study is limited to MNIST and Fashion-MNIST datasets which are well known to be easy datasets and far from realistic data encountered in practical applications. Further, Kemker et al. [31] compare three methods, EWC [28], PathNet [32] and replay based method GeppNet [17]. Their experiments are performed on a simple fully connected network based on three sequences of tasks composed of 3 datasets only: MNIST, CUB-200 and AudioSet. None of these works systematically addresses the questions raised above. Paper overview. In Section 2, we describe the task incremental setting adopted in most of the literature and also used in this paper. Next, Section 3 surveys different approaches towards continual learning, structuring them into three main groups: replay-based methods, regularizationbased methods and parameter isolation-based methods. An important issue when aiming for a fair comparison of methods is the selection of hyperparameters (in particular, the learning rate and the stability-plasticity trade-off). In Section 4 we introduce a novel framework to deal with this problem without requiring access to data from previous or future tasks. Section 5 provides details on the methods selected for our experimental evaluation. Section 6 describes the actual experiments and main findings, and compares the methods qualitatively. In Section 7 we look further ahead, highlighting additional challenges in the field, moving beyond the task incremental setting towards true continual learning. We emphasize the relation with other fields in Section 8. Section 9 concludes the paper. Finally, we add implementation details and further experimental results in the supplemental material.

THE TASK INCREMENTAL LEARNING SETTING
Due to the difficulty of the general continual learning problem and the various challenges that have to be dealt with, most methods relax the general setting to an easier task incremental one.
The task incremental setting considers a sequence of tasks. In this setting one task is received at a time along with its training data. An offline training is then performed until convergence. With X (t) a set of data samples for task t and Y (t) the corresponding ground truth labels, (X (t) , Y (t) ) randomly drawn from a distribution D (t) , the goal is to control the statistical risk of all seen tasks given limited or no access to data (X (t) , Y (t) ) from previous tasks t < T : where T is the number of tasks seen so far, f t represents the output of the network for task t and θ represents the network weights (parameters). For the current task (last term in the summation of Eq. 1), the statistical risk can easily be approximated by the empirical risk However, for the old tasks, the data is no longer available, so the statistical risk cannot be evaluated for new parameter values. How to determine the optimal parameters θ T , with the aim to (approximately) optimize Eq. 1, is the main research focus in task incremental learning. The word task here refers to an isolated training phase of a new batch of data that belongs to a new group of classes, a new domain, or a different output space (e.g. scene classification v.s. hand written digit classification). As such, following [33], a finer categorization distinguishes between incremental class learning, where P (Y t ) = P (Y t+1 ) but {Y t } = {Y t+1 }, incremental domain learning, where P (X t ) = P (X t+1 ) and P (Y t ) = P (Y t+1 ), and task incremental learning, with P (Y t ) = P (Y t+1 ) and P (X t ) = P (X t+1 ).
In this paper, our experiments will focus on the task incremental learning setting for a sequence of classification tasks.

CONTINUAL LEARNING APPROACHES
Early works considered the catastrophic interference problem as observed when learning sequentially examples of different input patterns (e.g. of different categories). Several directions have been explored, such as reducing representation overlap [22], [34], [35], [36], [37], replaying samples or virtual samples from the past [10], [24] or introducing dual architectures [38], [39], [40]. However, due to resource restrictions at the time, these works were mainly considering few examples (in the order of tens) and were based on specific shallow architectures.
With the recent increased interest in neural networks, continual learning and catastrophic forgetting also received more attention. [30] studied empirically the forgetting when learning two tasks sequentially using different activation functions and dropout regularization [41]. [42] studied incremental task learning from a theoretical perspective with the goal of transferring knowledge to future tasks.
More recent works have addressed continual learning with longer sequences of tasks and larger number of examples. In the following, we will review the most important works. We distinguish three families, based on how task specific information is stored and used throughout the sequential learning process: • Replay-based methods • Regularization-based methods • Parameter isolation-based methods Note that our categorization overlaps to some extent with that introduced in [27], [43]. However, we believe it offers a more general overview and covers most of the existing works. A summary can be found in Figure 1.

Replay-based methods
This line of work stores samples in their raw format or compressed in a generative model. The stored samples from previous tasks are replayed while learning a new task to alleviate forgetting. These samples/pseudo-samples can be either used for rehearsal, approximating the joint training of previous and current tasks, or to constrain the optimization of the new task loss not to interfere with the previous tasks.
Rehearsal methods [18], [45], [46], [61] explicitly retrain on a subset of stored samples while training on new tasks. The performance of these methods is upper bounded by joint training on previous and current tasks. Most notable is the work by Rebuffi et al. on incremental class learning [18], that stores a subset of exemplars per class, selected to best approximate the mean of each class in the feature space being learned. The method is constrained to a fixed budget, hence to accommodate new classes, old classes' exemplars are re-selected according to the same criterion. In settings where data is streaming with no clear task boundaries, [44] suggests the use of reservoir sampling to limit the number of stored samples to a fixed budget assuming an overall i.i.d. distributed data stream.
While rehearsal might be prone to overfitting the subset of stored samples and seems to be bounded by joint training, constrained optimization is an alternative solution that leaves more room for backward/forward transfer. As proposed in GEM [50] under the task incremental setting, the key idea is to only constrain the update of the new task to not interfere with the previous tasks. This is achieved through projecting the estimated gradient direction on the feasible region outlined by previous tasks' gradients through a first order Taylor series approximation. A-GEM [8] has relaxed the problem to projection on one direction estimated by randomly selected samples from a buffer of previous tasks data. [43] have recently extended this solution to a pure online continual learning setting where no task boundaries are provided. They propose to select a subset of samples that maximally approximate the feasible region of the historical data.
In the absence of previous samples, pseudo rehearsal is an alternative strategy used in the early works with shallow neural networks. Random inputs and the outputs of previous model(s) given these inputs are used to approximate the previous tasks samples [24]. With deep networks and large input vectors (e.g. full resolution images) random input cannot cover the input space [47]. Recently, generative models have shown the ability to generate high quality images [62], [63] which opened up the possibility to model the data generating distribution and retrain on the generated examples [14]. However, this also adds to the complexity of training the generative model continually, with extra care to balance the retrieved examples and avoid the mode collapse problem.

Regularization-based methods
When no storage of raw input is possible, an important line of works proposes an extra regularization term in the loss function to consolidate previous knowledge when learning on new data. The constraint of not storing any historical sample is mainly motivated by privacy reasons, as in the case of medical applications, in addition to being memory efficient. We can further divide these methods into data focused and prior focused methods.

Data-focused methods
The basic building block in data-focused methods is the knowledge distillation from a previous model (trained on a previous task) to the model being trained on the new data. It was first proposed by [10] to use the output of previous tasks' models given new task input images mainly for improving the new task performance. It has been reintroduced by LwF [53] to mitigate forgetting and transfer knowledge, using the output of the previous model as soft labels for previous tasks. Other works [54], [55] have been introduced with related ideas, however, it has been shown that this strategy is vulnerable to domain shift between tasks [7]. In an attempt to overcome this issue, [11] proposed to constrain the features of each task in its own learned low dimensional space through an incremental integration of shallow autoencoders.

Prior-focused methods
To mitigate forgetting, prior focused methods estimate a distribution over the model parameters, which is used as a prior when learning new data. As this quickly becomes infeasible w.r.t. the number of parameters in deep neural networks, parameters are usually assumed independent and an importance weight is estimated for each parameter in the neural network. During the training of later tasks, changes to important parameters are penalized. Elastic weight consolidation (EWC) [28] was the first to establish this approach. Variational Continual Learning (VCL) has introduced a variational framework for this family [64]. [51] estimates the importance weights during the training of a task based on the contribution of their update to the decrease of the loss, while [15] suggests to estimate the importance weights online based on unlabelled data which allows for user adaptive settings and increased flexibility in deploying the method. While the prior focused family relies on tasks boundaries to estimate the prior distribution, [65] extends [15] to task free settings.
Overall, the soft penalty introduced in the regularization family might not be sufficient to restrict the optimization process to stay in the feasible region of the previous tasks, especially with long sequences [27], which might result in an increased forgetting of earlier tasks, as we shall show in the experiments (Section 6).

Parameter isolation-based methods
To prevent any possible forgetting of the previous tasks, in this family, different subsets of the model parameters are dedicated to each task. When there is no constraints on the size of the architecture, this can be done by freezing the set of parameters learned after each previous task and growing new branches for new tasks [60], [66], or even making a complete copy of the model for each new task [7].
Alternatively, under a fixed architecture, methods proceed by identifying the parts that are used for the previous tasks and masking them out during the training of the new task. This can be either imposed at the parameters level [32], [56] or at the neurons level as proposed in [58].
Most of these works require a tasks oracle to activate the corresponding masks or task branch during prediction. Expert gate [7] avoids this problem through learning an auto-encoder gate.
In general, this family is restricted to the task incremental setting and better suited for learning a long sequence of tasks when models capacity is not constrained and optimal performance is a priority.

CONTINUAL HYPERPARAMETER FRAMEWORK
Methods tackling the continual learning problem typically involve extra hyper parameters to balance the stabilityplasticity tradeoff. These hyper parameters are in many cases tuned via a grid search that uses held-out validation data from all tasks, including previous ones. However, the use of such validation data violates the continual learning setting, namely the assumption of no access to previous tasks data. This may lead to overoptimistic results, that cannot be reproduced in a true continual learning setting. As it is of our concern in this survey to provide a comprehensive and fair study on the different continual learning methods, we need to define a standard protocol to set the hyper parameters of all methods adhering to the studied continual learning setting. This allows not only for a fair comparison over the existing approaches but for a general strategy that can be used in future research as well as in industry to deploy continual learning methods in real situations.
To comply with the considered setting of not requiring data from previous tasks, our proposed protocol only assumes access to the new task data. In order to achieve the best balanced tradeoff between retaining previous tasks knowledge, i.e. stability, and successfully integrating new tasks information, i.e. plasticity, we start off by setting the hyper parameters to values that ensure minimal loss on previous tasks performance. If a pre-defined threshold on the performance of the new task validation data can't be achieved, we then decay these parameters values until reaching the desired performance on the new task. Algorithm 1 illustrates the main steps of our protocol, which can be divided into two main phases, to be repeated for each new task: Maximal Plasticity Search. Starting from the model trained on the preceding tasks, parameterized by θ t , maximal plasticity search first finetunes a copy of the model θ t ′ on the new task data with a learning rate η * F T obtained via a coarse grid search with the goal of obtaining the highest accuracy A * F T on a held-out validation set from the new task. The accuracy A * F T represents the best accuracy that can be achieved while disregarding the previous tasks. Note that a continual learning method can still achieve a better performance on the new task than fine-tuning but this stands as a reference point for our hyper parameter search.
Stability Decay. In the second phase we train θ t with the acquired learning rate η * F T using the considered continual learning method with other hyper-parameters set to their highest values ensuring minimum forgetting. To avoid redundant stability decay iterations, decayed hyperparameters are propagated to later tasks. We define a threshold p indicating the tolerated decay on the new task performance compared to fine-tuning. When this threshold is not met, we decrease the hyperparameters values and repeat this phase. This corresponds to increasing the plasticity of the model in order to reach the desired performance threshold. A ← Finetune D t+1 , η; θ t ⊲ Finetuning accuracy 4: if A > A * then 5: A ← CLM D t+1 , η * ; θ t 10: if A < (1 − p)A * then 11: H ← α · H ⊲ Hyperparameter decay 12: end if 13: while A < (1 − p)A *

COMPARED METHODS
In Section 6, we carry out a comprehensive comparison between representative methods from each of the three families of continual learning approaches introduced in Section 3. For clarity, we first provide a brief description of the selected methods, highlighting their main characteristics.

Replay-based methods
iCaRL [18] was the first replay-based method, focused on learning in a class-incremental way. Assuming a fixed allocated memory, it selects and stores those samples (also called exemplars) closest to the feature mean of each class. During training, along with minimizing the estimated loss on the new classes, the method also minimizes the distillation loss between targets obtained from the predictions of the previous model and current model predictions on the previously learned classes. As the weight of this distillation loss correlates with preservation of the previous knowledge, we optimize this hyperparameter in our proposed protocol. In our study, we consider the task incremental setting where each task is represented by a group of classes. To perform a fair comparison with other methods, iCaRL is also implemented in a multi-head fashion.
GEM [50], on the other hand, exploits the exemplars to solve a constrained optimization problem, projecting the current task gradient in a feasible area outlined by the previous task gradients. The authors observe increased backward transfer by altering the gradient projection with a small constant γ ≥ 0, constituting H in our framework.
The major drawback of replay-based methods is the limited scalability, as storing of and computation on raw input samples for each of the tasks becomes intractable with an ever increasing amount of tasks. Although a fixed memory consumption can be imposed, this limits the amount of exemplars per task, decreasing the representativeness of the exemplar sets w.r.t. the original task distribution. Furthermore, storing the raw input samples may also lead to privacy issues.

Regularization-based methods
This survey strongly focuses on regularization-based methods, comparing five methods in this family. The regularization strength correlates to the amount of knowledge retention, and therefore constitutes H in our hyperparameter framework.
Learning without Forgetting (LwF) [53] retains knowledge of preceding tasks by means of knowledge distillation [67]. Before training the new task, network outputs for the new task data are recorded, and are subsequently used during training to distill prior task knowledge. However, the success of this method depends heavily on the new task data and how strong it is related to prior tasks. Distribution shifts with respect to the previously learned tasks can result in a gradual error build-up to the prior tasks as more dissimilar tasks are added [7], [53]. This error build-up also applies in a class-incremental setup, as shown in [18]. Another drawback of this approach is the computational overhead and a minimal additional memory consumption in the preprocessing step for training, requiring a forward pass for each of the new data points and the output recording to be stored. LwF is specifically designed for classification, but has also been applied to other problems, such as object detection [20].
Encoder Based Lifelong Learning (EBLL) [11] extends the concept of LwF by also preserving important low dimensional feature representations of previous tasks. For each task, after training the network, an under-complete autoencoder is inserted between the feature extractor and classifier of the network and trained in an end-to-end fashion, learning a feature projection on a lower dimensional manifold, optimized for the corresponding task. During training, an additional regularization term impedes the current feature projections to deviate from the optimal ones for the previous tasks. Although the required memory grows linearly with the number of tasks, the autoencoder size is generally only a small fraction of the backbone network. The main computational overhead occurs in the autoencoder training and collecting the feature projections for the samples in each optimization step.
Elastic Weight Consolidation (EWC) [28] applies the Bayesian framework for neural networks [68], which allows to find posterior distributions of parameters instead of mere point estimates in parameter space, by introducing uncertainty on the network parameters. Following sequential Bayesian estimation, the old posterior of previous tasks T 1:n−1 constitutes the prior for new task T n , founding a mechanism to propagate old task importance weights. The true posterior is intractable, and is therefore estimated using a Laplace approximation, assuming a Gaussian with the parameters θ n determining the mean and a diagonal precision estimated by the Fisher Information Matrix (FIM), which near a minimum shows to be equivalent to the positive semi-definite second order derivative of the loss [69]. In practice, for computational efficiency the FIM is typically approximated by the empirical FIM to avoid additional backward passes [70]. Therefore the importance weight Ω n k is defined as the squared gradient of loss function L w.r.t. parameter θ n k : and is only calculated after training for task T n . In the Bayesian intuition, parameter θ n k is deemed more important with higher certainty, i.e. with higher precision. From the optimization perspective, the second order derivative of the loss represents the curvature in parameter space. Hence θ n k with a steep surface significantly alters the loss of task T n , and should therefore get a high importance weight Ω n k . A first limitation of EWC is its need to store the FIM for each of the learned tasks, resulting in a ever-increasing memory requirement as the amount of tasks grows. Secondly, the task FIM is approximated after optimizing the task, inducing gradients close to zero, resulting in very little regularization. This is inherently coped with in our framework, as the regularization strength is initially very high and only lowers when stability decay is imposed. Variants of EWC are proposed to address these issues in [71], [52] and [16].
Synaptic Intelligence (SI) [51] breaks the EWC paradigm of determining the importance weights in a separate phase after training. Instead, they maintain an online estimate ω n during training of a new task T n , while taking into account the importance estimates of all previous tasks T 1:(n−1) : with ∆θ t k = θ t k − θ t−1 k the task-specific parameter distance, and damping parameter ξ avoiding division by zero. Note that the accumulated importance weights Ω n k are still only updated after training of task T n as in EWC. Nevertheless, the importance weights ω n k for the newly trained task are already acquired in an online fashion during training, and therefore no additional inference and backward passes are required to obtain the post-training gradients. However, the efficient online calculation knife cuts both ways. First, the widely used stochastic gradient descent (SGD) incurs noise in the approximated gradient during training, and therefore the authors state that the importance weights tend to be overestimated. Second, catastrophic forgetting of the knowledge from a pretrained network becomes inevitable, as importance weights can't be retrieved. In another work, Riemannian Walk [16] combines the SI path integral with an online version of EWC to measure parameter importance.
Memory Aware Synapses (MAS) [15] redefines the importance measure of parameters to an unsupervised setting. Instead of calculating the gradients of the loss function L as in (3), the authors obtain the gradients of the squared L 2 norm of the learned output function F of the network. Therefore, the importance weights become: Previously discussed methods require supervised data for the loss-based importance weight estimations, and are therefore confined to the mere available task-specific training data. MAS on the other hand, allows importance weights to be estimated on an unsupervised held-out dataset, and is therefore able to adapt to user-specific data. Incremental Moment Matching (IMM) [29] estimates Gaussian posteriors for each of the task parameters, in the same vein as EWC, but inherently differs in its use of model merging. In the merging step, the mixture of Gaussian posteriors is approximated by a single Gaussian distribution, i.e. a new set of merged parameters θ 1:n and corresponding covariances Σ 1:n . Although the merging strategy implies a single merged model for deployment, it requires to store a model during training for each of the learned tasks. In their work, two methods for the merge step are proposed: mean-IMM and mode-IMM. In the former, the weights θ k of the task-specific networks are averaged following the weighted sum: with α t k the mixing ratio of task T t , subject to n t α t k = 1. The second merging method mode-IMM, instead aims for the mode of the mixture of Gaussians. Here, the inverse covariance or precision matrices are required, which is again assumed diagonal and approximated by the FIM, equivalent to the importance weights Ω n k in (3), with: The importance weights of all tasks are mixed using mixing ratios in (8), resulting in Ω 1:n k which is subsequently used as a normalization constant when merging task-specific model parameters in (7). When two models converge to a different local minimum due to independent initialization, simply averaging the models might result in an increased loss, as there are no guarantees for a flat or convex cost surface between the two points in parameter space [72]. Therefore, IMM suggests three transfer techniques to aim for an optimal solution in the interpolation of the task-specific optimized models: i) Weight-Transfer initializes the network of the new task T n with the parameters of the previous task T n−1 ; ii) Drop-Transfer is a variant of dropout [73] with the parameters of previous task T n−1 as the zero point; iii) L2-transfer is a variant of L2-regularization, again with the parameters of previous task T n−1 redefining the zero point. In this study, we compare mean-IMM and mode-IMM with both weight-transfer and L2-transfer.

Parameter isolation-based methods
PackNet [56] iteratively assigns a subset of the parameters to each of the consecutive tasks by constituting a corresponding binary mask. For each new task T n PackNet requires two training phases. First, the network is trained while fixing the parameters θ 1:n−1 assigned to previous tasks. After the first training phase, a predefined proportion of the remaining non-fixed parameters is allotted to the new task, defined by mask m n . Selection of the parameters is determined by highest magnitude, serving as indicator for parameter importance in this work. In a second training round, this subset of most important parameters θ n is retrained. However, besides fixing all parameters of previous tasks θ 1:n−1 , the remaining unassigned parameters are masked out. This ensures preservation of performance for task T n in inference, utilizing solely parameters which are not masked out by m 1:n . Although PackNet allows explicit allocation of network capacity to each task, it remains inherently limited in the amount of tasks that can be assigned to a model.

EXPERIMENTS
In this section we first discuss the experimental setup in Section 6.1, followed by a comparison of all the methods on a common BASE model in Section 6.2. The effects of changing the capacity of this model are discussed in Section 6.3. Next, in Section 6.4 we look at the effect of two popular methods for regularization. We continue in Section 6.5, scrutinizing the behaviour of continual learning methods in a real-world setup, abandoning the artificially imposed balancedness between tasks. In addition, we investigate the effect of the task ordering in both the balanced and unbalanced setup in Section 6.6. Finally in Section 6.7, we elucidate a qualitative comparison in Table 8, which summarizes limitations and resource requirements for the methods.

Experimental Setup
Datasets. We conduct image classification experiments on two datasets, the main characteristics of which are summarized in Table 1. First, we use the Tiny Imagenet dataset [74]. This is a subset of 200 classes from ImageNet [75], rescaled to image size 64 × 64. Each class contains 500 samples subdivided into training (80%) and validation (20%), and 50 samples for evaluation. In order to construct a balanced dataset, we assign an equal amount of 20 randomly chosen classes to each task in a sequence of 10 consecutive tasks. One could argue that such a class-incremental setting is not a good fit with our evaluation per task, i.e. using an oracle at test time. We nevertheless opted for this setting, as it ensures that all tasks are roughly similar in terms of difficulty, size, and distribution, making the interpretation of the results easier.
The second dataset is based on iNaturalist [76]. It aims for a more real-world setting with a large number of finegrained categories and highly imbalanced classes. On top, we impose task imbalance and domain shifts between tasks by assigning 10 super-categories of species as separate tasks. We selected the most balanced 10 super-categories from the total of 14 and only retained categories with at least 100 samples. More details on the statistics for each of these tasks can be found in Table 7. We only utilize the training data, subdivided in training (70%), validation (20%) and evaluation (10%) sets, with all images measuring 800 × 600.
In this survey we scrutinize the effects of different task orderings for both datasets in Section 6.6. Apart from that section, discussed results are performed on a random ordering of tasks. Models. We summarize in Table 2 the models used for the experiments in this work. Due to the limited size of Tiny Imagenet we can easily run experiments with different models. This allows to analyze the influence of model capacity (Section 6.3) and regularization for each of the model configurations (Section 6.4). This is important, as the effect of model size and architecture on the performance of different continual learning methods has not received much attention so far. We configure a BASE model, two models with less (SMALL) and more (WIDE) units per layer, and a DEEP model with more layers. The models are based on a VGG configuration [77] but with less parameters due to the small image size. We reduce the feature extractor to comprise 4 max-pooling layers, each preceded by a stack of identical convolutional layers with a consistent 3 × 3 receptive field. The first max-pooling layer is preceded by one conv. layer with 64 filters. Depending on the model, we increase subsequent conv. layer stacks with a multiple of factor 2. The models have a classifier consisting of 2 fully connected layers with each 128 units for the SMALL model and 512 units for the other three models. The multi-head setting imposes a separate softmax output layer for each task. A detailed description can be found in supplemental. The size of iNaturalist imposes arduous learning. Therefore, we conduct the experiments solely for AlexNet [78], pretrained on ImageNet.
Evaluation Metrics. To measure performance in the continual learning setup, we evaluate accuracy and forgetting per task, after training each task. We define the measure of forgetting [16] as the difference between the expectation of acquired knowledge of a task, i.e. the accuracy when first learning a task, and the accuracy obtained after training one or more additional tasks. In the figures, we focus on the evolution of the accuracy for each task as more tasks are added. In the tables, we report the average accuracy and average forgetting on the final model, obtained by evaluating each task after learning the entire sequence of ten tasks.
Baselines. The discussed continual learning methods in Section 5 are compared against several baselines: 1) Finetuning uses the model learned on previous tasks as initialization and then optimizes the parameters for the current task. This baseline greedily trains each task without considering performance on previous tasks. This introduces catastrophic forgetting and serves as a weak lower bound of the average accuracy. 2) Joint training considers all data in the task sequence simultaneously, hence violating the continual learning setup (indicated with appended ' * ' in reported results). This baseline provides a weak upper bound.
For the replay-based methods we consider two additional finetuning baselines, extending baseline (1) with the benefit of using exemplars: 3) Basic rehearsal with Full Memory (R-FM) fully exploits the total available exemplar memory R, and incrementally divides the capacity equally over all the previous tasks. This is a baseline for replaybased methods defining memory management policies to exploit all memory (e.g. iCaRL). 4) Basic rehearsal with Partial Memory (R-PM) preallocates a fixed amount of exemplar memory R/T over all tasks T in the sequence. This assumes that the amount of tasks T is known beforehand, as used by methods lacking memory management policies (e.g. GEM).
Replay Buffers. The replay-based methods (GEM, iCaRL) and the corresponding baselines (R-PM, R-FM) can be configured with arbitrary size of replay buffer. Too large buffers would result in unfair comparison to the regularization and parameter isolation-based methods, with in its limit even holding all data for the previous tasks, as in joint learning, which is not compliant with the continual learning setup. Therefore, we use the memory required to store the BASE model as a basic reference for the amount of exemplars to store. This corresponds to the additional memory needed for propagating importance weights in the prior-focused methods (EWC, SI, MAS). For Tiny Imagenet, this gives a total replay buffer capacity of 4.5k exemplars. We also experiment with a buffer of 9k exemplars to examine the influence of increased buffer capacity. Note that taking the BASE model as reference implies that the replay-based methods use a significantly different amount of memory than the non-replay based methods; more memory for the SMALL model) and less memory for the WIDE and DEEP models. Comparisons between replay and non-replay based methods should thus only be done for the BASE model. In the following, notation of the methods without replay buffer size refers to the default 4.5k buffer. Learning details. During training the models are optimized with stochastic gradient descent with a momentum of 0.9.
Training lasts for 70 epochs unless preempted by early stopping. Unless otherwise noticed (in particular, in Section 6.4), we do not use any form of regularization. This may cause some level of overfitting, but avoids a possible interference with the incremental learning methods. We study these effects in more detail in a separate section. The framework we proposed in Section 4 can only be exploited in the case of forgetting-related hyperparameters, in which case we set p = 0.2 and α = 0.5. All methods discussed in Section 5 satisfy this requirement, except for IMM with L2 transfer for which we could not identify a specific hyperparameter related to forgetting. For specific implementation details about the continual learning methods we refer to the supplemental material.

Comparing Methods on the BASE Network
Tiny Imagenet. We start the evaluation of the continual learning methods discussed in Section 5 with a comparison using the BASE Network, on the Tiny Imagenet dataset, with random ordering of tasks. This provides a balanced setup, making interpretation of the results easier. Figure 2 shows the results of the regularization and parameter isolation-based methods. The results of the replaybased methods and the corresponding baselines can be observed in Figure 3. Each of these figures consists of 10 subpanels, with each subpanel showing the evolution of the test accuracy for a specific task (e.g. Task 1 for the leftmost panel) as more tasks are added for training. Since the nth task is added for training only after n steps, the curves get shorter as we move to the subpanels on the right. The average accuracy and average forgetting after training the whole sequence of tasks is given for each method in the figure legend.

Discussion
General observations. As a reference, it is relevant to highlight the soft upper bound obtained when training all tasks jointly. This is indicated by the star symbol in each of the subpanels. For the Tiny Imagenet dataset, all tasks have the same number of classes, same amount of training data and similar level of difficulty, resulting in similar accuracies for all tasks under the joint training scheme. The average accuracy for joint training on this dataset is 55.70%, while random guessing would give an average accuracy of 5%.
Further, as has been reported ample times in the literature, the Finetuning baseline suffers severely from catastrophic forgetting: initially good results are obtained when learning a task, but as soon as a new task is added, the performance drops, resulting in a poor average accuracy of only 21.30% and a high average forgetting of 26.90%.
With an average accuracy of 47.67%, PackNet shows the highest overall performance of all continual learning methods on the final model (after training all 10 tasks), i.e. without considering the doubled replay buffer size of 9k for which iCaRL attains 48.76%. When learning a new task, the need for compression means PackNet only has a fraction of the total model capacity available. Therefore, it typically performs worse on the new task compared to the other methods. However, by fixing the task parameters through masking, it allows complete knowledge retention until the end of the task sequence (no forgetting yields flat curves in the figure), resulting in the highest accumulation of knowledge at the end -at least when working with long sequences, where forgetting errors gradually build up for the other methods.
MAS and iCaRL show competitive results w.r.t. PackNet (resp. 46.90% and 47.27%). iCaRL starts with a significantly lower accuracy for new tasks, due to its nearest-neighbour based classification, but improves over time, especially for the first few tasks. Doubling replay-buffer size to 9k enhances iCaRL further to even outperform PackNet (48.76%).

Regularization-based methods.
In prior experiments where we did a grid search over a range of hyperparameters over the whole sequence (rather than using the framework introduced in Section 4), we observed MAS to be remarkably more robust to the choice of hyperparameter values compared to the two related methods EWC and SI. Switching to the continual hyperparameter selection framework described in Section 4, this robustness leads to superior results for MAS compared to the other two (46.90% vs. 42.43% and 33.93). Especially the smaller amount of forgetting stands out (1.58% vs 7.51% and 15, 77%). Further, SI underperforms compared to EWC on Tiny Imagenet. We hypothesize this may be due to the overfitting we observed when training on the BASE model (see results in supplemental). SI inherently constrains parameter importance estimation to the training data only, which is opposed to EWC and MAS able to determine parameter importance both on validation and training data in a separate phase after training of the task.
The data-driven methods LwF and EBLL obtain similar results as EWC (41.91% and 45.34% vs 42.43%). EBLL improves over LwF by residing closer to the optimal task representations, lowering average forgetting and improving accuracy. Apart from the first few tasks, the curves for these methods are quite flat, indicating a low level of forgetting (only 2.38% and 1.44%, for LwF and EBLL respectively).
Using importance weights to merge the models in mode-IMM clearly is superior to mean-IMM. Mode-IMM catastrophically forgets in evaluation on the first task, but shows transitory recovering through backward transfer in all subsequent tasks. Overall for Tiny Imagenet, the IMM methods do not seem competitive with the other strategies for continual learning -especially if one takes into account that they need to store all the models from previous tasks, making them much more expensive in terms of storage.
Replay-based methods. iCaRL starts from the same model as GEM and the regularization-based methods, but uses a feature classifier based on a nearest neighbor scheme. As indicated earlier, this results in lower accuracies on the first task after training. Remarkably, for about half of the tasks, the iCaRL accuracy increases when learning additional tasks, resulting in a salient negative average forgetting of −1.11%. Such level of backward transfer is unusual. After training all ten tasks, a competitive result of 47.27% average accuracy can be reported. Comparing iCaRL to its baseline R-FM shows significant improvements over basic rehearsal (47.27% vs 37.31%). Doubling the size of the replay buffer (iCaRL 9k) increases performance even more, making iCaRL outperform PackNet and all other methods with a global best 48.76% average accuracy.
The results of GEM are significantly lower than those of iCaRL (45.13% vs 47.27%). GEM is originally designed for an online learning setup, while in this comparison each method can exploit multiple epochs for a task. Additional experiments in supplemental exemplify the sensitivity of GEM to the amount of epochs, compared to iCaRL, from which we procure a GEM setup with 5 epochs for all the experiments in this work. Furthermore, the lack of memory management policy in GEM gives iCaRL a compelling advantage w.r.t. the amount of exemplars for the first tasks, e.g. training Task 2 comprises a replay buffer of Task 1 with factor 10 (number of tasks) more exemplars. Surprisingly, GEM 9k with twice as much exemplar capacity doesn't perform better than GEM 4.5k. This unexpected behavior may be due to the random exemplar selection yielding a less representative subset. Nevertheless, GEM convincingly improves accuracy of the basic replay baseline R-PM with the same partial memory scheme (45.13% vs 36.09%).
Comparing the two basic rehearsal baselines that use the memory in different ways, we observe that the scheme exploiting the full memory from the start (R-FM) gives significantly better results than R-PM for tasks 1 to 7, but not for the last three. As more tasks are seen, both baselines converge to the same memory scheme, where for the final task R-FM allocates an equal portion of memory to each task in the sequence, hence equivalent to R-PM.

Effects of Model Capacity
Tiny Imagenet. A crucial design choice in continual learning concerns the capacity of the network to be used. Aiming to learn a long sequence of tasks, a model with high capacity capable of capturing the series of tasks knowledge seems preferable. However, learning the first task using such model, with only data from a single task, holds the risk of overfitting, jeopardizing generalization performance. So far, we compared all methods on the BASE model. Next, we study the effects of extending or reducing the capacity of this architecture. Details of the models have been given in Section 6.1 and supplemental material. In the following, we discuss the results of the model choice, using again the random ordering of Tiny Imagenet. These results are summarized in the top part of Table 3 for regularization and parameter isolation-based methods and Table 4 for replaybased methods 1 .

Discussion
Overfitting. We observed overfitting for several models and methods, not only for the WIDE and DEEP models. Comparing the different methods, SI seems quite vulnerable to overfitting issues, while PackNet prevents overfitting to some extent, thanks to the network compression phase. General Observations. Selecting highest accuracy disregarding which model is used, iCaRL 9k and PackNet remain on 1. Note that, for all of our observations, we also checked the results obtained for the two other task orders (reported in the middle and bottom part of Table 3 and Table 4). The reader can check that the observations are quite consistent. the lead with 49.94% and 49.09%. Further, LwF shows to be competitive with MAS (resp. 46.79% and 46.90%). Table 3), we observe that it does not reach the same level of accuracy with the SMALL model as with the other models. In particular, the initial accuracies are similar, yet the level of forgetting is much more severe, due to the limited capacity of the model to learn new tasks.

Baselines. Taking a closer look at the finetuning baseline results (see purple box in
On the other hand, for joint training (blue box), results on the DEEP network are inferior compared to the shallower networks, implying the deep architecture to be less suited to accumulate all knowledge from the task sequence. As the performance of the joint learning serves as a soft upper bound for the continual learning methods, this already serves as an indication that a deep model may not be optimal for continual learning.
In accordance, replay baselines R-PM and R-FM show to be quite model agnostic with all but the DEEP model performing very similar. The baselines experience both less forgetting and increased average accuracy when doubling the replay buffer size.
SMALL model. Finetuning, SI and mean-IMM suffer from severe forgetting (> 20%) imposed by the decreased capacity of the network (see red underlining), making these combinations worthless in practice. Other methods experience alleviated forgetting, with EWC most saliently benefitting from a small network (see green underlinings).
WIDE model. Inspecting the results for the wide model, it's remarkable that, in contrast to the results we see with the other models, SI consistently outperforms EWC on the WIDE model (see orange box; even more clear in middle and bottom tables in Table 3, which we will discuss later). EWC performs worse for the high capacity WIDE and DEEP models and, as previously discussed, attains best performance for the SMALL model. LwF, EBLL and PackNet mainly reach their top performance when using the WIDE model, with SI performing most stable on both the WIDE and BASE models. IMM shows increased performance when using the BASE and WIDE model, for both mean and mode merging.
DEEP model. Over the whole line of methods (yellow box), extending the BASE model with additional convolutional layers results in lower performance. As we already observed overfitting on the BASE model, the additional layers may introduce extra unnecessary layers of abstraction, decreasing overall performance. For the DEEP model iCaRL outperforms all continual learning methods with both memory sizes.
Model agnostic. Over all orderings which will be discussed in Section 6.6, some of the methods don't exhibit a preference for any model, except for the common aversion for the detrimental DEEP model. This is in general most salient for all replay-based methods in Table 4, but also for PackNet, MAS, LwF and EBLL in Table 3.

Conclusion.
We can conclude that (too) deep model architectures do not provide a good match with the continual learning setup. For the same amount of feature extractor parameters, WIDE models obtain significant better results (on average 10% better over all methods in Table 3 and Table 4). Also too small models should be avoided, as the limited Fig. 2: Comparison of regularization and parameter isolation-based continual learning methods on Tiny Imagenet for the BASE model with random ordering, with average accuracy (average forgetting) for each method in the legend.  T2  T3  T4  T5  T6  T7  T8  T9  T10 Evaluation on Task  T1  T2  T3  T4  T5  T6  T7  T8  T9 T2  T3  T4  T5  T6  T7  T8  T9  T10 Evaluation on Task capacity that is available can cause forgetting. On average we observe a modest 0.94% more forgetting for the SMALL model compared to the BASE model. At the same time, for some models, poor results may be due to overfitting, which can possibly be overcome using regularization, as we will study next.

Effects of Regularization
Tiny Imagenet. In the previous subsection we mentioned the problem of overfitting in continual learning. Although an evident solution would be to apply regularization, this might interfere with the continual learning methods. Therefore, we investigate the effects of two popular regularization methods, namely dropout and weight decay, for the regularization and parameter isolation-based methods in Table 5, and for the replay-based methods in Table 6. For dropout we set the probability of retaining the units to p = 0.5, and weight decay applies a regularization strength of λ = 10 −4 .

Discussion
General observations. In Table 5 and Table 6, negative results (where the regularization hurts performance) are underlined in red. As one can see, this happens quite a few times, especially in combination with the SMALL model or with weight decay. Over all methods and models dropout mainly shows to be fruitful. This is consistent with earlier observations [79]. There are, however, a few salient exceptions (discussed below). Weight decay over the whole line mainly improves the wide network accuracies. In the following, we will describe the most notable observations and exceptions to the main tendencies.  Finetuning. Goodfellow et al. observe reduced catastrophic forgetting in a transfer learning setup with finetuning when using dropout [79]. Extended to learning a sequence of 10 consecutive tasks in our experiments, finetuning consistently benefits from dropout regularization. This is opposed to weight decay which results in increased forgetting and a lower performance on the final model. In spite of the good results for dropout, we regularly observe an increase in the level of forgetting, which is compensated for by starting from a better initial model, due to a reduction in overfitting. Dropout leading to more forgetting is something we also observe for many other methods (see blue boxes), and becomes more severe as the task sequence grows in length.
Joint training and PackNet. mainly relish higher accuracies with both regularization setups. By construction, they benefit from the regularization without interference issues. PackNet even reaches a top performance of almost 56%, that is 7% higher than the closest competitor (MAS).
SI. Most saliently thriving with regularization (with increases in performance of around 10%, see green box) is SI, which we previously found to be sensitive to overfitting. The regularization aims to find a solution to a well-posed problem, stabilizing the path in parameter space. Therefore, this provides a better importance weight estimation along the path. Even then, it's not competitive with the top performing methods in terms of obtained average accuracy.
EWC and MAS. The prior-focused methods suffer from interference when regularization is added. For dropout, having more redundancy in the model means there are less unimportant parameters left for learning the new tasks. For weight decay, parameters that were deemed important for the previous tasks are also decayed in each iteration, affecting the performance on the older tasks. A similar effect was noticed in [12]. However, in some cases, the effect of this interference is again compensated for by having better initial models. EWC only benefits from dropout on the WIDE and DEEP model, and similar to MAS prefers no regularization on the SMALL model.
LwF and EBLL suffer from using dropout, with only the DEEP model significantly improving performance. For weight decay the methods follow the general trend, enhancing the WIDE net accuracies.
IMM with dropout exhibits higher performance only for the WIDE and DEEP model, coming closer to the performance obtained by the other methods.
iCaRL and GEM. Similar to the observations in Table 5, there is a striking increased forgetting for dropout for all methods in Table 6. Especially iCaRL shows an increase in average forgetting, albeit consistently accompanied with higher average accuracy. Except for the SMALL model, all models for the replay baselines benefit from dropout. For GEM this benefit is only notable for the WIDE and DEEP models. For the replay-based methods weight decay doesn't improve average accuracy, nor forgetting, except mainly for the WIDE model. In general, the influence of regularization seems limited for the replay-based methods in Table 6 compared to the non-replay-based methods in Table 5.

Effects of a Real-world Setting
iNaturalist. Up to this point all experiments conducted on Tiny Imagenet are nicely balanced, with an equal amount of data and classes for each of the tasks. In further experiments with iNaturalist we scrutinize a highly unbalanced task sequence, both in terms of classes and available data per task. We train AlexNet pretrained on Imagenet, and track performance for each of the tasks in Figure 4a. The experiments exclude the replay-based methods as they lack a policy to cope with unbalanced data, which would make comparison highly biased to our implementation.

Discussion
General observations. Overall PackNet shows the highest average accuracy, tailgated by mode-IMM with superior forgetting by exhibiting positive backward transfer. Both PackNet and mode-IMM attain accuracies very close to joint training, e.g. Task 2 and 6 (PackNet), and Task 1 and 7 (mode-IMM).
Performance drop. Evaluation on the first 4 tasks shows salient dips when learning Task 5 for mean-IMM and regularization-based methods EWC, SI and MAS. In the random ordering Task 5 is an extremely easy task of supercategory 'Fungi' which contains only 5 classes and a small amount of data. Using the expert gates to measure relatedness, the first four tasks show no particularly salient relatedness peaks or dips for Task 5. Instead, this might imply that rather the limited amount of training data with only a few target classes enforces the network to overly fit to the specific task, causing forgetting of the previous tasks.
Path Integral. For iNaturalist we did not observe overfitting, which might be the cause of the stable SI behaviour in comparison to Tiny Imagenet.
LwF and EBLL. LwF catastrophically forgets on the unbalanced dataset with 13.77% forgetting, which is in high contrast with the results acquired on Tiny Imagenet. The supercategories in iNaturalist constitute a completely different task, imposing severe distribution shifts between the tasks. In the contrary, Tiny Imagenet is constructed from a subset of randomly collected classes, implying similar levels of homogeneity between task distributions. On top, EBLL constraints the new task features to reside closely to the optimal presentation for previous task features, which for the random ordering enforces forgetting to nearly halve from 13.77% to 7.51%, resulting in a striking 7.91% increase in average accuracy over LwF (from 45.39% to 53.30%).

Effects of Task Order
In this experiment we scrutinize how changing the task order affects both the balanced (Tiny Imagenet) and unbalanced (iNaturalist) task sequences. Our hypothesis resembles curriculum learning [80], implying knowledge is better captured starting with the general easier tasks followed by harder specific tasks. Tiny Imagenet. The previously discussed results of regularization and parameter isolation-based methods on top of Table 3, and replay-based methods on top of Table 4 concern a random ordering. We define a new order based on task difficulty, by measuring the accuracy over the 4 models obtained on held-out datasets for each of the tasks. This results in an ordering from hard to easy consisting of task sequence [5,7,10,2,9,8,6,4,3,1] from the random ordering, with the inverse order equal to the easy to hard ordering.
iNaturalist constitutes three orderings defined as follows. First, we alphabetically order the supercategories to define the random ordering, as depicted in Figure 4a. Further, the two other orderings start with task 'Aves' comprising most data, whereafter the remaining tasks are selected based on the average relatedness to the already included tasks in the sequence. Relatedness is measured using Expert Gate autoencoders following [7]. Selecting on highest relatedness results in the related ordering in Figure 4b, whereas selecting on lowest relatedness ends up with the unrelated ordering in Figure 4c. We refer to Table 7 for the configuration of the three different orderings.

Discussion
Tiny Imagenet. The main observations on the random ordering for the BASE model in Section 6.2 and model capacity in Section 6.3 remain valid for the two additional orderings, with PackNet and iCaRL competing for highest average accuracy, subsequently followed by MAS and LwF. In the following, we will instead focus on general observations between the three different orderings. Task Ordering Hypothesis. Starting from our curriculum learning hypothesis we would expect the easy-to-hard ordering to enhance performance w.r.t. the random ordering, and the opposite effect for the hard-to-easy ordering. However, especially SI and EWC show unexpected better results for the hard-to-easy ordering than for the easy-to-hard ordering. For PackNet and MAS, we see a systematic improvement when switching from hard-to-easy to easy-to-hard ordering. The gain is, however, relatively small. Overall, the impact of the task order seems insignificant. Replay Buffer. Introducing the two other orderings, we now observe that iCaRL doesn't always improve when increasing the replay buffer size for the easy-to-hard ordering (see SMALL and WIDE model). More exemplars induce more samples to distill knowledge from previous tasks, but might deteriorate stochasticity in the estimated feature means in the nearest-neighbour classifier. GEM does not as consistently benefit from increased replay buffer size (GEM 9k), which could also root in the reduced stochasticity of the constraining gradients from previous tasks. iNaturalist. The general observations for the random ordering remain consistent for the other orderings as well,    Amphibia  28  5319  755  1519  1  4  10  Animalia  10  1388  196  397  2  5  9  Arachnida  9  1192  170  341  3  8  7  Aves  314  65897  9362  18827  4  1  1  Fungi  5  645  91  184  5  6  2  Insecta  150  26059  3692  7443  6  9  3  Mammalia  42  8646  1231  2469  7  2  8  Mollusca  13  1706  241  489  8  7  4  Plantae  237  35157  4998  10045  9  10  5  Reptilia  56  10173  1447  2905  10  3  6 with PackNet and mode-IMM the two highest performing methods.
Small Task Dip. For the random ordering we observed a dip in accuracies for all preceding tasks when learning the very small Fungi task. Furthermore, for the unrelated ordering, the Fungi (Task 2) also experiences a performance dip for finetuning and EWC in Figure 4c (see leftmost evaluation panel of Task 1). On top, EWC shows the same behaviour for Fungi (Task 6) also in the related ordering (Figure 4b), and therefore seems to be most susceptible to highly varying task constitutions, and especially smaller tasks.
LwF and EBLL. Where EBLL improves LwF with a significant 7.91% accuracy in the random ordering, and 1.1% in the related ordering, it performs similar to LwF for the unrelated ordering.
SI and First Task. As observed on Tiny Imagenet as well, SI is sensitive to overfitting when estimating importance weights through the path integral. The same behaviour can be observed for the random ordering which starts with a rather small task Amphibia (5k training samples) showing very unstable learning curves. The two other orderings start from task Aves (65k training samples) from which the amount of data might act as a regularizer and prevent overfitting. Hence, the path integral can be stabilized, resulting in very competitive results compared to EWC and MAS.

Qualitative Comparison
The previous experiments have a strong focus on coping with catastrophic forgetting, whereas by construction each method features additional advantages and limitations w.r.t. the other methods. In Table 8 T2  T3  T4  T5  T6  T7  T8  T9  T10 Evaluation on Task T1  T2  T3  T4  T5  T6  T7  T8  T9 T2  T3  T4  T5  T6  T7  T8  T9  T10 Evaluation on Task Fig. 4: Continual learning methods accuracy plots for 3 different orderings of the iNaturalist dataset.
we emphasize task-agnosticity and privacy in our comparison. The heatmap is based on our experimental setup for Tiny Imagenet with the BASE model configured with batch size 200, all phases with multiple iterations have their epochs set to 10, gridsearches are confined to one node, and replay-buffers have 450 exemplars allocated per task. Note that we present the results in an indicative heatmap for a more high level relative comparison, as the results highly depend on our implementation. GPU Memory for training is slightly higher for EBLL, which requires the autoencoder to be loaded in memory as well.
At test time all memory consumption is more or less equal, but shows some deviations for GEM and PackNet due to our implementation. Especially PackNet is the only method to load additional masks in memory during evaluation, and hence increases memory requirements. However, the BASE model size (54MB) and hence mask size are rather small compared to concurrently loaded data (200 images of size 64 × 64 × 3 ). In our implementation we load all masks at once for faster prediction at test time, with the cost of increased memory consumption of the masks (factor 10 increase, one mask per task). This results in a doubling of memory requirement at test time for PackNet. Computation time for training is doubled for EBLL and PackNet (see light green), as they both require an additional training step, respectively for the autoencoder and compression phase. iCaRL training requires additional forward passes for each of the exemplar sets (factor 5 increase). GEM requires on top of that additional backward passes to acquire the gradient for each of the exemplar sets, and subsequently solve a quadratic optimization problem (factor 10 increase). During testing, iCaRL nearest-neighbor classifier is vastly outperformed by a regular softmax classifier in terms of computation time (factor 45 increase).
Storage indicates the additional required memory to be allocated to constitute the continual learning method. LwF and iCaRL both store the previous task model for knowledge distillation (although one could also store the recorded outputs instead ). On top of that, EBLL stores an autoencoder, small in size A relative to M , for each of the previous tasks. GEM instead needs to store the gradient for the exemplar set of each seen tasks. The prior-focused methods EWC and MAS store importance weights and the previous task model, with SI also requiring a running estimate of the current task importance weights. IMM, when naively implemented, stores for each task the model after training, and in mode-IMM additionally stores the importance weights. More storage efficient implementations propagate a merged model instead. Finally, PackNet requires a mask equal to the amount parameters in the model (M [bit]) for each task.
Task-agnostic and privacy. In deployment PackNet requires an oracle to indicate the task for a given sample in order to load the appropriate masks. Therefore, this setup inherently prevents task-agnostic inference. Next, we emphasize the privacy issues for replay-based methods iCaRL and GEM when storing raw images as exemplars.

Experiments Summary
We try to summarize our main findings for each of the methods in Table 9.

LOOKING AHEAD
In the experiments performed in this survey, we have considered an incremental task learning setting where tasks associated with their training data are received one at the time. An offline training is performed on each task data until convergence. This setting requires knowledge of the task boundaries (i.e. when tasks switch) and allows for multiple passes over large batches of training data. Hence, it resembles a relaxation of the desired continual learning system that is more likely to be encountered in practice. Below, we describe the general continual learning setting in which continual learning methods are expected to be deployed and outlines the main characteristics that future developed methods should aim to realize. The General Continual Learning setting considers an infinite stream of training data where at each time step, the system receives a (number of) new sample(s) {(x t , y t )} drawn non i.i.d from a current distribution D t that could itself experience sudden or gradual changes. The main goal is to optimally learn a function f parametrized by θ that minimizes a predefined loss ℓ on the new sample(s) without interfering with and possibly improving on those that were learned previously. Desiderata of an ideal continual learning scheme include: 1. Constant memory. The memory consumed by the continual learning paradigm should be constant w.r.t. the number of tasks or the length of the data stream. This is to avoid the need to deal with unbounded systems.

No task boundaries.
Being able to learn from the input data without requiring a clear task division brings great flexibility to the continual learning method and makes it applicable to any scenario where data arrives in a never ending manner.

Online learning.
A largely ignored characteristic of continual learning is being able to learn from a continuous stream of data without offline training of large batches or separate tasks. 4. Forward transfer. This characteristic indicates the importance of the previously acquired knowledge to aid the learning of new tasks. 5. Backward transfer. A continual learning system shouldn't only aim at retaining previous knowledge but preferably also at improving the performance on previous tasks when learning future related tasks. 6. Problem agnostic. A continual learning method should be general and not limited to a specific setting (e.g. only classification). 7. Adaptive. Being able to learn from unlabeled data would increase the method applicability to cases where original training data no longer exists and further open the door to a specific user setting adaptation.

No test time oracle.
A well designed continual learning method shouldn't rely on a task oracle to perform predictions. 9. Task revisiting. When revisiting a previous task again, the system should be able to successfully incorporate the new task knowledge. 10. Graceful forgetting. Given an unbounded system and infinite stream of data, a selective forgetting of unimportant information is an important mechanism to achieve a balance  After establishing the general continual learning setting and the main desiderata, it is important to identify the critical differences between continual learning and other closely related machine learning fields that share some of the characteristics mentioned above with continual learning.

FIELDS
The ideas of knowledge sharing, adaptation and transfer depicted in the outlined desiderata have been studied previously in machine learning and developed in related fields. We will describe each of them briefly and highlight the main differences with continual learning (see also Figure 5). Multi Task Learning. Multi-Task Learning (MTL) considers learning multiple related tasks simultaneously using a set or subset of shared parameters. It aims for a better generalization and a reduced overfitting using shared knowledge extracted from the related tasks. We refer to [21] for a survey on the topic. Multi-task learning performs offline training on all tasks simultaneously and doesn't involve any adaptation after the model has been deployed, as opposed to continual learning. Transfer Learning. Transfer learning aims to aid the learning process of a given task (the target) by exploiting knowledge acquired from another task or domain (the source). Transfer learning is mainly concerned with the forward transfer desiderata of continual learning. However, it doesn't involve any continuous adaptation after learning the target task. Moreover, the performance on the source task(s) is not taken into account during transfer learning. Domain Adaptation. Domain adaptation is a sub-field of transfer learning where the source and target tasks are the same but drawn from different input domains. The target domain data is unlabelled (or has only few labels) and the goal is to adapt a model trained on the source domain to perform well on the target domain. In other words, it relaxes the classical machine learning assumption of having training and test data drawn from the same distribution [81]. As mentioned above, as for transfer learning, domain adaptation is unidirectional and doesn't involve any accumulation of knowledge [6]. Learning to Learn (Meta Learning). The old definition of learning to learn was referring to the concept of improving the learning behaviour of a model with training experience. However, more recently, the common interpretation is the ability for a faster adaptation on a task with few examples given a large number of training tasks. While these ideas seem quite close to continual learning, meta learning still follows the same assumption of offline training but with data being randomly drawn from a task training distribution and test data being tasks with few examples. Hence, it is not capable, alone, of preventing forgetting on those previous tasks. Online Learning. In traditional offline learning, the entire training data has to be made available prior to learning the task. On the contrary, online learning studies algorithms that learn to optimize predictive models over a stream of data instances sequentially. We refer to [82], [83] for surveys on the topic. Note that online learning assumes an i.i.d data sampling procedure and considers a single task/domain, which sets it apart from continual learning. Open World Learning. Open world learning [84], [85] deals with the problem of detecting new classes at test time, hence avoiding wrong assignments to known classes. When those new classes are then integrated in the model, it meets the problem of incremental learning. As such, open world learning can be seen as a sub task of continual learning.

CONCLUSION
In this work we scrutinized recent state-of-the-art continual learning methods, confining the study to task-incremental classification with a multi-head setup, as mainly applied in literature. Within these outlines we proposed a constructive taxonomy and tested each of the compared methods in an attempt to answer several critical questions in the field. In order to fulfill the urge for a fair comparison compliant with the continual learning paradigm, we proposed a novel continual hyperparameter framework which dynamically defines forgetting related hyperparameters, i.e. the stabilityplasticity trade-off is determined in a continual fashion with only the current task data available. The overview of our experimental findings in Table 9 shed an empirical light on which methods perform best, supplemented by recommendations for model capacity and the use of dropout or weight decay. The experiments extend to two datasets, namely Tiny Imagenet and iNaturalist, from which the latter resembles a real-world dataset to challenge methods with an unbalanced data distribution and highly varying tasks. On top, we addressed the influence of task-ordering and found it minimally influential towards the general trends of the compared continual learning methods.
Although the state-of-the-art made some significant progress to tackle catastrophic forgetting in neural networks, the majority of results originates from a highly confined setup, leaving numerous opportunities for further research to reach beyond classification, multi-head evaluation and a task-incremental setup.
We set the finetuning accuracy drop margin to p = 0.2 and decaying factor α = 0.5, unless mentioned otherwise.

A.3 Methods Setup
General setup for all methods: • Stochastic Gradient Descen with a momentum of 0.9.
• Max 70 training epochs with early stopping and annealing of the learning rate: after 5 unimproved iterations of the validation accuracy the learning rate decays with factor 10, after 10 unimproved iterations training terminates.
• All methods use batch size 200, except PackNet uses batch size 32 as defined in its original work.

•
The baselines and PackNet start from scratch, other methods continue from the same model trained for the first task.
• All softmax temperatures for knowledge distillation are set to 2.
We observed in our experiments that the initial hyperparameter values of the methods consistently decay to a certain order of magnitude. To avoid overhead, we start all methods with the observed upper bound, with an additional inverse decay step as margin.
Replay-based setup. GEM doesn't specify a memory management policy and merely divides an equal portion of memory to each of the tasks. The initial value for the forgetting-related hyperparameter is set to 1.
In our task-incremental setup, iCARL fully exploits the total available exemplar memory M , and incrementally divides the capacity equally over all the seen tasks. Similar to LwF, the initial knowledge distillation strength is 10. Regularization-based setup. Initial regularization strength based on consistently observed decays in the framework: EWC 400, SI 400, MAS 3. LwF and EBLL both start with the knowledge distillation strengths set to 10, EBLL loss w.r.t. the code is initialized with 1. EBLL hyperparameters for the autoencoder are determined in a gridsearch with code dimension {100, 300} for Tiny Imagenet and {200, 500} for iNaturalist, both with reconstruction regularization strength in {0.01, 0.001}, with learning rate 0.01, and for 50 epochs.
For IMM on Tiny Imagenet dataset we observed no clear influence of the L2 transfer regularization strength (for values 10e −2 , 10e −3 , 10e −4 ), and therefore executed all experiments with a fixed value of 10e −2 instead. In order to merge the models, all task models are equally weighted, as there are no indications in how the values should be determined in [29].
Parameter isolation-based setup. For PackNet the initial value of the forgetting-related hyperparameter amounts to 90% pruning per layer, decayed with factor 0.9 if not satisfying finetuning threshold.

B.1 Synaptic Intelligence (SI): Overfitting and Regularization
The observed overfitting for SI on Tiny Imagenet is illustrated in Table 10 for the BASE model. Training accuracy without regularization tends to reach 100% for all tasks, with the validation phase merely attaining about half of this accuracy. This results in an average discrepancy between training and validation accuracy of 48.8%, indicate significant overfitting on the training data. However, this discrepancy can be greatly reduced by the use of regularization, such as dropout (20.6%) or weight decay (30.8%).
Without regularization the overfitting results of SI are very similar to the finetuning baseline (48.8% and 48.0% discrepancy). However, finetuning does not clearly benefit from regularization, retaining training accuracies near 100%. Observing different weight decay strengths in Table 11, severely increasing the standard weight decay strength in our experiments (λ = 0.0001) by up to a factor 100 merely reduces average training accuracy to 85.5%, leaving a significant discrepancy with the validation accuracy of 37.8%.
Continual learning methods EWC and LwF in the regularization-based family of SI, inherently reduce overfitting (resp. 38.4% and 28.9% discrepancy) without any use of the traditional weight decay and dropout regularizers.

B.2 Extra Replay-based Experiments: Epoch Sensitivity
In initial experiments with 70 epochs we observed inferior performance of GEM w.r.t. iCARL and the rehearsal baselines R-PM and R-PM. In the original GEM setup [50] only a single epoch is assumed for each task, while in our experiments methods get the advantage of multiple epochs. This might impose a disadvantage when comparing GEM to the other methods, wherefore we conduct this extra experiment to attain a fair comparison (Table 12). As in the other experiments in this work, we apply early stopping after 10 non-improved epochs (indicated with * ), which makes epochs 20 to 70 upper bounds of the actual trained epochs. The GEM setup with only 5 epochs shows superior average accuracies, indicating a better trade-off between the online  Accuracy  T1  T2  T3  T4  T5  T6  T7  T8  T9   setup GEM was designed for and exploiting several epochs to optimize for the current task. Therefore, we conduct all GEM experiments in this work with 5 epochs. The other replay-based method iCARL, on the other hand, mainly benefits from more epochs, stagnating at more than 10 epochs, with only a small increase in average forgetting. TABLE 12: GEM and iCARL sensitivity analysis on the amount of epochs for average accuracy (average forgetting) on the BASE network for Tiny Imagenet. Both exemplar memories of size 4.5k and 9k are considered using the standard setup of our experiments, i.e. early stopping after 10 non-improved epochs (indicated with * ).