{'fraction_non_alphanumeric': 0.03016940037644528, 'fraction_numerical': 0.013834364076364614, 'mean_word_length': 4.704939407884645, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research

We introduce the Never Ending VIsual-classification Stream (NEVIS'22), a benchmark consisting of a stream of over 100 visual classification tasks, sorted chronologically and extracted from papers sampled uniformly from computer vision proceedings spanning the last three decades. The resulting stream reflects what the research community thought was meaningful at any point in time. Despite being limited to classification, the resulting stream has a rich diversity of tasks from OCR, to texture analysis, crowd counting, scene recognition, and so forth. The diversity is also reflected in the wide range of dataset sizes, spanning over four orders of magnitude. Overall, NEVIS'22 poses an unprecedented challenge for current sequential learning approaches due to the scale and diversity of tasks, yet with a low entry barrier as it is limited to a single modality and each task is a classical supervised learning problem. Moreover, we provide a reference implementation including strong baselines and a simple evaluation protocol to compare methods in terms of their trade-off between accuracy and compute. We hope that NEVIS'22 can be useful to researchers working on continual learning, meta-learning, AutoML and more generally sequential learning, and help these communities join forces towards more robust and efficient models that efficiently adapt to a never ending stream of data. Implementations have been made available at https://github.com/deepmind/dm_nevis.


Introduction
The machine learning community has focused extensively on the stationary batch setting, for which there exists a static and unchanging data distribution used to sample fixed training and test sets from (Vapnik, 1998). This has enabled the rigorous evaluation of learning systems, and driven unprecedented progress over the last four decades, across a wide range of domains (e.g. LeCun et al., 2015;Jumper et al., 2021;Brown et al., 2020;Alayrac et al., 2022). Throughout this journey, researchers have spent a considerable amount of time and compute developing algorithmic and architectural improvements, adapting methods to new application domains, and developing insights into how to transfer their knowledge and know-how to new and more challenging settings.
In this work, we consider whether this traditional procedure for developing models can be made more efficient and further automated. This is important to enable our community to make faster progress and to further scale up our models, a process that has been so critical to the recent success of the field. Taking the perspective that the machine learning community itself can be seen as an agent interacting with data to produce artifacts, we explore approaches that can robustly and automatically develop models for new tasks, while accruing knowledge over time to further accelerate learning in the future.
Unfortunately, there is no consensus on how to measure the speed of adaptation or knowledge accrual (Parisi et al., 2019;Hadsell et al., 2020;Mundt et al., 2022). Moreover, current benchmarks often focus on other problems, such as catastrophic forgetting (French, 1999), are too small, or lack sufficient diversity (Chaudhry, 2020). This motivates us to introduce Nevis'22, a challenging stream which comprises 106 tasks, all representing publicly available datasets from the last 30 years of computer vision research. By construction, Nevis'22 tracks what the vision community has deemed interesting over time, since tasks are sorted according to the year in which they appeared in publications. Over time, new and more challenging domains are considered, datasets get larger, and overall there are more opportunities to transfer knowledge from an ever growing set of related tasks.
Since each task in isolation is very well understood, the only research question remaining is how to robustly and effectively adapt to tasks over time. As an indirect measure of whether a system is capable of accruing knowledge over time, we assess performance not just in terms of final error rate, but also compute needed to reach such performance. The assumption is that, if a method can transfer knowledge from related past tasks, then it can quickly learn the next task using less compute.
We believe Nevis'22 should appeal to and challenge researchers in several communities. It should attract researchers in continual learning (Ring, 1994;Thrun, 1994) because the stream is non-stationary. Some of the tasks are repeated over time, providing a natural opportunity to measure forgetting and forward transfer (Lopez-Paz and Ranzato, 2017;Nevis'22 Chaudhry et al., 2018;Schwarz et al., 2018;Hadsell et al., 2020;Parisi et al., 2019). It should empower researchers in meta-learning (Thrun and Pratt, 1998) because there is a rich structure across tasks, which should enable the study of learning-to-learn. Finally, it should be useful to researchers in AutoML (Thornton et al., 2013) as each task has to be solved in a black box manner, without humans in the loop. Since our metrics include the compute used during hyper-parameter search, Nevis'22 incentivizes the development of efficient approaches for algorithm, architecture and hyper-parameter search. For the very same reasons, however, Nevis'22 also constitutes a challenge, as it requires tools from each of these communities. Moreover, Nevis'22 is the first benchmark simulating supervised never-ending learning at this scale, and with such rich and diverse set of realistic tasks. Nevis'22 is accompanied by code to reproduce the stream, the training and evaluation protocols, and representative baselines we have considered.

Related Work
In this section we put our work in the broader context of the literature with a twofold goal. First, we relate the Nevis'22 learning setting with existing learning frameworks such as continual learning, meta-learning and AutoML. Second, we contrast Nevis'22 with existing benchmarks and highlight its unique features.
Continual (or Lifelong) Learning studies the question of learning under a non-stationary data distribution (Silver et al., 2013;Chen and Liu, 2018;Hadsell et al., 2020;Parisi et al., 2019). It typically assumes a series of tasks. The objective is to learn sequentially while achieving a list of desiderata ranging from avoiding catastrophic forgetting (McCloskey and Cohen, 1989), to leveraging forward or backward transfer (Lopez-Paz and Ranzato, 2017). Additional restrictions are typically considered, such as preventing access to previous data, limiting or accounting for the use of compute or memory. Given the multitude of potential desiderata and trade-offs of interest (Hadsell et al., 2020), the literature has flourished with a considerable number of specialized benchmarks, each targeting a different scenario. In this work, we make the following central assumption: The learner cannot access data from novel future tasks. In particular, accessing past data (or even past models) is permitted, although the compute cost of doing so will be taken into account in the final reporting. Our rationale is that, in modern applications of machine learning, memory is cheap relative to compute and time. We therefore focus on leveraging forward transfer for future tasks of interest to the community, rather than avoiding catastrophic forgetting or data storage limitations.
In continual Reinforcement Learning (RL) (Ring, 1994;Khetarpal et al., 2020), the nonstationarity is either imposed by changing environments, or it emerges from the interaction with the environment. While RL provides a natural test-bed for continual learning, it also makes it difficult to separate the challenges raised by exploration and learning with sparse rewards from the core continual learning problem of accruing knowledge over time. Attempts to studying the question of forward transfer have nonetheless been made (Wolczyk et al., 2021). Similarly in the language domain, there have been studies on continuous training of language models with related benchmarks defined on sorted streams of text (Liska et al., 2022;Jang et al., 2022). Although very interesting, the lack of clear task structure or distinctions makes it difficult to assess when new concepts are introduced in data streams, and when the system is expected to learn new capabilities or skills.
In vision research, most existing benchmarks focus on measuring catastrophic forgetting and are derived from popular datasets such as MNIST (LeCun et al., 1998b), CIFAR-10 (Krizhevsky, 2009b), ImageNet (Deng et al., 2009b) or Omniglot (Lake and Tenenbaum, 2015), whereby a stream is created by partitioning the data into disjoint subsets. This construction however greatly limits the diversity of the resulting stream. There are however exceptions. The Core50 dataset (Lomonaco and Maltoni, 2017) specifically collected realistic images of objects under different poses, to test continual learning capabilities in a setting most relevant for robotics. The CLEAR benchmark (Lin et al., 2021) looks at temporal evolution of a set of visual concepts. CLOC (Cai et al., 2021) is a geo-localization task with a large collection of chronologically ordered images. Once again, these benchmarks target the setting of a single non-stationary task, as opposed to a sequence of a diverse set of tasks. Forward transfer has become a more prominent goal of CL through benchmarks such as CTrL (Veniat et al., 2021), which is however limited in its scale and diversity, because it is entirely derived from just a handful of small datasets.
Meta-learning assumes access to a distribution of tasks (Thrun and Pratt, 2012;Finn, 2018;Hospedales et al., 2022). The goal is to learn from the observed tasks a mechanism that allows efficient learning on hold-out tasks from the same distribution. Most popular benchmarks like VTAB (Zhai et al., 2019) and Meta-Dataset (Triantafillou et al., 2020) focus on few-shot learning, while Nevis'22 considers a variety of dataset sizes (including some with a handful of examples) and goes beyond a few steps of adaptation to characterize performance efficiency trade-offs. For example, Nevis'22 accounts for compute in addition to error rate.
Much of the field of AutoML is focused on tabular tasks and shallow predictors. Benchmarks are often derived from the OpenML platform (Vanschoren et al., 2013). The major limitation of current AutoML approaches is their computational cost, since one evaluation requires a full training run with a particular hyper-parameter setting. For instance, naïve neural architecture search would be too costly if applied within the inner loop of Nevis'22, as there are over 100 tasks. More recently, there has been excitement around lifelong Au-toML (Feurer et al., 2015;Lindauer and Hutter, 2018), but again, evaluation has been limited to very few and very similar tasks. Nevis'22 gives a more natural playground to explore ways to transfer knowledge at the level of the meta-learner thanks to shared structure across tasks. Moreover, Nevis'22 sets incentives towards the development of more efficient AutoML methods because the evaluation accounts for compute spent during hyper-parameter search and prizes more parsimonious meta-learners.
Transfer learning (Pan and Yang, 2010;Bengio, 2012;Weiss et al., 2016;Tan et al., 2018;Zhuang et al., 2020) is a general and well studied paradigm for addressing the problem of leveraging one or a few related source tasks to improve the learning of a known target task. Techniques such as self-supervised learning (? Grill et al., 2020) and large-scale pretraining (Jia et al., 2021;Radford et al., 2021) are recent examples of successful approaches in computer vision, see (Jaiswal et al., 2020) for a survey. The sequential aspect of data acquisition is often neglected in the transfer learning setting. Nevis'22 provides a good test bed for testing transfer learning ideas at scale, and in a more realistic setting, where the system needs to keep adapting over time, as opposed to only once. In this work, we do consider several variants of pretraining among our baselines, and demonstrate that enabling continuous adaptation further improves their performance.

Stream Statistics
Nevis'22 is a stream composed of 106 image classification datasets totalling approximately 8 million images. There is a large diversity in data. For instance, the input resolution goes from 3 × 3 all the way to 2000 × 3000. Some datasets have fixed resolution, while for others each example has its own resolution.
The number of examples in each dataset also varies considerably, spanning four orders of magnitude, as it can be seen in Figure 1 left panel. Dataset size tend to increase over 1   5   10   50   100   1992  1998  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021 Datasets per year Datasets so far  Right: Upper-triangular transfer matrix on a subset of tasks extracted from Nevis'22. The figure shows at position (i, j) the advantage of pretraining on task i before finetuning on task j, compared to learning task j from scratch. The upper triangular section shows only transfer from tasks that had already occurred in the sequence. Shade of blue indicate positive transfer (i.e. pretraining was useful), while shade of red indicate negative transfer. We notice that there exists tasks that are good for pretraining, leading to positive transfer to most future task (e.g., green rectangles which correspond to ImageNet and SUN397), tasks that can transfer well from any other tasks (e.g., the blue rectangle corresponding to the Stanford Cars dataset) and tasks that do not transfer well from any other task (e.g., the red rectangle corresponding to the Mall dataset Designer chooses M's hyper-parameters λ M .

10:
Initialize meta-learner state s 0 based on λ M .

11:
for T i ∈ S Tr do 12: s j ← M updates its state using D tr j , s j−1 and λ M .

26:
e j ← error rate of P j on D ts j 27: end for 28: E(S Ts ) = n+m i=n+1 e i . 29: cFLOP(S Tr + S Ts ) = n+m i=1 FLOP i . 30: Return and report E(S Ts ), cFLOP(S Tr + S Ts ) The left plot of Figure 2 shows the major families of domain and their evolution over time. There are interesting patterns of non-stationarity, for instance with some domains appearing throughout the stream (e.g., object), while others being popular only in short time windows (e.g., satellite). On the right side of Figure 2 we can also see how datasets relate to each other. The transfer matrix shows interesting structure, with both positive and negative transfer.
The scale, diversity and agnostic selection used in the construction process are the defining elements of Nevis'22, compared to existing benchmarks that might satisfy some of these desirable axes but, to the best of our knowledge, not all of them.

Learning & Evaluation Protocol
We recall that the Nevis'22 benchmark operates on a sequential stream of diverse tasks; and that learners are evaluated on their ability to efficiently generalize what they have learned early on in the stream to tasks that appear later. An important aspect of this setting is that learners are not permitted to use future observations to influence their behavior on a given task. Most notably, this condition is also intended to apply to the selection of the hyper-parameters used by learners. In particular, learners are not permitted to select hyperparameters based on metrics computed on runs over the full stream, since this is equivalent to using information about future tasks in the stream to influence the present. This decision has been made to encourage algorithm designers to build robust learning systems that can truly adapt to changing distributions automatically, rather than through careful human-driven hyper-parameter tuning. These requirements necessitate a rigorous evaluation protocol that supports both the iteration and development of learners (including their meta-learning components), and also meaningful comparisons of learner implementations once they have been tuned. In this section, we outline the strategies we have adopted to support these requirements.
In all generality, we assume that there exists a "meta-learner" M that is in charge of instantiating a predictor P i for the i-th task. M is responsible for determining any hyperparameters (such as learning rate and label smoothing values) needed to construct P i , and also for possibly leveraging the results of previously observed tasks, if doing so could enable the learner to more efficiently solve the current task. For instance, M could initialize the parameters of P i from the parameters of a network trained on a related previous task. Furthermore, M itself might have hyper-parameters; for instance, they could be the range of values considered by random hyper-parameter search, or the choice of transfer learning method. How shall we cross-validate the meta-learner M and each predictor P i ? How should we account for the compute used by both M and P i ? To answer these questions, we propose the training and evaluation protocol described in Algorithm 1.
Since we are interested in assessing generalization to future tasks, we divide Nevis'22 into two sub-streams (line 2 and 3): the "meta-train stream", denoted by S Tr , and "meta-test stream", denoted by S Ts . S Tr comprises the tasks contained in the first 27 years for a total of 79 tasks, S Ts contains the 27 tasks from the last 3 years. Moreover, each task T i consists of three datasets, namely training D tr i , validation D val i , and test D ts i . Next we explain how these stream and datasets splits are used.
During the meta-training phase (lines 8 to 19), a designer (effectively, a meta meta-learner) might run the meta-learner M multiple times over S Tr to tune M 's hyper-parameters λ M ; for instance, this might include choosing neural network architectures, optimizers, dataaugmentation strategies and intialization parameters. It is up to the designer to decide which configurations to try next (line 9), and when to stop the search (line 19). At every such iteration, M sweeps over the tasks (or any subset thereof) of S Tr . It first extracts the task specific training and validation set (line 12). Then it uses the training set D tr i to produce a predictor P i for the i-th task. This process typically involves some form of task-level search over P i 's hyper-parameters. In order to support this, Nevis'22 provides a default decomposition of D tr i into two sets, one used for actual training of P i and the other used for task-level cross validation. However, it is up to the meta-learner to decide whether to use this or other ways to partition D tr i to better support P i 's hyper-parameter search. The result of this step is not only P i , a predictor for task i, but also the total number of floating point operations used during this training and hyper-parameter search process, denoted by FLOP i .
Notice that M uses a certain configuration of its own hyper-parameters λ M and an internal state s i−1 to find P i . Examples of λ M could be the range of learning rate values used during the actual random search of P i 's learning rate. The state s i instead is what represents the knowledge accrued up to the i-th step. This can be an empty set if M instantiates independently learners to tasks. It could also consists of the set of parameters used by P j for j < i, supporting various kinds of finetuning strategies from models trained on previously encountered tasks. This state is updated by M in line 14. In the previous example, this merely consists of adding an additional parameter vector to a pre-existing set of parameter vectors, one for each previous task. Finally, P i is evaluated on D val i (line 15) in terms of error, as follows: where acc i is the average accuracy on the task i; mAP i is the mean average precision on the task i. Ultimately, these task level metrics, (e i , FLOP i ) are aggregated at the stream level via averaging or sum. Given a stream S with K tasks, we define: where we denote the average error rate with E and the cumulative floating point operations with cFLOP. By varying hyper-parameters such as the number of gradient steps used, the size of the architecture or the number of trials used during hyper-parameter search, M will strike different trade-offs between average error rate and total compute. The search process over λ M will aim at pushing the Pareto front of these points. In our work (as it is standard practice), the process of searching over λ M requires a human in the loop to determine the best meta-learner's configuration to explore next. Note that one may choose to train on all tasks at once, or to pick only the largest one. Yet the meta-training loop is still used to find λ M . Researchers are however free to explore other ways to cross-validate M , for instance by introducing a meta-validation stream. In this work, we opted for simplicity, but we leave open the question of how to best cross-validate M and hope that Nevis'22 can be useful also to address this research question.
Once λ M is determined, we evaluate M (lines 21 to 30). The process follows the same steps discussed previously, with a few exceptions. First, M must do a pass over the entire stream, S = S Tr + S Ts . Notice that at step i M cannot access any task T j with j > i; in particular, since there is no outer loop over M 's hyper-parameters, tasks in S Ts are observed once and only once in sequence. During training on task T i , P i can do several epochs over D tr i , but M cannot revisit T i twice because otherwise we would not be able to assess generalization to unseen tasks of S Ts . The second difference is that P i is evaluated on D ts i . Finally, the average error rate is calculated only using tasks belonging to S Ts . The rationale is to remove tasks over which we did any kind of cross-validation to prevent overfitting. However, we still account for the cost of development of M by including the FLOPS used while learning on S Tr (although for the sake of simplicity, we do not consider how many times S Tr was visited during the meta-training phase). 2 Ultimately a given method will yield a tuple, (E, cFLOP), where E is the average error over S Ts and cFLOP is the cumulative compute over the whole stream S. The best methods will be the ones at the Pareto front, delivering the lowest E for a given total compute budget. A clever method could early stop on tasks whenever its gains in terms of error rate are too marginal, and use the saved compute on future tasks that do require more compute to achieve lower error rate.

Codebase
An open source implementation of the Nevis'22 benchmark has been made available at https://github.com/deepmind/dm_nevis. This repository implements the training and evaluation protocol described in this section, downloads and prepares the data, and enables other researchers to reproduce the main results we obtained with our baselines.
The implementation has been designed to be modular, compartmentalizing data processing, handling of the stream, learners and metrics. In particular, no knowledge of the particular stream or metrics is needed in order to implement a new learner. Moreover, the learner interface is minimal, requiring only the implementation of functions that initialize, train, and compute predictions. This may be implemented using any appropriate Python based machine learning library such as JAX or PyTorch.

Experiments
In this section we describe the baseline approaches and results we obtained on the Nevis'22 stream. In Sec. 5.2 we conclude with ablations showing what factors contribute to the performance of current approaches, and how the diversity and scale of Nevis'22 enable better assessment of life-long learners.
Preprocessing and Data Augmentation The tasks in Nevis'22 have images spanning a wide range of different resolutions. Even within a task the resolution may vary from image to image. For our baselines, we adopted a two-part strategy which favored simplicity over performance. During training, images are randomly resized and cropped to a fixed resolution of 64 × 64 pixels, and then left-right flipped with probability 0.5. During evaluation, we take the central square crop of min(w, h) × min(w, h), where w is the width of the image and h is its height, and resize it to 64 × 64 pixels with no additional augmentation.
2. Using a separate set of test tasks, S Ts , and test splits for the datasets, D ts , might seem overly zealous.
This was however useful to assess potential overfitting to S Tr , and for some of our ablations which analyzed the full stream S = S Tr + S Ts . For instance, the ablation by domain of Sec. 5.2 required the analysis on the full stream.  Table 1: Differentiating baselines across three axes: 1) How the parameter's of the predictor are initialized, 2) What data is used to train the predictor and 3) How hyper-parameter search of each predictor is conducted.
Note that this strategy is clearly sub-optimal for tasks involving fine details, such as crowd counting. Nonetheless, this choice simplifies the design of architectures. More details can be found in the sensitivity analysis of Fig. 8.
Architecture Unless otherwise stated, our baselines use the ResNet34 backbone tailored for low resolution images (He et al., 2016, Sec. 4.2), since all images are resized to 64 × 64 pixels. Each task is assigned a task-specific head mapping backbone features to output logits.
Meta-Learning Each of our baselines includes a meta-learner that selects the task-specific hyper-parameters used during training of the actual predictor. For the sake of speeding up tuning of the meta-learner, we performed stream-level cross-validation on a subset of S Tr containing 24 tasks only. This is what we call short. On this stream subset, we tuned the choice of the architecture and the ranges used by the hyper-parameter search. In particular, we identified a set of hyper-parameters that are robust enough to be kept fixed throughout the learning experience on the stream: 1) cosine learning rate scheduling with warm-up phase proportional to the number of gradients updates in conjunction with SGD with Nesterov momentum (set to 0.9, with a weight decay of 0.0001), 2) data augmentation consisting of random cropping and flipping, and 3) a heuristic to set the batch size as a function of the dataset size, where B is the maximum batch size, D is the dataset size, and p is a constant set to 0.0025 in our experiments.
Learning All baselines we have considered in our empirical evaluation yield one predictor per task, without any parameter sharing across tasks. This is the simplest setting which corresponds to the most widely used design choice in practice. There are three independent factors that are used to create each baselines: 1) The choice of initialization, 2) The choice of which data is used to train a given predictor, and 3) The choice of the algorithm used to search in hyper-parameter space. The particular combination of three factors determines the meta-learner M described in Sec. 4. In this work we have considered the six most widely used combinations of these three factors, as summarized in Tab. 1, namely: 1) Independent (Indep): The meta-learner initializes the parameters of the predictor for each task at random, trains using data for the current task only, and searches over hyperparameters using random search. This is the the most naïve baseline. It is the reference that any other method should beat, as it is the simplest method that does not support any form of transfer learning (the state of the meta-learner s k is null for all k).

2) Finetuning (FT):
The meta-learner initializes the parameters of the predictor from the parameters of a network trained on a previous task, trains using data for the current task only, and searches over hyper-parameters using random search. In this case the state of the meta-learner s k consists of the union of the model parameters trained on all tasks observed so far, from 1 till k − 1. Knowledge for this learner is the set of model parameters, which correspond to one expert per observed task.
We have considered various criteria to select the previous task from which to finetune: 1) temporal proximity by taking the most recent task (FT-prev), 2) task relatedness by taking the most related past task. For the latter, we have been using as a proxy of task relatedness the performance of a k-nearest neighbor classifier in the feature space produced by the previous predictors, using as training and validation data a small subset of D tr (up to 10000 and 5000 images, respectively). In other words, we use the previous predictors as feature extractors to encode data from the current task, similarly to Veniat et al. (2021). We have two versions of this. An offline or static version (FT-s) where the features are computed using pretrained independent predictors as in Indep above, and a dynamic version (FT-d) where features are computed online using the actual predictors trained so far (which could have been finetuned themselves on other tasks). Fig. 17 in Appendix shows an actual example of such learned chain of finetuned models.

3) Multitasking (MT):
The meta-learner initializes the parameters of the predictor using the same approach as in FT-d, trains using data of both the current task and some previous tasks, and searches over hyper-parameters using random search. Both the selection criterion for what previous task to take for parameter initialization and what previous (auxiliary) tasks to take for additional training data are based on task relatedness using the same k-nearest neighbor classifier score described in the FT-d baseline above. Unlike FT, the multitask baseline can take k ≥ 1 most related auxiliary tasks for additional training data. During training the network is trained in a multitask fashion, weighing the losses of the auxiliary tasks by a single scalar hyper-parameter λ. This hyper-parameter is subject to hyperparameter search by the meta-learner, and it sets the relative importance of the auxiliary tasks against the current task. In a single training step, the gradients are accumulated across the mini-batches of current and k auxiliary tasks. At test time, the classification heads of the auxiliary tasks are disregarded. With reference to Algorithm 1, the state s k of the meta-learner M consists of the set of datasets and predictors trained so far. In this case, Nevis'22 knowledge is represented both as the set of model parameters of already observed tasks, as well as the set of datasets encountered so far.

4) Pre-training (PT):
The meta-learner initializes the parameters from a pretrained model, trains using data for the current task only, and searches over hyper-parameters using random search. This is a special case of FT, where all task predictors are finetuned from exactly the same pretrained model. Note that there is no form of knowledge accumulation for this baseline.
We have considered two pretrained models from which to finetune: 1) A ResNet34 pretrained on ImageNet by supervised learning (PT-ISup) and 2) A Normalizer-Free network (NFNet-F0) (Brock et al., 2021) pretrained on two very large external datasets: ALIGN and LTIP (Alayrac et al., 2022) using CLIP (PT-ext) (Radford et al., 2021). In this case, the state s k of the meta-learner M is constant over time, as it merely contains the fixed set of pre-trained parameters.

5) Fine-Tuning with Pre-training (PT + FT):
This variant is exactly the same as FT-d, except that the set of parameters avaiable for finetuning includes not only the parameters of networks trained on previous tasks but also the pretrained model used by PT-ext.

6) Bayesian Hyper-Parameter Optimization (BHPO):
The meta-learner initializes the parameters of the predictor at random, trains using data for the current task only, and searches over hyper-parameters using a Gaussian Process to estimate the function value (expected loss at convergence). Instead of running a search in parallel over a set of hyperparameter configurations like in random search, BHPO runs the search in sequence, using the Upper Confidence Bound acquisition function (Srinivas et al., 2010) provided by Google Vizier (Golovin et al., 2017) to select the next configuration to search over.
Experimental Setup. The search space of Indep, FT, PT and PT+FT consists of initial learning rate and label smoothing, which means that for each task we search over the values of these two hyper-parameters. MT adds to the search space also λ, the weight on the auxiliary tasks. BHPO adds five additional hyper-parameters compared to Indep, which include the choice of learning rate schedule (cosine learning rate, piece-wise constant decay), batch size, choice of architecture (VGG, ResNet34), choice of the two data augmentations (random cropping and flipping).
To vary the compute budget and study the Pareto front of average error rate versus compute, and unless otherwise stated, we vary the number of hyper-parameter configurations over which the meta-learner searches over at each task (ranging from 2 to 32), and the total number of weight updates (ranging from 10000 to 100000). Note that different combinations of number of updates and trials per task can lead to the same computational budget, but different performance.
In the following section we will report results for a total compute budget in the range between 10 18 and 10 21 , aiming at models that achieve competitive final performance (for the chosen 64 × 64 resolution) on landmark datasets. For instance, the basic Indep baseline achieves 4.2% error rate on SVHN, 0.7% on MNIST, 6.8% on CIFAR10 and 34% on ImageNet. In Appendix B, we will report results using smaller computational budgets (and hence worse final error rate) for users that have more limited computational resources at their disposal.

Findings
In this section we report the main results we obtained by applying the previously described baselines to Nevis'22. A priori, common wisdom would suggest that methods that perform multi-task using all available data would perform the best, while methods that rely on sequential finetuning to grossly underperform (Ash and Adams, 2020). We would also expect methods based on pretraining to perform the best, but to gain little if anything by combining with other forms of transfer learning as their representations are potentially already general enough. We would also expect Indep to work more poorly on smaller datasets, and that no method is best across the entire spectrum of compute budget. Our results show that not all these intuitions find empirical support in Nevis'22.
We start by reporting the pareto fronts of average error rate versus compute in Fig. 3. In this figure, we show a selected baseline from each of the families described in Sec. 5. More results are reported in the Appendix (Sec. D).
1. We observe that all methods perform better than learning from scratch (Indep). This shows that indeed there is rich shared structure across tasks in Nevis'22 which supports beneficial transfer. Note that a run with 3 different seeds of the independent baseline (at the second highest cost) yields a standard deviation of 0.003, suggesting that performance gaps between baselines are significant except for FT-d and MT at the highest computational budget.
2. BHPO reduces the error compared to Indep, but at the cost of more compute. Overall, BHPO does not improve the pareto front. Given that hyper-parameter search substantially impacts cFLOP (typically by a factor of ten or more), we surmise that there could be more clever and efficient ways to explore the hyper-parameter space which could lead to a better performance-compute trade-off.
3. There is a rather large gap of about 5% absolute error between the best and the worst method, namely Indep and FT-d if we restrict to training data from Nevis'22. Given the simplicity of FT-d in terms of its approach to transfer learning and its naïve use of compute (random hyper-parameter search and no early stopping), we would expect future approaches to further reduce both compute and error rate.
4. The performance of FT-d suggests that sequential finetuning, even when applied on chains longer than 10 steps, as it can be appreciated in Fig. 17 of Appendix, works remarkably well.

5.
The choice of what to finetune from matters, as FT-p is significantly worse than FT-d as shown in the Appendix (Sec. D.1). Therefore, there could be improvements by using better approaches to estimate task relatedness.
6. MT works well but the additional compute spent on relearning representations on past tasks does not compensate for the improvements in generalization. Overall, FT-d strikes a better trade-off than MT.
7. As expected, pretraining improves the performance significantly. Starting with a pretrained network (PT-ext) leads to a significantly lower average error. Notice that PT-ext leverages both a much larger amount of external data and a more powerful architecture. In the Appendix D.2 we also show that pretraining the same architecture used for the other baselines on ImageNet (PT-ISup) reduces the gap between Indep and the best performing baselines (FT-d and MT).
8. More surprisingly, using FT-d with a pretrained network (PT-ext + FT-d) lowers the average error further, see bottom plot of Fig. 3. This demonstrates that leveraging the structure in the stream can improve the already general representations that the pretrained model provides, which opens a new avenue of research on large-scale models that continuously adapt over time. More details on the structure that FT-d discovers starting from the pretrained model are provided in the Appendix D.1.2.
In order to better understand how methods fare on each task, we also present a regretlike plot in Fig. 4. This shows the cumulative error over time relative to Indep, picking hyper-parameters such that all methods use roughly the same amount of compute. When the curve is horizontal it means that a method performs comparably to Indep. When the slope is negative it means that it outperforms Indep, and vice versa. We observe that no method, including PT variants, transfers well to datasets in the OCR and medical domains.
In particular, all regret curves are nearly horizontal over the last nine datasets, which are mostly datasets in a new x-ray domain (Covid-19 related classification tasks from 2021). This shows a clear limitation of current approaches which are not yet capable to a) transfer well to minor domains, and b) accrue knowledge over time, as many of these 9 last tasks are closely related to each other. Finally, the regret plot over the entire stream in the bottom of Fig. 4 shows that overall methods provide a linear improvement over the Indep baseline. FT-d starts flat as expected (since initially there is nothing to transfer) and later on exhibits a linear gain. However, no method improves over time in the second part of the stream. In other words, none of the baselines we tried was capable to become more accurate as it receives more data and it makes new learning experiences. While this is expected for PT which cannot accrue knowledge by construction, we surmise there could be a more clever variant of FT that actually improves over time.
Overall, these findings indicate that Nevis'22 is a good playground for research in never-ending learning. Methods that transfer do significantly better than methods that do not, and yet there seems to be ample room for improvement over the current set of baselines.  Unlike our expectations, methods based on multi-tasking did not yield better trade-offs, they might achieve a lower error rate but this does not compensate for the increase in the amount of compute. Instead, sequential FT has worked remarkably well despite the simplicity of the heuristic used to determine task relatedness. This observation still holds when starting from a large pretrained model, opening the question of how to best accumulate knowledge in foundational models. Finally, we have reported more results using BHPO in Appendix E.2. These show that current BHPO is effective only when increasing the search space. In this setting, BHPO does find better hyper-parameter configurations than random search but the gains are currently too limited relative to the additional compute required.

Ablations
In this section we further analyze Nevis'22, studying how results vary by domain, dataset size, task ordering, etc. The goal is to understand which factors affect the performance of the baselines the most, and ultimately, which unique features Nevis'22 has to offer relative to existing benchmarks.
Image Domain. In this experiment, we take baselines which have been trained on the entire stream, and instead of evaluating on all tasks of S Ts , we evaluate on all tasks of S Tr ∪ S Ts but filtering tasks by their domain. For the evaluation we use the test split of each task. We report results on 4 representative domains, namely OCR, medical, face and object. Note that in methods like FT, networks that are trained on a dataset of a certain domain, could be finetuned from a task belonging to a different domain. Moreover, results across domains are not directly comparable since each domain has its own set of tasks.
Results are shown in Figure 5. We observe a strong dependence on the domain type. The gains over Indep brought by baselines that allow transfer learning are minimal in OCR but substantial for object, for instance. Even more interestingly, the ranking of the various baselines is domain dependent, and there is no winner across all domains. For instance, PT is the best method in the medical domain but the worst in OCR. We conjecture that the reason could be that OCR has relatively large datasets, but it is perhaps the most distant domain relative to the domain used for pretraining. Notice how these insights have been enabled thanks to the richness of domains in Nevis'22.
Dataset size. We now study how performance correlates with dataset size. We use the same methodology used in the previous analysis by domain, but aggregate test results filtering by the size of the datasets. We have defined four groups of increasing size of the training set, namely datasets with a training set with fewer than 1,000 examples, datasets with a number   Fig. 6. Unsurprisingly, Indep becomes competitive on very large datasets, but surprisingly, methods that support transfer learning (e.g., FT) do not gain much over Indep on very small datasets. The gains are more significant on datasets of intermediate size.
Overall, adapting to small datasets is still a challenge for the baselines we have considered. Once again, Nevis'22 has enabled this analysis thanks to its diversity of dataset sizes.
Image resolution. With a methodology similar to the one used in the previous experiments, we again train on the full stream (using the default fixed resolution of 64 × 64 pixels), evaluate on the test split of each task but report metrics by selecting only tasks with average image resolution within a certain range. Fig. 7 shows that the error rate on datasets with smaller resolution is very low, and on those datasets methods that transfer work comparably to Indep. As the resolution increases we observe a remarkable improvement of FT, PT and MT over Indep. Finally, while we cannot directly compare results across various image resolutions (since each contain a different subset of tasks), we notice that the average error rate on datasets of larger resolution images is much bigger, suggesting that downsizing the resolution to only 64 × 64 pixels might deteriorate performance on such tasks, and that architectures that handle variable resolution images might strike better trade-offs. Alternatively, we also studied how the performance of Indep changes as we vary the choice of the input image resolution used during training and evaluation of each task. This  (512), and varying the model architecture, the number of updates and the number of hyper-parameter configurations over which we search for each task. This frontier is provided as a reference. We can see that varying the image resolution is yet another way to trade-off error rate versus compute. It is future work to assess whether varying the image resolution or other factors like the architecture size could improve the Pareto fronts.
Task ordering. In this experiment we study whether the performance of baselines is affected by the ordering of the tasks. For this purpose, we create two new variants of Nevis'22, one where we pick a different shuffling of the datasets within a year (note that in Nevis'22 this is already arbitrary), and one where we shuffle the order of the datasets across the entire stream, S Tr ∪ S Ts .
Results are showin in Fig. 9. We have found that shuffling the task ordering within a year does not affect the error rate in average, meaning that if we average across several stream variants that differ in the within year shuffling we obtain a similar error rate to the default version of Nevis'22.
Randomizing the order of the tasks over the entire stream has instead more dramatic effects. Whenever larger datasets like ImageNet are moved to earlier times, the average error rate is lower. The average error rate over 3 random orderings is 0.253, while the average error rate when shuffling withing years is 0.260 when using FT-prev that accrues knowledge over time. Recall that the standard deviation of the Indep baseline is 0.003 (also displayed on the figure). Overall this suggests that the order of the tasks does affect performance in   Baselines of random tasks are run with 5 random seeds and we report the mean and std of average test error rate.
Nevis'22, and that current baselines struggle to transfer from several smaller datasets to bigger datasets.
Other Stream Variants. In Tab. 2 we study how tasks in S Tr affect performance of FT-d on S Ts . We picked FT-d since this is a baseline whose performance on S Ts is expected to depend on what it has learned on S Tr . The average test error using the default version of S Tr is 0.273. If we remove ImageNet, the average error rate increases by 3%, which is not surprising as the network trained on ImageNet is selected for finetuning by several subsequent tasks, as shown in Fig. 17 of Appendix. Shortening S Tr by selecting only tasks belonging to the major domains or larger datasets increases the error rate slightly. FT-d does not seem to leverage well smaller datasets and minor domains. Without improving the transfer ability, this baseline could in fact strike a better trade-off by removing tasks from S Tr as shown when removing the smallest datasets (with less than 10, 000 training examples). The first 30 tasks from the stream contain tasks that many subsequent tasks are finetuned from, including ImageNet, Caltech256, Scene8, etc. Therefore, in this case removing the first 30 tasks from the stream deteriorates performance.

Forward
Transfer. An ideal never-ending learner should be able to learn faster by transferring knowledge from the past to the future. In this study we compare two fine-tuning approaches, namely FT-prev and FT-s, in terms of forward transfer. There are 9 tasks that are presented twice in the full stream. For each of these tasks, higher forward transfer should imply faster learning when the same task is presented the second time. Notice that the learner has to figure out task relatedness even on duplicate tasks, as every task (including duplicates) is assigned a unique task id and classification head. To measure forward transfer, we adapt the metric proposed in Wolczyk et al. (2021) by computing the normalized difference between the area under the first learning curve and the area under the second learning curve: where AUC 1 and AUC 2 are the areas under the accuracy curves on the evaluation dataset when the task was presented for the first and the second time, respectively. The resulting

Method Name
Indep FT-d FT-prev MT Figure 11: Results on the Split-ImageNet stream with 100 tasks.
metric is less or equal to 1, and higher value indicates better forward transfer. Fig. 10 shows that these FT learners do achieve positive forward transfer in average, with FT-s outperforming FT-prev. In fact, on some tasks like Olivetti and MNIST the forward transfer of FT-s approaches 1.0. Since FT-prev finetunes from a possibly interfering task, on the 15 scenes dataset it obtains an even negative transfer, highlighting again the importance of estimating task relatedness.
Split-ImageNet. In our last study, we apply the same baselines and training and evaluation protocol to another stream, Split-ImageNet. Its much smaller variant, Split-MiniImageNet, is one of the most popular large-scale streams used in continual learning research (Shim et al., 2021;Mai et al., 2022). Split-ImageNet is a stream derived from ImageNet, where the original 1000 classes are partitioned into 100 disjoint groups, creating a stream of 100 10-way classification tasks. The goal of this study is to assess whether this benchmark yields similar findings as Nevis'22. From Fig. 11, it can be seen that the difference between the approaches dramatically reduces as the computation budget increases. Moreover, the ranking of the approaches is rather different. On Split-ImageNet FT-prev performs the best. This is not surprising since tasks in Split-ImageNet are highly related and very homogeneous. This however is an artifact of the lack of diversity of the Split-ImageNet stream, and it highlights the usefulness of the proposed Nevis'22 benchmark.

Ethical Considerations
A potential concern about the methodology used to build Nevis'22 is the use of relatively old datasets that might suffer even more greatly from issues that our community only recently has started to analyze. For example, face datasets have been found to lack demographic representation across characteristics such as race and gender, leading to disproportionate lower performance on these subjects (Prabhu and Birhane, 2021). Additionally, some machine learning datasets scraped from the internet have been collected without subject consent, prompting to privacy concerns in the collection process (Paullada et al., 2021).
The standards and criteria used to build datasets evolve over time, and therefore, is it sensible to even consider a stream built upon historical datasets which might not meet today's best practices on issues such as representation and consent? We posed ourselves this question and concluded that the proposed training/evaluation protocol offers a sufficient mitigation. In particular, the ultimate evaluation is on the meta-test part of the stream, which only includes tasks extracted from the last three most recent years. Moreover, we plan to update the benchmark on a regular basis, to maintain a fresh meta-test stream. This not only alleviates potential model overfitting, but also it enables Nevis'22 to track community's standards for what datasets one should consider for evaluation purposes.
A related question is around deprecated datasets. To the best of our knowledge, we have removed any deprecated dataset during the construction of Nevis'22, or used the most recent non-deprecated version of a dataset (e.g., most recent version of ImageNet). However, it is possible that at some point in the future a dataset currently in Nevis'22 will be deprecated. In that case, we will take the responsibility to update the stream by removing any instance of such deprecated dataset in the stream.
From a machine learning point of view, deprecation raises a very interesting question of how to remove knowledge of a particular dataset from a never-ending learning system which presumably has accrued knowledge over time, including from that particular deprecated dataset. While we do not have an answer to this question, we believe that Nevis'22 offers an excellent realistic environment to assess whether methods are capable of such manipulation of learned knowledge.
We also reflected on the potentially harmful downstream use cases related to vision tasks, specifically tasks related to facial recognition. Our objective is to enable the development of a core capability of an AI system as opposed to target a particular application, such as face recognition, In particular, our models are trained in a closed world setting, meaning that a classification model trained on a dataset in Nevis'22 cannot be used to recognize faces of subjects outside those present in the training set, greatly alleviating concerns related to misuse of models trained on this data for surveillance related applications. Given all these considerations we opted for keeping face datasets that satisfied our requirements on deprecation.
By virtue of the methodology used to construct the stream which relies on existing datasets, we acknowledge the above mentioned limitations of Nevis'22. Further progress is certainly required on data collection methods to tackle issues of representation, deprecation, consent, and potential misuse.
However, we overall believe that the net outcome of this research is positive for our research community and society at large, as Nevis'22 encourages the design of more computationally efficient models, that better leverage previous knowledge to learn the next thing more quickly. Given the amount of resources that large-scale models consume, we believe that taking such perspective is very important and it will be even more important in the future as the community further scales up foundation models. While it is still an open question how to effectively learn sequentially while saving compute, the requirement to explicitly measure not just accuracy but also the compute will encourage researchers to strike better trade-offs than it is otherwise possible today.

Conclusions and Future Work
In this work, we introduce Nevis'22, a benchmark for evaluating life-long learners on a stream of visual classification tasks. These have been derived by uniformly sampling papers from major computer vision proceedings over the last three decades. Since each task is well understood, the main challenge is learning over time to accrue and transfer knowledge. Only by doing so, can learners become more accurate and efficient over time.
Nevis'22 comes equipped with a rigorous training and evaluation protocol that is designed to prevent overfitting to the evaluation set. In particular learners are asked to go through the tasks of the last three years only once, and without accessing data of future tasks. Moreover, the evaluation consists of not only an assessment of the classical generalization error, but also compute in terms of FLOPs. Had we not controlled for compute, results would have been different and less revealing, as beating method A with method B would be easier once we provide B with more compute than A. Finally, Nevis'22 makes meta-learning a first class citizen, since the assessment of the compute spent while learning includes the compute spent while doing hyper-parameter search. Therefore, methods that have more efficient meta-learning algorithms will be favored.
In general, Nevis'22 is not just about a particular stream, but it is also a process to build benchmarks. A similar construction method could have been used on other domains, like natural language processing or reinforcement learning, for instance. Nevis'22 is open-sourced with scripts to recreate the data stream, the training and evaluation framework and code implementing the most classic baselines.
Our initial results obtained by applying standard baseline approaches to Nevis'22 demonstrate the importance of using such a diverse stream. We have found that methods that do transfer perform better than methods that do not, although results vary significantly by domain, image resolution and number of training examples. In particular, we have found that methods that shuffle data to learn generic representations currently strike a worse trade-off between error rate and compute than smarter versions of finetuning. Pre-training approaches perform very well, but they can achieve even superior trade-offs by adapting their representation over time.
Nevis'22 opens up several avenues of future research in never-ending learning. One direction is towards architectures that support variable resolution inputs like the recent Perceiver (Jaegle et al., 2021) and architectures that support efficient inference despite the large number of parameters, like mixture of experts models (Gross et al., 2017). Another direction is on learning algorithms that enable better transfer, model growth over time (Caccia et al., 2022;Gesmundo and Dean, 2022), and parameter sharing across related tasks (Rebuffi et al., 2017a). Another avenue is meta-learning to better answer questions about how to initialize predictors, how to learn more quickly future tasks and how to better shape the search space of architectures, optimizers and learning algorithms. Ultimately, we conjecture that the best method in Nevis'22 will need advances at the intersection of continual learning, meta-learning and AutoML, because it will have to adapt to a non-stationary stream of data, leveraging structure across tasks in order to more efficiently tune hyper-parameters for a new predictor.
In the future, we plan to keep evolving Nevis'22 over time, by moving the current S Ts to S Tr , and forming a new S Ts by adding tasks after 2021. This will prevent overfitting and make sure Nevis'22 tracks the community interests and standards. In the near future, we are eager to learn how the community uses Nevis'22 and we want to understand whether there are ways to improve it. Eventually, we plan to extend Nevis'22 towards a multi-task and multi-modal stream, while retaining the same rigorous training and evaluation protocol we have defined in this work.

A. Individual Contributions
If you wish to contact us, please email us at nevis@deepmind.com. For questions about a specific part of this work, please reach out directly to the relevant authors. Table 3    In the experiment of Fig. 12 we explore another region of the pareto front: the extremely frugal learners that require between 10 17 and 10 18 to run on the full stream. This amounts to about 4 days on a single GPU with 16GB of RAM. Unlike what we presented in the main paper where we used a ResNet34 adapted to low-resolution images (with many more channels and without as much spatial sub-sampling), here the network is a default ResNet18 and ResNet34 architecture, which have been designed for higher resolution images. These architectures are cheaper because a) they have at most the same number of blocks, but b) they have many fewer channels. We further reduce compute by fixing the label smoothing parameter to the mid value of our search range, 0.15, and by searching only over four values of learning rate, namely {1e − 4, 1e − 3, 1e − 2, 1e − 1}.
We can see that these cheaper architectures and hyper-parameter strategy extend the Pareto front at lower budgets. This suggests that another possible avenue of future work is the design of more efficient architectures and hyper-parameter search methods. Importantly, this might provide an easy entry point to Nevis'22 for researchers who have limited computational resources at their disposal.

C. Architectures
In this section, we study the impact of the architecture choice. We focus on the independent baseline used as reference in all our experiments, and we conduct two experiments. In the first, we use different architecture variants: VGG, ResNet34, ResNet50 and Vision Transformers (ViT) (?). In this manuscript we use a ViT-B8 with the modifications proposed by ? where the classifier is applied after global average pooling over the vision tokens. As all the experiments are conducted using the same resources, the bigger architectures (ResNet50 and ViT) are run with a smaller maximum batch size (see Eq. (4)), and similar search spaces for the number of updates and learning rate. It is therefore expected to see these architectures underperforming, as we observe in Fig. 13.
The second experiment focuses on ResNet34, chosen as default architecture, and varies the number of channels in the different residual blocks. The results of this experiment complete Fig. 8, providing another axis to trade off error rate versus compute. The full results are shown in Fig. 14. In the legend, rez-rxr indicates the input resolution, ch-c corresponds to the number of channels in the first residual block (the number of channels in the remaining blocks are consecutively doubled), and bsz-b shows the maximum batch size. We observe that reducing the number of channels saves a significant compute budget, without a significant loss in performance.

D. Additional Baseline Results
In this section, we report additional results on the Nevis'22 stream. We namely show the results of the finetuning and pretraining families, and complete the experiments reported in Sec. 5.1.

D.1 Finetuning
This section focuses on the Finetuning family. In Fig. 15, we show the Pareto fronts of FT-d, FT-prev and FT-s. Fig. 16 reports the regret plot of the same methods, i.e. the cumulative error over time relative to Indep, picking hyper-parameters such that all methods use roughly the same amount of compute (same as in Fig. 4). These results demonstrate the importance of the choice of what to finetune from, as FT-p is significantly worse than FT-s and FT-d, and FT-s is worse than FT-d suggesting that it is important to make an accurate (up-to-date) estimate of task relatedness. The regret plot further shows that the tested finetuning strategies fail to increase transfer over time (linear slope over the full stream) and to accumulate knowledge over the last 9 tasks of the sequence (horizontal curves). In Fig. 17 we report an example of the finetuning sequence learned by FT-d (this is the version which was trained on every task with 50, 000 steps using 16 trials of hyper-parameter search). We notice that there are several hubs from which several other models are finetuned from. The biggest hub is ImageNet, followed by Caltech256, MNIST, Caltech101, Scene8, etc.
In the graph tasks are mostly organized by visual similarity and domain. For instance, we see two clusters of OCR tasks in red, cluster of medical images in cyan, and a large cluster of generic object recognition tasks in yellow. Finally and perhaps most surprisingly, we observe fairly long chains of finetuning models. It seems that finetuning even more than ten times can produce high performing models. Figure 17: Graph showing the inner working of FT-d. Each box correspond to a task, the color represents the domain. An arrow connecting task i to task j indicates that task i was selected as initialization for the network trained on task j.

D.1.2 Analysis of PT+FT
In Fig. 18 we report an example of the finetuning sequence learned by PT+FT, i.e., FT-d starting from the model pretrained on external data (PT-ext). We report the version which is reported in the regret plots in Figures 4 and 20, which corresponds to the best compute-error trade-off in Figures 3 and 19. Compared to the finetuning sequence learned by FT-d when learning from scratch (see Fig. 17), we observe that while chains are shorter, not all tasks choose to be initialized from the original pre-trained model. ImageNet is still a strong hub, but there are many chains of length two or three that cluster datasets by domain and visual similarity. Figure 18: Graph showing the inner working of PT+FT (i.e., FT-d starting from a pretrained model). Each box correspond to a task, the color represents the domain. An arrow connecting task i to task j indicates that task i was selected as initialization for the network trained on task j. Compare this graph to the one of FT-d (starting from scratch) in Fig. 17. Nevis'22

D.2 Pretraining
This section focuses on the Pretraining family. Below, we report results using models pretrained on ImageNet using full supervision (PT-ISup), on the meta-train part of the stream (PT-MT), and on ALIGN and Stock using CLIP (PT-ext). Note that this last variant uses not only a much larger (external) dataset, but also a substantially more powerful architecture.
In Fig. 19, we show the Pareto fronts of these methods, with (bottom) and without (top) the computational cost of pretraining. As expected, we observe that PT-ext leads to a significantly higher performance. However, when the pretraining cost is taken into account, the trade-off between performance and compute is less impressive. We also observe that training on the whole training part of the stream (which includes ImageNet) performs worse than pretraining on ImageNet only.
In Fig. 20, we report the regret plots of the pretraining methods on the last 3 years (top) and the full stream (bottom) with respect to the Indep baseline. As none of these methods accumulate knowledge, it is not surprising to observe that their transfer does not improve over time. However, it is more surprising to observe that they are on a par with Indep for the last tasks (horizontal curves). This shows that even the pretrained model on the largest dataset (PT-ext) fails to transfer well to the medical datasets that appear towards the end of the stream.

E. Task Ordering
This section completes the results reported in Fig. 9. In Fig. 21, we show the regret plots with respect to the Indep baseline for different random stream orderings. corresponding to the "Random Order (All stream)" experiments in Fig. 9. The positions of ImageNet in the stream are displayed on top of each of the panels, and highlighted on the curves. These results show how the position of this dataset, which dominates the stream in terms of size and complexity, influences the performance. The experiment where ImageNet appears at the beginning of the stream corresponds to the best performance reported in Fig. 9.

E.1 Multitask Ablation
In Fig. 22 we show how various hyper-parameters choices affect the performance of MT. In particular, we observe that picking all tasks versus sampling a subset of them does not yield a better trade-off, but merely extends the Pareto front towards higher compute regimes. We   Figure 22: Pareto front of meta-learners using different multitasking strategies on SHORT stream. ft_dknn+top1_similar refer to multitask baseline where the meta-learner initializes the predictor of a new task using most similar previous task using dynamic KNN transfer matrix, and use only k = 1 auxiliary task. Similarly, ft_dknn+top3_similar uses k = 3 auxiliary tasks. ft_prev+all and ft_prev+random refer to multitask baselines where the predictor of a new task is initialized form most recent previous task's parameters and metalearner either uses "all" the previous tasks as auxiliary tasks, or randomly picks one of the previous tasks as an auxiliary task during an SGD update. Finally, ind+all and ind+random refer to multitask baselines where the parameter are initialized randomly and either all or one of the previous tasks are used as auxiliary tasks.

Multitask (MT_Strategies)
also observe a very large gap due to how the multitask network is initialized. Despite the multitask learning objective, it is very beneficial to initialize the network, even using the parameters of the most recent previous task. While there is no best initialization across all the compute budgets, i.e., no initialization dominates all the others, we have found FT-d strategy to work better in the intermediate compute budgets. Further, in that setting, it is empirically best to co-train with the task the network is finetuned from (top-1), as opposed to co-train with other tasks as well (top-3). Hence, we use FT-d (top-1) as the MT strategy while reporting the metrics in the main paper. On architectures that use BatchNorm (Ioffe and Szegedy, 2015), we found it to be critical for performance to not update the BatchNorm statistics, running means and variances, with the batches of auxiliary tasks. Further, using a small fixed batch size of 64 for all the auxiliary tasks resulted in a compute-efficient learner without hurting the performance as compared to a learner that uses large variable batch sizes for the auxiliary tasks (see eq. 4).

E.2 BHPO Ablation
In this section we study the impact of the choice of the search space and optimization algorithm (BHPO versus random search). Using Indep as a case study, we compare two search spaces, a small search space with only two hyper-parameters (learning rate and label smoothing as in the default setting) and a larger space with 7 hyper-parameters which include also how data is augmented, the network architecture, etc. We also vary the number of training steps per trial (5K, 10K, 20K, 50K) and the number of trials (20,40). As shown in Fig. 23a a large search space allows one to find a smaller error rate given the same training steps. This is mostly evident with a small number of training steps when the training has not converged yet. With a larger number of steps per training, the advantage diminishes. Also, it becomes harder the find the optimal hyper-parameter setting in a larges space given the same number of trials. Moreover, as the large search space includes optimizing the batch size, the computation cost is much higher. In Fig. 23b we compare random search with the more sophisticated BHPO method. The latter finds a better setting in most cases in the large search space where the optimization is more difficult, but there is no clear difference in the easier 2-parameter space.

F. Ensembling
In this section, we start by observing that during random hyper-parameter search we perform N trials in parallel, one for each particular configuration of hyper-parameters. Since a hyper-parameter search often yields a set of accurate and diverse models, it seems wasteful to retain only the model that performed the best on the validation set. Instead, we can create an ensemble from these already trained models (Dietterich, 2000). Fig. 24 shows that we can significantly lower the error rate by ensembling, or for the same error rate we can drastically reduce the compute. For instance, by ensembling networks trained on 4 trials only, we can attain lower error rate at half of the training compute of a (single component) baseline trained with 8 trials.
In our experiments we have found that the best ensembling approach uses a weighted sum of probability distributions. The weights are the output of a softmax with temperature Figure 24: Error rate of ensembling as a function of the number of components. The reference is FT-d using only the best model found during random hyper-parameter search. equal to 0.1 using as input the top-1 accuracy obtained during cross-validation. This has the effect of weighting more the top performing models.
Notice however that while ensembling reduces the training compute at a given level of error rate, it increases linearly the cost at inference time.

G. Measuring Compute
For the Nevis'22 evaluation, we had the key goal of enabling a fair comparison between approaches that efficiently achieve strong results -such as via knowledge re-use or carefully applied meta-learning strategies -versus methods that always re-train from scratch over all data, and perform large hyper parameter sweeps to obtain good results. We explicitly wanted to choose a metric that would favor efficient learners that can scale gracefully as the data and the number of tasks increase.
Some options that we considered include 1) Counting the total number of training examples used (including multiplicity) 2) counting the total number of optimization steps 3) Estimating the time elapsed during training. Ultimately we found problems with all of these approaches (and other similar approaches). A common issue is that we felt the measurement should not limit in any way the solution space available to those implementing learners. Counting optimization steps may disadvantage learners that take many very small steps, counting the number of examples accessed has the the problem that, in-reality, accessing data that already exists is a relatively inexpensive operation -and it's easy to imagine that this could penalize learners that make use of replay buffers or caches that ultimately increase learner efficiency. Nevis'22 Counting floating point operations also suffers from challenges -notably since most hardware accelerator devices have a fixed maximum throughput of floating point operations per second. Achieving this maximum throughput is key to training learners quickly, and making best use of available resources. Learners that use very sparse compute sparingly may ultimately take a lot longer (in wall clock time) to train than learners that can efficiently use the dense compute available in modern accelerators (such as dense matrix multiplications). Extending this further, learners that can efficiently be trained in parallel can require far less wall-clock time to train compared to learners that are inherently sequential. Many real-world applications would prefer learners that make use of large amounts of compute if they are highly parallelizable. Furthermore, the actual number of floating point operations performed is dependent on the underlying hardware platform (for example due to padding vectorized computations), and the levels of optimization performed on the linear algebra primitives themselves.
When considering our approach, we ultimately kept in mind the following core constraints, 1) the approach should not penalize or limit any particular learner implementation, 2) the approach should reasonably approximate cost of training the learner in a way that maps to reality, 3) storage and memory are typically cheaper than overall numerical compute, and so can be ignored without affecting the ordering of results, 4) it should be practically feasible for users of the benchmark to compute comparable values for the resources used, independently of the hardware they have available. In these respects, we feel that the cumulative floating point operations used offers a fair compromise.