{'fraction_non_alphanumeric': 0.027485468029665262, 'fraction_numerical': 0.017413309280416917, 'mean_word_length': 5.067649741562785, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Continual Development Methodology for Large-scale Multitask Dynamic ML Systems

The traditional Machine Learning (ML) methodology requires to fragment the development and experimental process into disconnected iterations whose feedback is used to guide design or tuning choices. This methodology has multiple efficiency and scalability disadvantages, such as leading to spend significant resources into the creation of multiple trial models that do not contribute to the final solution.The presented work is based on the intuition that defining ML models as modular and extensible artefacts allows to introduce a novel ML development methodology enabling the integration of multiple design and evaluation iterations into the continuous enrichment of a single unbounded intelligent system. We define a novel method for the generation of dynamic multitask ML models as a sequence of extensions and generalizations. We first analyze the capabilities of the proposed method by using the standard ML empirical evaluation methodology. Finally, we propose a novel continuous development methodology that allows to dynamically extend a pre-existing multitask large-scale ML system while analyzing the properties of the proposed method extensions. This results in the generation of an ML model capable of jointly solving 124 image classification tasks achieving state of the art quality with improved size and compute cost.


INTRODUCTION
The traditional approach to the development of Machine Learning (ML) models assumes a fragmentation of the design and evaluation into multiple disconnected iterations causing inefficient use of available resources and engineering effort. Alternative design choices are normally implemented and tested separately rather than being modularized and integrated in a search space whose exploration can be automated. Furthermore each experimental iteration requires to reset the model's knowledge to a comparable initial state such as random initialization, thus preventing reuse or accumulation of knowledge across trials. This fragmented development methodology is applied also to aspects such as hyperparameter tuning, as the training is usually repeated multiple times to compare different hyperparameter assignments even for a fixed model design. This approach leads to spend a significant part of the available compute into the training of multiple models that provide no contributions to the final ML solution. Furthermore, it relies on significant human intervention rather than aiming to automate any of the design or tuning exploration.
We propose and demonstrate a methodology that integrates the ML design and the experimental evaluation iterations into the continual expansion of an arbitrary pre-existing ML model. The proposed ML methodology allows any design variation and model knowledge enrichment to be evaluated as an extension of the latest state of the dynamic ML system. Successful extensions can be integrated as base for future experimentation and further extensions. This methodology enables the creation of dynamic multitask systems at a scale that would not be feasible to achieve with the constraint of running parallel repetitions from a neutral start state.
The two major contributions of the presented work are: • A novel method for the generation of multitask dynamic systems: We define a set of design and method extensions applicable to the µ2Net method (Gesmundo & Dean, 2022b), that provide improvement and generalization of different aspects such as: multi-factor model scoring, evolutionary agent capabilities and automated hyperparamter tuning (Section 3). We refer to the novel method to as µ2Net+. The novel methods capabilities are first analyzed by using the traditional ML methodology on a smaller scale for feasibility (Section 4). • A novel ML development methodology integrating design and evaluation into the continual and unbounded extension of an ML system. We demonstrate empirically the effectiveness of the proposed methodology by incrementally extending the model generated by the experiments described in Gesmundo & Dean (2022b). The model is incrementally enriched as the µ2Net+ improvements are introduced and as additional tasks are learned. This large scale continual experiments allows to analyze the effects of the method improvements (as done with the traditional methodology), while also continually extending the model into a large-scale multitask ML system capable of learning 124 image classification tasks with approximately state of the art quality and improved size and compute efficiency. (Section 5).

RELATED WORK
Many methods have been proposed to achieve dynamic architecture extensions (Chen et al., 2016;Cai et al., 2018), some also focusing on an unbounded stream of tasks (Yoon et al., 2018), some achieving immunity from catastrophic forgetting (Rusu et al., 2016;Rosenfeld & Tsotsos, 2020) or optimizing for quality/cost trade-offs (Tan et al., 2019) (refer to Appendix A for an extended survey). Though, a methodology that integrates the continuous development of methods for dynamic architecture extensibility and knowledge enrichment of a multitask ML system jointly with the human design and experimentation process has not yet been proposed.
Berner et al. (2019) describes a methodology that allows for the continued variation of model architecture and training method while preserving its knowledge. Although, the dynamic architecture extension capabilities are not provided by the definition of higher level abstractions but rather require surgery operations requiring custom human intervention. Berner et al. (2019) tackles a single highcomplexity task, and the ability of exploring architecture and method variations without resetting the knowledge is deemed a critical accelerator in such context. Advancements in ML models dynamicity, extensibility and reusability can contribute to accelerate progress toward further high-complexity achievements such as "Artificial General Intelligence" (AGI).
The ability of jointly solve a large amount of tasks is another theme that is commonly associated with progress toward AGI. Advancements in scaling language models (Brown et al., 2020;Thoppilan et al., 2022) allowed to achieve discourse and reasoning capabilities that can be applied to new tasks without requiring additional training. Recent work aims to extend these achievements beyond text modality by defining static architectures for an extended subset of modalities (Alayrac et al., 2022;Reed et al., 2022). These are few examples of the ML models contributing to the line of research achieving incremental milestone toward AGI. Though, each model is trained from scratch with considerable resources consumption. The introduction of abstractions allowing to modularize, dynamically extend and reuse these large models may contribute to accelerate the rate of innovation.

METHOD
The proposed method extends µ2Net (Gesmundo & Dean, 2022b) with the intent of generalizing and improving its capabilities. In this section we detail the four method extensions introduced with µ2Net+: 1) multi factor scoring function, 2) improved hyperparameters search space, 3) additional mutation actions, 4) learned µ(·) function. Refer to Gesmundo & Dean (2022b) for a detailed definition of the µ2Net method being extended.
The first method extension consists of the introduction of a new multi factor scoring function that integrates additional size and compute penalty factors by employing an exponential decay penalty: (1) q(m) denotes the quality metric achieved by model m on the validation set. s ∈ ]0, 1] is the cost penalties scaling factor. #f lops(m) denotes the amount of floating point operations required by model m at inference time, and #accounted-params(m) is the sum of parameters used by model m, dividing each parameter count by the number of models sharing its use: #accounted-params(m) = p∈P(m) 1 #models(p)+1 (2) P(m) denotes the set of all parameters used by m. #models(p) denotes the count of models for tasks different from the task m is trained on, that are currently sharing the use of parameter p. Intuitively, if s = 0.99, then a relative reduction of 1% of the quality metric is applied if a tasks is accounted for P parameters or requires F flops.
The base µ2Net method allows for 2 types of mutation actions: 1) any layer from the parent model can be cloned to create a trainable copy carrying its parameters and optimizer state (e.g. momentum statistics), any other layer is shared in a frozen state, 2) any hyperparameter value can be changed to one of the neighbouring values in the sequence of valid values defined by the search space (see Table 1). µ2Net+ includes a third mutation type, layer removal: the top transformer layer of the parent model can be removed from the sequence of layers defining the child models (see Figure 1). The addition of this mutation type allows to sample models with a number of transformer layers that is smaller than the number of layers of the root model. This allows to generate models using less parameters and compute, thus introducing an additional degree of freedom to optimize for the introduced cost factors.
The hyperparameters search space extension includes different types of additions (see Table 1). All the models generated by µ2Net are constrained to use 384 pixels image resolution. This is the resolution recommended for ViT fine-tuning (Steiner et al., 2021). Although the ViT model we are using as a root model has been pretrained on 224 resolution. We extend the search space to include the option of using 224 resolution. This provides an additional option to generate models requiring less compute, as one of the objectives of the empirical study is to analyze the quality/compute trade-off. The options for image preprocessing are extended by adding the possibility to alter the image quality. This is a commonly used technique that was not included in the recommended fine-tuning setup (Steiner et al., 2021) and therefore was not included in µ2Net search space. Though, we find that this new preprocessing function is learned to be activated for a significant portion of the generated models configuration even if set as inactive by default (see Figure 5). Rarely used values have been removed. Steiner et al. (2021) recommends a mild preprocessing configuration for ViT fine-tuning. The preprocessing hyperparameter default values for µ2Net search space were set to match the recommended configuration. Instead, µ2Net+ set the default values to disable all the preprocessing. This change is based on the observation done during preliminary experiments that preprocessing can moderately benefit most of the tasks, but it can have critical negative impact for a subset of tasks. New values have been added to the valid values ranges of some hyperparameters based on observations done over the distributions learned in previous experiments such as those reported in Gesmundo & Dean (2022b). For example, the most common selected value for the "learning rate schedule warm up ration" was the edge value of 0.01, thus the range has been extended by adding the value 0, that allows to disable the learning rate warm up. Also notice that µ has been removed from the search space as a hyperparameter since it is now defined as a learnable function conditioned on the mutation type, as described in the following paragraph.  Figure 1: Graphical representation of the three types of mutations defined by the proposed method. New models are generated by applying a sampled subset of possible mutations.
In Gesmundo & Dean (2022a) the mutation sampling is parameterized by a global mutation probability constant µ. Gesmundo & Dean (2022b) extends the approach by defining µ as a per model hyperparamter tuned by the evolutionary method, thus enabling the selection of distinct mutation probabilities for each model and adapt the their value through time. µ2Net+ allows for further per action customization of the learned mutation probabilities by defining a µ function, µ(δ|m), for each model m. This function maps each mutation action, δ, to the probability of applying the mutation action. This extension enables the selection of a distinct mutation probability for each possible mutation action. In practice, the µ function is implemented as look-up table mapping every possible mutation to a distinct mutation probability value. The values of the lookup table are inherited and mutated across generations following the same incremental value mutation logic applied to the hyperparameters. All the values in the lookup table are initialized to 0.2, and the sequence of valid mutation probability values is defined as the multiples of 0.02 in the range [0.02, 0.3].

PARALLEL TRAINING COMPARISON
In this section we use the standard ML methodology to compare different method by running parallel replicated experiments each using comparable configuration and amount of compute. In particular we focus on quality/cost trade-off comparison. Thus, we compare against common baseline methods that allow to explore the Pareto frontier of such trade-offs. We compare against the muNet method (Gesmundo & Dean, 2022a), and against standard transfer learning techniques such as fine-tuning and residual adapters that provide a cover of the considered trade-off spectrum. As the proposed method introduce the joint optimization of quality, size and compute, we consider two trade-offs: quality/size and quality/compute. For comparability, the experiment setup and benchmarks are equivalent to those defined in Gesmundo & Dean (2022a) and are summarized in this paragraph. To test the generality of the results, we repeat the experiments on 2 benchmarks: 1) the Multitask Character Classification Benchmark (MCCB), composed by 8 character classification tasks, 2) the Visual Domain Decathlon Benchmark (VDDB), composed by the 10 tasks of the Visual Domain Decathlon challenge. The VDDB is more computational intensive and tasks have been selected to represent a diverse set of domains (see Table 5 for datasets details and references). Experiments are run on TPUv3 with 8 parallel cores. Each experiment is repeated multiple times to measure variance: 5 times for MCCB and 3 times on VDDB (the lower number of repetitions is due to the higher experiment cost). Every experiment is allocated an equivalent training budget in terms of training epochs per task. Epochs are capped to 51200 training samples for all datasets to smooth the distribution of compute over the set of tasks. are achieved by freezing an increasing number of bottom layers, and again all the frozen layers are shared across multiple tasks to decrease the average number of parameters per task.
The only difference between the experiment configuration of the baseline muNet method and the proposed µ2Net+ method is in how the equivalent training budget is distributed. For µ2Net+, the training budget is distributed to perform more but shorter task iterations. The motivation for this change is to bring the ratio between number of task iterations and length of each iteration closer to what would be representative of the setup for a long term continual learning experiment, that require a significantly higher number of task iterations than the 2 that were originally allocated to muNet experiments. Thus we change the distribution to increase the similarity with the setup used by the large scale experiments reported in Gesmundo & Dean (2022b) and their extension with the novel methodology reported in Section 5. These experiments perform 4 training epochs per generation, 4 generations per task iteration and allow for an unbounded number of task iterations. In details, the training budget distribution change consists of: for the MCCB, muNet is configured to perform 2 task iterations, 8 generations for each task iteration and each model is trained for 5 epochs, while µ2Net+ performs 5 task iterations, 4 generations for each tasks, and 4 training epochs per model, totalling the same amount of training epochs per task. While, for the VDDB, muNet is configured to perform 2 task iterations, 8 generations for each task iteration and each model is trained for 30 epochs, while µ2Net+ performs 30 task iterations, 4 generations for each tasks, and 4 training epochs per model.
The µ2Net+ size cost factor is configured with parameters equivalent to those used for muNet: P = 1.48M for MCCB and P = 85.6M for VDDB proportionally to the number of parameters in the root model. While for the compute cost factor we set: F = 15M for MCCB and F = 100M for VDDB proportionally to the inference flops required the root model with an image size of the higher end of the allowed range.
The results are represented graphically in Figure 2. The overall comparison shows that the proposed method outperforms all the baseline methods in both benchmarks across both the quality/size and quality/compute spectrum. We also compare with a version of µ2Net+ without compute cost factor. In the quality/size spectrum we observe that the addition of the compute cost factor does not alter significantly the achieved Pareto frontier. While, we notice a significant gain in the quality/compute trade-off. For example, notice that µ2Net+ without compute cost factor does not outperform muNet in lower end of the quality/compute spectrum for the VDDB experiments. The metrics of the configurations achieving the best quality for each method are reported in Table 2.
It is also interesting to notice that the curves representing the Pareto frontier achieved by Fine-tune top layers becomes a vertical line in the quality/compute spectrum, because the inference compute is independent from the number of layers that are frozen and shared. Also notice that the Residual adapters Pareto frontier outperforms the Fine-tuning one in the quality/size domain. That is expected since Residual adapters are usually employed as a parameter-efficient transfer-learning technique. Although, the order is inverted in the quality/compute spectrum, since Residual adapters add compute cost on top of the root model.

CONTINUAL MODEL EXTENSION
In this section we describe and demonstrate the proposed continual development methodology that aims to analyze the properties of method and knowledge augmentations through a sequence of experiments while dynamically extending a single large-scale ML model. With this methodology, we aim to characterize and quantify the effects of the introduced extensions by measuring changes thought time, rather than comparing the metrics achieved by parallel runs of method variations (as done with the methodology used in Section 4). The proposed continual development approach has the following advantages: 1. allows to focus all the available compute on the enrichment of a single ML system, thus increasing the accessibility of larger scale experiments, 2. lifts the requirement of training multiple model variations that do not contribute to the final solution for the tasks at hand, 3. enables the unbounded extension of dynamic large-scale multitask systems as the one considered in this section, since it would not be feasible at this scale to train multiple randomly initialized copies of such systems to evaluate each method variation that can be interesting to analyze.
The following experiments also demonstrate how the proposed µ2Net+ method can be applied to extend a pre-trained multitask system generated with a different method. The large-scale multitask system generated by the experiments described in ( (2022b), notice that the reported test accuracy is computed on a test set that does not overlap with the training set and validation set (whose validation accuracy is used for the reward computation) to avoid overfitting and selection bias. 2) Accounted parameters averaged across all the models composing the multitask system at the time of measure (not limited to task-set A). This measure corresponds to the size cost factor included in µ2Net+ reward function (see Section 3). 3) Compute cost measured as flops required to produce inference for one input sample, also averaged across all the models composing the multitask system at the time of measure (not limited to task-set A). This measure corresponds to the compute cost factor included in µ2Net+ reward function (see Section 3).
multitask system. This starting multitask system can already solve a set of 69 image classification tasks. With task-set A we refer to that first set of 69 tasks. With task-set B we refer to an additional set of 55 image classification tasks that will be introduced in following experiments.
The continual extension of the system is described as a sequence of experiment segments. Each segment introduces extensions to the method or task-set. The the effects on the system in terms of quality, efficiency and capabilities are measured through additional task iterations.

EXPERIMENT SEGMENT 1: µ2NET CONVERGENCE
The first experiment segment applies the µ2Net method as defined in (Gesmundo & Dean, 2022b) for an additional number of tasks iterations with the aim of achieving a sufficient system stability and convergence on task-set A. Achieving convergence with the baseline method before transitioning to an extended version allows to attribute the effects observed the following experiment segments to the corresponding method extensions, as the method extensions are compared sequentially rather with standard parallel experiments.
We observe that 2 additional task iterations are required for µ2Net to achieve sufficient convergence on task-set A (see Figure 3). During the first task iteration, the average test accuracy increases by 0.18%. While during the second task iteration, the test accuracy does not improve. The trend of the "average accounted parameters per task" metric is similar, as the variation is significantly smaller during the second iteration. Notice that all the models generated by the µ2Net method have a constant compute cost, that matches the compute cost of the used ViT Large root model with input image resolution of 384×384 pixels. This is the case since the µ2Net method does not provide mutation actions that can alter the compute cost of the generated model, such as adding/removing transformer layers or changing the image resolution.

EXPERIMENT SEGMENT 2: µ2NET+ INTRODUCTION
The second experiment segment introduces the proposed µ2Net+ method. To produce mild size and compute cost penalties, the scoring function parameters P and F are set respectively to ∼ 10 times the accounted parameters and inference flops averaged across all the models that are part of the system at the beginning of this experiment segment: P = 2.2×10 9 and F = 3.8×10 12 . s is set to 0.99. All the common evolutionary agents meta parameters, such as those determining the compute utilized per model training and exploratory budget per task iteration, are unchanged for comparability (Gesmundo & Dean, 2022b).
Two additional task iterations on the task-set A are performed. The metrics reported in Figure 3 show that the introduction of the novel method leads the newly generated models to share more parameters and be more computationally efficient. Despite this, the average test quality does not deteriorate. Overall resulting in a 3.6% reduction in compute and 21.7% reduction in size with no loss in quality.
This efficiency improvement is enabled by: 1) the generation of cheaper models is enabled by the introduction of mutation actions allowing to reduce the number of transformer layers and the extension of the hyperparameter search space allowing to chose smaller input image resolution, 2) cheaper models selection is encouraged by the introduction of the size and compute cost factors. 3) the learning process of the µ(·) function increases the probability of sampling cheaper models by customizing the probability of occurrence of different mutation types. For example, the probability of performing a layer cloning action decreases from 21.4% to 17.4%, thus decreasing the expected accounted parameter cost of newly sampled models. Furthermore, the learned µ(·) functions appear to assign lower cloning probability to lower layers in average (see Figure 6).

EXPERIMENT SEGMENT 3: TASK-SET B INTRODUCTION
The third experiment segment consists in extending the number of jointly solved tasks from 69 to the 124. This is achieved by performing one task iteration on the additional 55 tasks of task-set B to introduce them into the system. These additional image classification tasks are also publicly distributed via the Tensorflow Datatasets catalog.
The average test accuracy achieved on task-set B after one task iteration is 90.66%. If we compare to the accuracy achieved at convergence in the next experiment segment (90.78%), we can observe that the system is able to achieve close to converge quality with only one task iteration. While, it took more task iterations to achieve the same relative delta to convergence quality on task-set A: at least 4 task iterations (including the task iteration described in Gesmundo & Dean (2022b)).
This observation can be an empirical evidence supporting the following hypothesis: The expected convergence time for a new task is inversely proportional to the amount of knowledge embedded in the system. This hypothesis is based on the intuition that a new task can be learned faster if a model or sub-set or parameters with as relevant knowledge as possible is available to be used or fine-tuned. And the probability for a new task of fining relevant knowledge within the system increases as the system learns more tasks. This hypothesis can be also supported by the sharp drop in "average accounted parameters per task": 14.5% relative reduction during this iteration (see Figure 3), as the new models added to the system seem to take advantage of layers and knowledge sharing with a higher degree.
The average test quality measured on task-set A remains unchanged since its tasks are excluded from this iteration. The average inference compute per task appear to continue to decrease: 4.3% relative reduction in this iteration.

EXPERIMENT SEGMENT 4: µ2NET+ CONVERGENCE
To conclude we perform 2 additional task iterations on the union of task-sets A and B.
Interestingly, the average test quality measured on task-set A improves significantly after achieving no significant changes for 3 consecutive task-iterations (see Figure 3). With respect task-set A, the only variation introduced in this experiment segment is the possibility of accessing the new knowledge introduced by learning to solve the additional 55 tasks of task-set B. We measure that, during this experiment segment, 15 tasks of task-set A find an better scoring model by transferring knowledge from at least one new task of task-set B, and 9 of these improved models transfer knowledge from 2 or more task-set B tasks.
The average inference flops also further improves. Computationally cheaper models are rendered possible by the introduction of the possibility to remove transformer layers and use smaller input image resolutions. Before the introduction of the proposed method all the models used 384 pixels input image resolution and 24 transformer layers as the ViT Large root model. Since the introduction of µ2Net+, 30.4% of the 124 selected models use 224 image resolution, and 27.2% use 23 transformer layers, 1.6% use 22 and 2.4% use 21 transformer layers (see Figure 5).

CONCLUSION
The reported series of system and method extensions demonstrates the ability of the proposed ML development methodology of integrating the continual design and evaluation process into the unbounded extension of large-scale multitask ML systems. The proposed continual development methodology not only allows for the continual addition of knowledge and tasks into a dynamic multitask system, but also allows for dynamic extensions of the search space defining the architectures and configurations that are possible to explore and furthermore allows to iteratively extend the method exploring such search space while also adapting through time the guiding reward function to adjust according to possibly changing requirements.
Overall, the methodology and empirical results demonstrated in by this work are encouraging signs that a highly automated and incrementally extensible ML system for handling thousands or millions of tasks with improved quality and efficiency is achievable. Future work can continue to build toward a system that can acquire further capabilities and knowledge across multiple modalities.

A EXTENDED RELATED WORK SURVEY
The proposed method is designed to learn an unbounded number of tasks in a continual learning fashion. In such contexts it aims to learn each task with higher quality and efficiency by automating and optimizing the knowledge transfer among any subset of tasks that can provide useful knowledge to one another. The proposed model is designed to be immune from common multitask learning pitfalls: catastrophic forgetting, gradients interference, negative transfer. Cross-task transfer-learning has gained popularity, especially through transfer learning from a model pre-trained on a large amount of data for one or a few general tasks, and then fine-tuned on a small amount of data for a related downstream task. This approach has been shown to be very effective in a wide variety of problems across many modalities, including language (Devlin et al., 2019; Raffel et al., 2020) and vision (Dosovitskiy et al., 2021;He et al., 2016). The success of transfer-learning applications hinges on adequate prior knowledge selection to avoid typical negative transfer pitfalls (Rosenstein, 2005;Wang et al., 2019b). Common solutions rely on data or model selection techniques, often putting emphasis on the efficiency of the exploration (Zhang et al., 2020;Mensink et al., 2021), also method aiming to automate knowledge selection at a layer level have been proposed Sun et al. (2020a). Transfer learning capabilities are critical for multitask models. ML models trained jointly on multiple tasks can be affected by gradients interference if any subset of parameters receive gradients jointly from multiple sources (Chen et al., 2018;Yu et al., 2020), and by catastrophic forgetting of prior knowledge as new tasks are learned (McCloskey & Cohen, 1989;French, 1999). These knowledge loss problems can be alleviated with weighted combination of tasks (Liu et al., 2019b;Sun et al., 2020b) and gradient transformation methods (Chen et al., 2018;Sener & Koltun, 2018;Kendall et al., 2018). Stronger guarantees are provided by methods that compartmentalize task specific knowledge in dedicated parameter subsets (Rebuffi et al., 2017;Houlsby et al., 2019;Rusu et al., 2016;Rosenfeld & Tsotsos, 2020). Addressing catastrophic forgetting and identifying what subset of parameters/knowledge that is beneficial to share with each task is also critical for continual learning or life long learning methods (McCloskey & Cohen, 1989;French, 1999;Ramesh & Chaudhari, 2022).
The proposed method relies on an evolutionary approach to jointly search the spaces of models architectures, hyperparameters, and prior knowledge selection while optimizing for an possibly multi-factor non-differetiable reward function. The automation of hyperparameter tuning has been commonly addressed with Bayesian optimization (Srinivas et al., 2010;Bergstra et al., 2011;Snoek et al., 2012), evolutionary methods have also been explored for this purpose (Jaderberg et al., 2017;Zhang et al., 2011). Hyperparameters tuning can be considered related to the neural architecture search (NAS), as architectures can be defined by the selection of a sequence of architectural hyperparameters. Initially, NAS methods have been based on reinforcement learning techniques (Zoph & Le, 2017) but also sample efficient evolutionary approaches have been proposed (Real et al., 2019;Maziarz et al., 2018). Parameter-sharing based NAS methods aim to reduce the typically high training cost (Pham et al., 2018;Liu et al., 2019a;Kokiopoulou et al., 2019). Optimization for multi-factor quality/cost trade-offs have been explored (Tan et al., 2019).
The proposed method is capable to dynamically extend the system, adding capacity or novel structures in an unconstrained fashion. A few methods have been proposed to achieve dynamic architecture extensions (Chen et al., 2016;Cai et al., 2018), some also focusing on an unbounded stream of tasks (Yoon et al., 2018;Yao et al., 2020), or achieving immunity from catastrophic forgetting (Rusu et al., 2016;Li & Hoiem, 2018;Li et al., 2019;Rosenfeld & Tsotsos, 2020).
The proposed method is sparsely activated, thus the unbounded growth of knowledge and parameters is decoupled from the growth of computational cost. The growth in capabilities of state of the art models often requires growth in terms of trainable parameters (Kaplan et al., 2020). Sparse activation techniques at sub-layer level (Shazeer et al., 2017;Du et al., 2021) or network route level (Fernando et al., 2017) allow to decouple model size growth from compute cost. This is achieved by integrating a routing technique that selects the appropriate subset of parameters storing the most relevant knowledge for each task, sample or token/patch.
The ability of jointly solve a large amount of tasks is commonly associated with progress toward Artificial General Intelligence (AGI). Advancements in scaling language models (Brown et al., 2020; Thoppilan et al., 2022) allowed to achieve novel discourse, reasoning and zero/few shot learning capabilities that can be applied to new tasks without/minimal additional training. Recent work aims to extend these achievements beyond text modality by defining static architectures for an extended subset of modalities (Alayrac et al., 2022;Reed et al., 2022). These are few examples of the ML models contributing to the line of research achieving incremental milestone toward AGI. Though, each model is trained from scratch with considerable resources consumption. The introduction of abstractions allowing to modularize, dynamically extend and reuse these large models may contribute to accelerate the rate of innovation.

B EXPERIMENTS DETAILS
All the experiments reported in this paper can be reproduced by using the following public resources: • We also publish the µ2Net+ checkpoint resulting from the large-scale multitask experiment reported in Section 5. This checkpoint can be used for inference on any of the 124 learned image classification tasks, or for further analysis, or even to be extended with additional tasks or methods. For information about the checkpoint and its license refer to: https://github.com/google-research/googleresearch/tree/master/muNet The experiments reported in Sections 4 have been executed on a TPUv3 (Jouppi et al., 2017) machine with 8 cores. While, the large scale experiments reported in Section 5 have been executed on a larger scale infrastructure using 32 TPUv4 chips in MegaCore mode, by using the Pathways orchestration layer (Barham et al., 2022), to accommodate for the increased memory requirements and scale. Table 3: Video representation of evolution of the large scale multitask system extended by each of the experiment segment described in Section 5 Segment Video 1: µ2Net 2 task iterations on task-set A https://youtu.be/hDDCMpfCJ3s 2: µ2Net+ 2 task iterations on task-set A https://youtu.be/tf5tzuxbxx8 3: µ2Net+ 1 task iteration to add task-set B https://youtu.be/jGkzXE2WLV0 4: µ2Net+ 2 task iterations on task-set A+B https://youtu.be/CW_BAwkQ9e8 Figure 4: Graph representing the architecture of the multitask system solving jointly 124 image classification tasks generated by the large scale continual learning experiments described in Section 5. Each task is identified with a unique color. Bottom triangular nodes represent the data input of each task. Top rectangular nodes represent the head layer of each task. Each edges sequence of the same color connecting a task input to its head defines the layers sequence composing the model for each task. Each path traverses a sequence of round nodes representing ViT L/16 internal layers in the following order from bottom to top: patch embedding, class token, position embedding and a variable number of transformer layers. Internal nodes are represented with the color of the task on which the parameters of the corresponding layer were trained last. Except for the gray nodes that have not received gradient updates from any of the 124 tasks and still carry the parameters of the root model that was loaded from a checkpoint of the ViT Large model pretrained on the imagenet-21k dataset as described by Gesmundo & Dean (2022b). See Table 3 for animations of the evolutionary process. The structure displays the formation of two main clusters of models/paths having a high degree of layers sharing. These are models selected for tasks that can achieve peak quality with a lower degree of knowledge specialization. Few models form smaller and disconnected clusters of paths. Notice that knowledge sharing is effective even for models/paths that result in a disconnected sub-graph, since ancestors models may have been trained on multiple tasks whose current best model has fully branched in a separate disconnected structure during the evolutionary process. Among these small clusters it is visible a high degree of sharing among related tasks. For example, the leftmost disconnected subgraph of 3 paths contains only characters classification tasks: bangla, devanagari and binary_alpha_digits (see Table 4 for datasets details). The majority of the disconnected subgraphs of 2 paths contain each a VTAB-Full task paired with the matching VTAB-1k "few-shots learning" version of the same task, from left to right: 1) clevr/count_all and clevr/count_all 1k , 2) dmlab and dmlab 1k , 3) dsprites/label_orientation and dsprites/label_orientation 1k , 4) eurosat and eurosat 1k , 5) resisc45 and resisc45 1k , 6) smallnorb/label_azimuth and smallnorb/label_azimuth 1k . Clone probability Figure 6: Displays the mutation probabilities learned by the µ function for cloning transformer layers at different depth, lower layer ids correspond to transformer layers closer to the model input. The displayed mutation probabilities are averaged across the values learned by the µ functions learned for the 124 models included in the multitask system at the end of the sequence of experiments described in Section 5. The red line is fitted to minimize the squared distance to the curve.   1 of 3). For each dataset used in the experiments, this table reports: 1) accuracy achieved on the test and validations sets by the large scale multitask model generated by the experiments described in Section 5 2) dataset name indicative of the Tensorflow Datasets Catalogs identification string and linking to the corresponding catalog page, 3) train, validation and test data splits, represented with the standard Tensorflow Datasets format ("validation" has been abbreviated as "val"). 4) corresponding scientific publication reference. Datasets are listed in the order of introduction into the system. Notes: [1] The test split of the imagenet_v2 dataset is used as validation set for imagenet2012.
[2] The test split of the cifar10_1 dataset is used as validation set for cifar10.
[3] The VTAB-full benchmark also includes the cifar100 task. Cifar100 has been introduced to the system as part of the initial benchmark. In the VTAB-full results tables we refer to the top 1 test accuracy achieved in the latest cifar100 training iteration without retraining it as part of the VTAB-full active training iteration.
[5] VTAB additional task, not included in the standard scoring set. These tasks were added to further scale the system and analyze transfer across related tasks.