{'fraction_non_alphanumeric': 0.027121825906530974, 'fraction_numerical': 0.015117083292164806, 'mean_word_length': 5.179181929181929, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Multi-Agent Framework for the Asynchronous and Collaborative Extension of Multitask ML Systems

Tradition ML development methodology does not enable a large number of contributors, each with distinct objectives, to work collectively on the creation and extension of a shared intelligent system. Enabling such a collaborative methodology can accelerate the rate of innovation, increase ML technologies accessibility and enable the emergence of novel capabilities. We believe that this can be achieved through the definition of abstraction boundaries and a modularized representation of ML models and methods. We present a multi-agent framework for collaborative and asynchronous extension of dynamic large-scale multitask intelligent systems.


INTRODUCTION
Traditional large-scale software systems enable teams of hundreds or thousands of software engineers to work collectively on a single software artifact, through decomposition of the problem into many sub-problems, and through well-defined abstraction boundaries. We currently lack the ability to have thousands of Machine Learning (ML) engineers and researchers collectively contribute to the continual extension of a single intelligent system. The presented work aims to define a methodology for the automatic incorporation of new tasks and knowledge into a single running system, and the asynchronous and continual development of distinct methods each capable of extending the system in both capabilities and structure. We see a direction where many individuals or teams can all contribute to the continuous improvement of a single intelligent system that is suited to a growing number of tasks and modalities, and that incorporates the learning and knowledge facilitated by the many people working on the same system each pursuing distinct objectives. Each user of the system can benefit from the knowledge, methods and structures already embedded in the system, and in turn can contribute to expanding them while pursuing their objectives. In addition to enabling thousands of engineers and researchers to contribute to a single system, it may even be possible to have tens of millions of people without ML knowledge to contribute enriching a single ML system, by providing new tasks and data while applying the skills that have been taught to the system by others.
We present a multi-agent framework that enables the exploration of the proposed methodology for collaborative and asynchronous extension of large-scale multitask dynamic ML systems.

METHOD
We introduce the proposed framework as an extension of the continual ML development methodology and the µ2Net+ method introduced by Gesmundo (2022). µ2Net+ is a method capable of evolving any modularizable ML model, such as deep neural networks, into a multitask ML system solving an unbounded set/stream of tasks. This method is based on an evolutionary approach that iteratively samples and mutates models by applying different types of mutation actions: architectural changes (e.g. insert or remove layers), knowledge transfer selection (e.g. choose from which other model transfer parameters and which subset to fine-tune), and hyperparameter tuning (e.g. select piece-wise constant schedules for the hyperparameters of each generated model and of the evolutionary method µ2Net+). Every mutated model is trained and scored on a single task, and can be subject to further mutations in future iterations. The best performing model for each task is retained in the multitask Improves Single program  Figure 1: Graphical representation of the single program sequential execution (left) and the proposed multi-agent parallel execution (right). The single program iterates over the set of tasks and improves one model at the time. Instead, multiple agents can be run in parallel, each works asynchronously to improve a substructure within the multitask system. In the example represented, each agent searches for an improved model of the assigned task.
ML system as part of a connected network of shared layers. For details of the baseline µ2Net+ refer to Gesmundo & Dean (2022b) and Gesmundo (2022).
In prior work, the µ2Net+ method has been applied with a single program sequential execution. This constrained the method to search for an improved model for only one ask at the time, the active task. The single program is able to find the best model for each task by activating them one at the time, and searching for an improved model for the active task while keeping immutable all the other models currently composing the multitask system. With task-set iteration we refer to the process of performing one active task iteration on each available task. Multiple task-set iterations can be performed to achieve a higher degree of knowledge transfer.
The framework proposed in this paper allows for multiple programs, agents, to jointly extend a shared multitask ML system, by each asynchronously modifying distinct substructures of the system's architecture. In the presented instantiation of this framework, each agent is assigned a single task and applies the µ2Net+ method with the objective of finding an improved model for the assigned task. In this context, a task-set iteration can be achieved by having each agent perform one task iteration on the assigned task in parallel. Notice that, even with this asynchronous execution, µ2Net+ guarantees immunity against catastrophic forgetting, negative transfer and gradient interference. Because, the immutability of parent models is still guaranteed during the mutation and training of child models. In the following paragraphs we highlight a few additional properties achieved by the proposed multi-agent framework.
System scaling A capability enabled by the proposed framework is the possibility to achieve system scaling through agent parallelism. Different scaling techniques have become commonly applied to increase the compute resources accessible to ML model with the aim of increasing its size or accelerate its training on bigger datasets. Data parallelism techniques aim to replicate the model on multiple machines while synchronizing gradients across replicas (Bottou, 2010;Li et al., 2014). Model parallelism techniques allow to parallelism the execution of large layers across parallel hardware requiring to gather/distribute the output/input of each layer shard (Dean et al., 2012;Krizhevsky, 2014). Pipelining techniques allow to place continuous sequences of layers on different machines but require extra communication between the pipeline stages and idle machine cycles for stages waiting for input being propagated in both forward and backward DNN training phases Narayanan et al., 2019).
The proposed framework allows for a different dimension of scaling that we can refer to as agent parallelism. In practice, more compute resources can be employed to accelerate the training throughput by running in parallel multiple agents instead of scaling the single program with any of the common scaling technique. Notice that, agent parallelism does not require any communication overhead since each agent works asynchronously and just updates its working image of the overall system state by reading it from disk on a need basis (i.e. system state image is updated at the beginning of each evolutionary iteration when structures generated by other agents could be sampled as parent models). In multi-agents experiments executed on TPUv4 with Megacore configuration optimized for dense computation (i.e. repurposing resources normally allocated for sparse computation), we have recorded an average of 99.2% active TPU duty cycles, that normally would not be expected to be achievable with the common scaling techniques due to the additional synchronization required. The properties of system scaling through task parallelism are further analyzed in Section 3. Notice that, the different scaling techniques mentioned are not exclusive, rather any subset of those techniques can be combined, e.g. Park et al. (2020).
Method heterogeneity In the described instantiation of the proposed framework, all agents apply the same µ2Net+ method to tasks that all have the same image classification framing. Though, in general, the framework supports agents differing in any aspect of the method such as: reward function, input/output task modalities, root model, space of generatable architectures, mutation actions and hyperparameters search space or even search method, for example employing Reinforcement Learning or Bayesian methods instead of an Evolutionary approach.
Hardware heterogeneity The general principles of the proposed framework and the particular implementation we provide 1 , allow for each agent to be run as a distinct program on a different type of hardware. In a preliminary experiment, we have tested the hardware heterogeneity capability by generating a multitask system solving three tasks with three distinct agents each run on different hardware type ( Figure 1). The first agent was run on a TPUv3 machine with 8 cores (thus sampling and training 8 models in parallel). The second agent was run on a Tesla P100 GPU machine with 1 core. While, the third agent was run on a single CPU. Furthermore, each agent was executed on a different data-center. This capability is achieved with no software customization, since each agent runs as a distinct program and synchronizes only via the sharded system state stored on disk, i.e. cloud storage. This hardware heterogeneity capability allows to modulate the resources assigned to each agent according to availability and requirements of different system contributors (e.g. different teams or individuals), that can also change through time.
Collaborative ML development The proposed framework provides the abstractions enabling the novel collaborative ML development methodology. This framework allows multiple users, each aiming to solve a different research or applied problem, to introduce an agent designed to solve the task at hand with improved quality and efficiency by leveraging the knowledge and methods already embedded in the shared multitask dynamic ML system, and in turn, each new agent enriches the ML systems with new knowledge, structures and capabilities.

EXPERIMENTS
This section reports an empirical analysis of the properties of the proposed multi-agent framework employed as a scaling technique. We compared against the baseline "single program" sequential execution.
To validate the generality of the analyzed properties, the experiments are repeated on two benchmarks: the Multitask Character Classification Benchmark (MCCB) and the Visual Domain Decathlon Benchmark (VDDB). The MCCB is defined by a set of eight character classification tasks similar in domain type, while VDDB is composed by ten tasks with larger datasets that have been explicitly selected to each represent as distinct domains as possible (for tasks details and references see Gesmundo & Dean (2022a)).
The experiments representing the single program sequential execution apply the µ2Net+ method with configuration equivalent to that described in Gesmundo (2022 Single program Multi-agent a checkpoint pretrained on Imagenet 21k as described in Steiner et al. (2021). The only difference, compared with the configuration used by Gesmundo (2022), is that the cost scale factor is fixed to 1 and the number of task-set iterations is set to 10 for both benchmarks.
The experiments representing the multi-agent execution use the same µ2Net+ configuration and equivalent training compute budget in terms of total training steps per tasks. The only difference, compared with the single program execution, is that each task-set iteration is executed by a pool of agents in parallel. In more details, we create one agent per task, each agent performs one task iteration on the assigned task by applying the same µ2Net+ evolutionary method to find an improved model for its task asynchronously and in parallel to the other agents. The only constraints imposed is that agents cannot start the n th task iteration until all the agents have completed the (n − 1) th iteration. Notice that agents iteration time may vary since the µ2Net+ method allocates a bigger train budget to tasks with bigger dataset, since that leads to a longer train epoch. Thus, in this specific application, the end-to-end completion time is bottlenecked by the speed of the slowest agents.
The results of these comparison experiments are displayed in Figure 2. In both benchmarks it is noticeable that the multi-agent execution achieves a significantly lower accuracy after the first task-set iteration. During the very first iteration, the parallel agents can generate models that transfer knowledge only from the root model. This leads to reduced quality compared to the sequential approach that allows for knowledge to be transferred also from all the tasks that are preceding in the sequence. In few iterations, the gap between the quality achieved by the two method is reduced until the quality deltas are withing statistical noise range. The initial gap on the MCCB is significantly larger, this could be justified considering that the MCCB tasks are similar in domain (all character classification tasks) so they can be expected to suffer more from reduced knowledge transfer in the initial iteration compared to VDDM tasks that have explicitly selected to represent a set of domains as different as possible.
The proposed multi-agent execution achieves a sharp reduction in execution time (Figure 2 right).
The MCCB experiments result 6.64 times faster. Notice that, as MCCB is composed by 8 tasks, and its experiments are parallelized with 8 agents, thus we could expect to achieve an upper bound of 8 times speed up for methods constraining all the agents to use the same training budget. The VDDB experiments result 7.66 times faster, while its speed up upper bound is 10.

RELATED WORK
This paper contributes to a line of research aiming to define and demonstrate a novel ML research methodology and methods that enable the creation of dynamic large-scale multi-modal multitask intelligent systems that can be indefinitely and collaboratively extended: (Gesmundo & Dean, 2022a;Gesmundo, 2022).
The proposed method is designed to learn an unbounded number of tasks in a continual learning fashion. In such contexts it aims to learn each task with higher quality and efficiency by automating and optimizing the knowledge transfer among any subset of tasks that can provide useful knowledge to one another. The proposed model is designed to be immune from common multitask learning pitfalls: catastrophic forgetting, gradients interference, negative transfer. Cross-task transfer-learning has gained popularity, especially through transfer learning from a model pre-trained on a large amount of data for one or a few general tasks, and then fine-tuned on a small amount of data for a related downstream task. This approach has been shown to be very effective in a wide variety of problems across many modalities, including language (Devlin et al., 2019;Raffel et al., 2020) and vision (Dosovitskiy et al., 2021;He et al., 2016). The success of transfer-learning applications hinges on adequate prior knowledge selection to avoid typical negative transfer pitfalls (Rosenstein, 2005;Wang et al., 2019). Common solutions rely on data or model selection techniques, often putting emphasis on the efficiency of the exploration (Zhang et al., 2020;Mensink et al., 2021), also method aiming to automate knowledge selection at a layer level have been proposed Sun et al. (2020a). Transfer learning capabilities are critical for multitask models. ML models trained jointly on multiple tasks can be affected by gradients interference if any subset of parameters receive gradients jointly from multiple sources Yu et al., 2020), and by catastrophic forgetting of prior knowledge as new tasks are learned (McCloskey & Cohen, 1989;French, 1999). These knowledge loss problems can be alleviated with weighted combination of tasks (Liu et al., 2019b;Sun et al., 2020b) and gradient transformation methods Sener & Koltun, 2018;Kendall et al., 2018). Stronger guarantees are provided by methods that compartmentalize task specific knowledge in dedicated parameter subsets (Rebuffi et al., 2017;Houlsby et al., 2019;Rusu et al., 2016;Rosenfeld & Tsotsos, 2020). Addressing catastrophic forgetting and identifying what subset of parameters/knowledge that is beneficial to share with each task is also critical for continual learning or life long learning methods (McCloskey & Cohen, 1989;French, 1999;Ramesh & Chaudhari, 2022).
The proposed method relies on an evolutionary approach to jointly search the spaces of models architectures, hyperparameters, and prior knowledge selection while optimizing for an possibly multi-factor non-differetiable reward function. The automation of hyperparameter tuning has been commonly addressed with Bayesian optimization (Srinivas et al., 2010;Bergstra et al., 2011;Snoek et al., 2012), evolutionary methods have also been explored for this purpose (Jaderberg et al., 2017;Zhang et al., 2011). Hyperparameters tuning can be considered related to the neural architecture search (NAS), as architectures can be defined by the selection of a sequence of architectural hyperparameters. Initially, NAS methods have been based on reinforcement learning techniques (Zoph & Le, 2017) but also sample efficient evolutionary approaches have been proposed (Real et al., 2019;Maziarz et al., 2018). Parameter-sharing based NAS methods aim to reduce the typically high training cost (Pham et al., 2018;Liu et al., 2019a;Kokiopoulou et al., 2019). Optimization for multi-factor quality/cost trade-offs have been explored (Tan et al., 2019).
The proposed method is capable to dynamically extend the system, adding capacity or novel structures in an unconstrained fashion. A few methods have been proposed to achieve dynamic architecture extensions (Chen et al., 2016;Cai et al., 2018), some also focusing on an unbounded stream of tasks (Yoon et al., 2018;Yao et al., 2020), or achieving immunity from catastrophic forgetting (Rusu et al., 2016;Li & Hoiem, 2018;Li et al., 2019;Rosenfeld & Tsotsos, 2020).
The proposed method is sparsely activated, thus the unbounded growth of knowledge and parameters is decoupled from the growth of computational cost. The growth in capabilities of state of the art models often requires growth in terms of trainable parameters (Kaplan et al., 2020). Sparse activation techniques at sub-layer level (Shazeer et al., 2017;Du et al., 2021) or network route level  allow to decouple model size growth from compute cost. This is achieved by integrating a routing technique that selects the appropriate subset of parameters storing the most relevant knowledge for each task, sample or token/patch.
The ability of jointly solve a large amount of tasks is commonly associated with progress toward Artificial General Intelligence (AGI). Advancements in scaling language models (Brown et al., 2020;Thoppilan et al., 2022) allowed to achieve novel discourse, reasoning and zero/few shot learning capabilities that can be applied to new tasks without/minimal additional training. Recent work aims to extend these achievements beyond text modality by defining static architectures for an extended subset of modalities (Alayrac et al., 2022;Reed et al., 2022). These are few examples of the ML models contributing to the line of research achieving incremental milestone toward AGI. Though, each model is trained from scratch with considerable resources consumption. The introduction of abstractions allowing to modularize, dynamically extend and reuse these large models may contribute to accelerate the rate of innovation.

CONCLUSION
We introduced a multi-agent framework for collaborative and asynchronous extension of large-scale multitask dynamic ML systems. We have demonstrated how the framework can be employed to achieve hardware heterogeneity and efficient scaling for a multitask system. Future work can continue to build toward an extended ecosystem of agents implementing different methods and tasks varying in type and modalities, further extending a dynamic large-scale ML system.