{'fraction_non_alphanumeric': 0.03198082803074126, 'fraction_numerical': 0.011197421700685894, 'mean_word_length': 5.428419654714475, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adaptor: Objective-Centric Adaptation Framework for Language Models

This paper introduces Adaptor library, which transposes traditional model-centric approach composed of pre-training + fine-tuning steps to objective-centric approach, composing the training process by applications of selected objectives.We survey research directions that can benefit from enhanced objective-centric experimentation in multitask training, custom objectives development, dynamic training curricula, or domain adaptation.Adaptor aims to ease reproducibility of these research directions in practice. Finally, we demonstrate the practical applicability of Adaptor in selected unsupervised domain adaptation scenarios.


Introduction
Recent development in Natural Language Processing (NLP) heavily benefits from a high level of maturity of open-source frameworks, such as Fairseq (Ott et al., 2019) or HuggingFace Transformers (Wolf et al., 2020). Thanks to the standardized interfaces, these libraries allow for immediate experimentation with the most recent research results, practically fostering the speed of further progress in the area. While their use is seamless for countless conventional use-cases of transformer models and fine-tuning to a specific end-task (Devlin et al., 2019;Radford and Narasimhan, 2018), divergence from this framework requires feasible, but elaborate and complex customizations, increasing the risk of logical errors and complicating the reproducibility of experiments. A characteristic group of problems 1 github.com/gaussalgo/adaptor requiring significant changes to the standard pipeline are multi-step and multi-task adaptations. This paper introduces the AdaptOr library, which aims to simplify the more complex training processes that their training objectives can easier describe. AdaptOr challenges the conventional modelcentric framework, where data and task selection are constrained by the requirements of the selected language model architecture. Instead, it introduces an objective-centric training pipeline, with Objective as the central abstraction of the process.
The AdaptOr framework aims to help NLP researchers and practicioners engage in projects that include any of the following: • Multi-objective training: when training a language model on more than one task or data set, including languages, AdaptOr can signif-icantly simplify the custom code base that needs to be implemented. Even if the objective is custom, the user can avoid adjustments to other parts of the training pipeline.
• Custom data schedule: when users need to perform dynamic data sampling, AdaptOr allows them to implement a custom Schedule (see Figure 2), leaving the data and model adjustment logic intact. This simplifies systematic experimentation and reproducibility, and minimizes the risk of errors.
• Objectives design & evaluation: AdaptOr exposes top-level declaration of training objectives, which enables easy experimentation with custom objectives. Objective-level monitoring can provide custom behavioural insights and allows for pruning less promising experiments earlier in the lengthy training process, saving computational costs.
• Robustness evaluation: The objectivecentric paradigm provides an easy robustness estimation by evaluating on out-ofdistribution samples. In the standard sequential adaptation scenario, objective-centric evaluation exposes characteristic flaws of adaptation, like exposure bias or catastrophic forgetting.
This paper is structured as follows: Section 2 provides an overview of recent work demonstrating the potential of multi-objective training in domain and task adaptation. Section 2.4 also describes other software frameworks applicable for similar use cases. Section 3 describes the design of AdaptOr, showing the users how to confidently integrate novel objectives and schedules. In Section 4, we describe and implement a set of non-trivial, yet promising domain adaptation experiments using AdaptOr and collect their results. As AdaptOr remains under active development, we close in Section 5 with an outline of the upcoming features. We welcome contributions of novel objectives and schedules.

Background
This section provides an overview of recent work that demonstrates the potential of multi-objective training and schedules that motivated the design of AdaptOr. Our overview consists of a nonexhaustive list of applications that AdaptOr aims to make more accessible for practical use and in future research.

Multi-Task Training
Multi-task training has a long history in both traditional machine learning (Caruana, 1997) and in deep learning (Crawshaw, 2020). This section describes examples of multi-task (i.e. multi-objective) training, outlining its benefits and potential.
Under some circumstances, multi-task training enhances distributional robustness of neural models. Tu et al. (2020) demonstrate this on adversarial data sets, exposing common heuristic biases of the language models (McCoy et al., 2019). Enhanced model generalization can also be achieved by introducing one or more latent tasks that do not directly correspond to the end task but reflect specific desired properties of the model. One of a few studies in this direction is Sharpness-Aware Minimisation of Foret et al. (2021), performing multi-objective training on image classification using cross-entropy and a novel, sharpness-aware objective, reflecting the model's monotonicity on the local neighborhood. In context of Neural Machine Translation (NMT), Wang and Sennrich (2020) incorporate Minimum Risk Training (MRT) objective (Ranzato et al., 2016), optimising an arbitrary sequence-level measure of outputs. In composition with the traditional token-level cross-entropy objective, MRT improves distributional robustness.
By aggregating multiple objectives, Xie et al. (2019) show that combining sentence classification objective with maximizing representation consistency to augmented samples fosters data efficiency.
The intuition on the benefits of multi-task training presumes that by optimizing the training by multiple cost functions, the final model is less prone to the weaknesses of a specific task (Collobert et al., 2011), possibly reflecting on higherlevel, task-invariant properties of language (Bengio et al., 2013).

Data-Sampling Schedules
Exposing a model to training samples in a systematic schedule, also referred to as a curriculum, can lead to an improvement of the accuracy of the final model (Bengio et al., 2009). While the positive effects of more complex schedules based on sample "difficulty" with transformers remain to be explored, multiple studies show the potential of confidence-based sampling to improve accuracy and generalization. Biased samples can be identified, according to model's confidence (Pleiss et al., 2020; or using Bayesian methods such as the Product of Experts (Hinton, 2002). Then, they can be either eliminated (Bras et al., 2020) or downweighted (Utama et al., 2020).
More complex scheduling methods are applied in training NMT models. Bengio et al. (2015) use decay schedule to sample from both references and the previous outputs of a NMT model, minimizing the discrepancy between training and inference. Zhang et al. (2019) successfully use the same sampling strategy in a sequence-level objective. The results of Lu et al. (2020) underline the potential of sampling in NMT training, suggesting that the accuracy of transformers on reported MT benchmarks can be outperformed by simpler RNN models by combining objectives in decay schedule.
Despite the reported improvements, we find that custom scheduling strategies are rarely used. We attribute this to their complicated integration into the standard training process. To foster the research and applicability of scheduling methods, AdaptOr makes the implementation of custom scheduling strategies easy, comprehensible, and reproducible.

Domain Adaptation
Objective-centric frameworks are well-suited for domain adaptation techniques, where AdaptOr provides support for combining traditional end-task objectives with unsupervised adaptation or auxiliarytask objectives in a user-selected schedule. The goal of domain adaptation is to maximize performance on a specific data domain, often denoted as the adapted or target domain (Saunders, 2021).
Perhaps the most common adaptation approach using pre-trained language models is to continue pre-training on unsupervised samples of the adapted domain (Luong and Manning, 2015;Beltagy et al., 2019). This approach has been successfully extended in various directions. For instance, Gururangan et al. (2020) show that adapting to a shared task on different domain can enhance accuracy of the eventual application. If supervised data is sparse, other auxiliary tasks, described earlier in Section 2.1, can be used as concurrent objectives (Xie et al., 2019).
In cases where larger volumes of data of given task is available in a different language, adaptation using cross-lingual transfer can be considered. Pretrained language models show that cross-lingual transfer works well with large-data unsupervised objectives (Conneau and Lample, 2019), but it can also be applied for low-resource supervised objective, such as very low-resource translation (Neubig and Hu, 2018).
If even unsupervised target-domain data is sparse, another option is to subset arbitrary unsupervised sources to automatically identify samples of adapted domain, by applying domain classifier (Jiang and Zhai, 2007;Elsahar and Gallé, 2019). If the boundary between the training and the adapted domain is known, an auxiliary objective can minimise a discrepancy of representations between the training and possibly low-resource target domain (Chadha and Andreopoulos, 2018).
Despite the possibilities, adaptation can also introduce undesired biases. In the scope of NMT, adaptation can cause problems of "catastrophic forgetting", when the model experiences performance degradation on the originally well-performing domains (Saunders, 2021), or "exposure bias", when the model overfits the non-representative specifics of the target domain, such as the artifacts of data collection (Ranzato et al., 2016). Additionally, by normalizing a single type of bias, such as lexical overlap (McCoy et al., 2019), the model might degrade its accuracy on other domains (Utama et al., 2020). Addressing multiple biases concurrently (Wu et al., 2020) can mitigate this problem.
AdaptOr allows the knowledgeable user to construct a reproducible and robust adaptation pipeline using native multi-objective evaluation. Covering multiple domains in separate objectives, AdaptOr can expose the above pitfalls, without the need to implement complex separate evaluation routines.

Related Software Frameworks
The Adapters architecture (Houlsby et al., 2019), having only a small set of parameters, might be a good fit when performing adaptation of transformer with modest hardware or data. Recently, the AdapterHub library (Pfeiffer et al., 2020) makes training and sharing of Adapters convenient. Compared to AdaptOr, AdapterHub does not provide support for more complex adaptation cases, such as using multiple objectives, scheduling, or extended evaluation. However, since both libraries build upon the HuggingFace Transformers library (Wolf et al., 2020), their close integration is feasible.
If the robustness of models to heuristic shortcuts ( Robustness Gym library (Goel et al., 2021) provides a comprehensive evaluation over an extendable set of different kinds of heuristic biases. Robustness Gym provides much deeper evaluation compared to AdaptOr Evaluators, and could be integrated as an AdaptOr Evaluator. Unlike Robustness Gym, AdaptOr enables an evaluation of robustness also on generative tasks, with specified out-of-domain data sets.

AdaptOr Design
This section describes the structure and functions of the AdaptOr framework. We introduce its primary components bottom-up. Figure 3 depicts the relations of these components and compares user interaction with the traditional model-centric pipeline.

LangModule
A LangModule instance provides a management of inputs, outputs and objective-specific model components, referred to as heads. Once an objective with given LangModule is instantiated, an objective-compatible model is either initialised, or given by the user (see Section 3.2) and the parameters of this model are merged with the parameters of the previously-registered objectives.
The merge works as follows: If no previous objective was registered, then the model of the given objective is considered a base model. The models of the second-and later-registered objectives are then merged with the base model: first, pairs of PyTorch modules of the same name in the base and the new model are identified. If the dimensions and weights of these modules match, the respective module of the newly-adding model is replaced with a module of the base model.
In the case of pre-trained transformers, the weights of heads are initialized randomly by default, resulting in a registration of a distinct head for each objective and sharing the remaining parameters. Users can control which parameters (not) to merge by explicitly setting their respective weights as (non-)equal.
It is possible to use LangModule with any Py-Torch module that uses a HuggingFace tokenizer, compatible with the given neural module. Therefore, LangModule is also suitable for other models such as recurrent networks.

Objective
Objectives are the primary component of AdaptOr's training pipeline. Most importantly, an Objective serves two functions: sample encoding and loss computation. By implementing these and choosing the type of a model's head, AdaptOr users can define and experiment with novel training objectives. If they additionally provide an explicit definition of the Objective's model (the objective_module attribute), the new objective does not even have to comply with common model heads; shared parameters of the given objective_module would still be merged with the given lang_module.
If no objective_module is given, the Objective will request that a LangModule assigns the Objective a module of the Objective's default compatible_head (see Section 3.1).
Additionally, every Objective instance performs its own logging, evaluation, and state updates, such as its convergence, based on a valuation of given val_evaluators, or draws a progress bar, based on the state of its sample iteration. However, the training flow is guided by a Schedule (see Section 3.3). Objectives can implement custom data sampling, but if possible, we recommended to do so in a custom Schedule instance.
Since data encoding is also objective-specific, Objectives expose a higher-level user interface of data inputs than other frameworks: instead of encodings, users provide an Objective with a texts_or_path and a labels_or_path containing raw texts and respective labels. AdaptOr provides an implementation of standard Objectives for sequence and token classification and sequenceto-sequence tasks. When implementing a custom Objective, note that sampling and encoding are performance bottlenecks on current high-end GPUs.

Schedule
Schedules control the training flow through the interfaces provided by HuggingFace Transformers library. Primarily, they deliver 1) a set of standard stopping strategies based on the state of the Objectives and 2) an IterableDataset instance, constructed by sampling Objectives according to a sampling strategy implemented in its _sample_objectives. A Schedule also ensures that outputs of distinct lang_modules' heads are delivered to the respective Objectives for loss computation. This relatively complex sampling framework provides a very simple interface for custom Schedule implementations (see Section 2.2). For instance, a pre-defined ParallelSchedule is implemented with three lines of code (see Figure 2).

Adapter
An Adapter is customization of the HuggingFace Trainer with only minor adjustments. Specifically, Adapter redirects loss computation to a Schedule, which further distributes outputs to corresponding Objectives and extends native training logs with logs of Objectives' Evaluators. Furthermore, Adapter adjusts persistence of the models so that a model of every head can be reloaded without the use of AdaptOr, by simply using HuggingFace Transformers' AutoModelForXY.from_pretrained.
Based on the actively-developed HuggingFace Transformers library, the AdaptOr allows its users to benefit from all other native features of Hugging-Face Transformers, such as the support for the most recent models, custom logging platforms, or dis-tributed parallel training. Furthermore, it can simplify integration with other custom libraries (see Section 2.4).

Experiments
We use AdaptOr in a set of domain adaptation experiments for a machine translation use-case, aiming to answer the following research question: How well can unsupervised objective(s) substitute labeled parallel data. In our methodology, we permute the easily-configurable parts of AdaptOr's training configuration 2 and compare the results of the resulting model to a baseline adaptation scenario. We experiment with an architecture identical to the base model of Vaswani et al. (2017), with a configuration of Junczys-Dowmunt et al. (2018).
Data. We train the model on English-to-Czech translations on different domains of OPUS (Tiedemann, 2012) chosen for their significant distinctiveness: we use Wikimedia as a large-scale, supervised domain (denoted as in-domain, i.e. ID), OpenSubtitles as an Adapted Domain (AD) and Bible for the evaluation of a model's robustness on Out-Of-Domain (OOD) samples.
Pre-training vs. fine-tuning. We simulate two basic scenarios: training the model from a random initialization and fine-tuning the existing translation model with no control over its pre-training data. In the latter cases, we perform fine-tuning from the checkpoint of Tiedemann and Thottingal (2020).
Schedules. We implement and experiment with two objective schedules: i) Sequential schedule, sampling and differentiating the model sequentially by each objective until convergence by evaluation loss, or for a maximum of 100,000 updates. ii) Parallel schedule, concurrently sampling training batches uniformly from every given objective.
Using gradient accumulation, we differentiate the model based on all given objectives. We perform updates until the convergence of all objectives, or for a maximum of 50,000 updates for each objective.
Objectives selection. We implement and experiment with the following AdaptOr objectives: • Sequence-to-sequence (seq2seq) objective, as introduced by Vaswani et al. (2017), maps a combination of encoder inputs in the source language and previously-generated outputs as decoder inputs to a distribution over the nextpredicted tokens.
• Denoising objective introduced by Lewis et al. (2020) is an unsupervised instance of the seq2seq objective that performs random token permutation on the input and trains the model to map such 'noisy' text to the original version of the input. We use this objective on the target-data domain to enhance its comprehension by the model.
• Back-translation objective, as used e.g. by Sennrich et al. (2016) is also an unsupervised seq2seq objective, which uses an external translator in reverse direction to obtain pseudo-inputs. This objective is profitable when we have unlabeled data of the target domain.
Using these components, we construct the following experiments: • Baselines: pre-training (1) and fine-tuning (6) on ID data from a domain different from the Application Domain (AD) using a single traditional seq2seq objective.
• Sequential adaptation: we pre-train using seq2seq on ID and afterwards adapt using either unsupervised Back-translation (2), or supervised seq2seq (3) on AD to quantify the unsupervised adaptation gap.

Results
Table 1 evaluates the base transformer after the given number of updates on held-out deduplicated validation splits of In-Domain (ID), Adapted-Domain (AD), and the third Out-Of-Domain (OOD) data. Note that the results for the BLEU score are properly comparable only within the same domain.
We observe that the model trained on a single domain (1, 6) degrades on all other domains. In a pre-training scenario, domain robustness improves when incorporating data of adapted domain in any objective. However, in a sequential schedule, we observe catastrophic forgetting towards any mostrecent domain of adaptation (2, 3). This is improved by using the Parallel schedule for a negligible price of in-domain accuracy (4, 5).
In the fine-tuning scenario, we show that incorporating unsupervised Back-translation to AD (7,9) improves ID BLEU comparably to supervised adaptation (10). Interestingly, Denoising on AD (8) improves in-domain performance but seems less efficient than Back-translation.

AdaptOr Usage Complexity
To give an idea about the relative complexity of using AdaptOr as compared to model-centric frameworks, we compare selected measurable code features of the complexity of our experimental implementation to an example implementation using the HuggingFace Trainer 3 . We pick the experiment of supervised pre-training + unsupervised fine-tuning, including evaluation, in the sequential schedule (2), as this can still be addressed using HuggingFace Transformers relatively easily; Implementing the parallel multi-objective schedule in the Transformers framework would require major customisations of selected model and Trainer objects.
The training script using HuggingFace Trainer contains 654 lines of code, 135 variable assignments, 186 method calls and the initialisation of 9 custom objects. Additionally, in the pre-training + fine-tuning framework, this script has to be run twice, initialising the second training from the selected checkpoint of the first one, with updated configurations. Back-translated pseudo-labels are generated by a different script, not included in this assessment.
Using AdaptOr, we construct an equivalent routine from the provided demo script. Our implementation contains 124 lines of code, 31 variable assignments, 37 method calls and the initialisation of 14 custom objects. Despite its brevity, our script wraps the whole training process, and hence, together with the associated version of AdaptOr or its fork, it provides a reproducible fingerprint of the experiment.

Conclusion and Future Work
This paper introduces the AdaptOr library, which provides objective-centric training framework wellsuitable for multi-task and multi-domain training scenarios, and the development of novel objectives and sampling schedules. We find that even in the conventional single-objective training routines, AdaptOr can reduce volumes of custom implemen-tation and increases readability and reproducibility. Having used AdaptOr already for several production use cases, we are happy to share it with the NLP community.
Our future work aims to further enhance AdaptOr's user comfort with existing and novel unsupervised objectives, dynamic schedules, and demonstrations on novel use cases.

Broader Impact
Thanks to the ubiquity of objective-centric training, AdaptOr can accelerate the applicability of the most recent research in multi-task and multilingual modeling and enrich the research with the practical experience of the industry.
We further identify the benefits of AdaptOr's definite training pipelines in saving unnecessary financial and environmental expenses of reproducing the reported results of large language models, otherwise often including expensive hyperparameter optimization over unreported parameters. Due to these aspects, AdaptOr could also ease the spread of state-of-the-art language technologies to underresourced languages and more specialized domains with a sufficient amount of unsupervised sources.
Finally, objective-centric training might help expose the potential of unsupervised objectives to the generalization and interpretability of models. AdaptOr can foster the research in unsupervised learning by lowering the relatively high entry level of technical proficiency needed for experimentation with novel language objectives.