{'fraction_non_alphanumeric': 0.031154551007941355, 'fraction_numerical': 0.009227405716490372, 'mean_word_length': 4.858730457713317, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unsupervised Domain Adaptation for Text Classification via Meta Self-Paced Learning

A shift in data distribution can have a significant impact on performance of a text classification model. Recent methods addressing unsupervised domain adaptation for textual tasks typically extracted domain-invariant representations through balancing between multiple objectives to align feature spaces between source and target domains. While effective, these methods induce various new domain-sensitive hyperparameters, thus are impractical as large-scale language models are drastically growing bigger to achieve optimal performance. To this end, we propose to leverage meta-learning framework to train a neural network-based self-paced learning procedure in an end-to-end manner. Our method, called Meta Self-Paced Domain Adaption (MSP-DA), follows a novel but intuitive domain-shift variation of cluster assumption to derive the meta train-test dataset split based on the self-pacing difficulties of source domain’s examples. As a result, MSP-DA effectively leverages self-training and self-tuning domain-specific hyperparameters simultaneously throughout the learning process. Extensive experiments demonstrate our framework substantially improves performance on target domains, surpassing state-of-the-art approaches. Detailed analyses validate our method and provide insight into how each domain affects the learned hyperparameters.


Introduction
Given enough supervision, modern deep learning models can learn a new task with great accuracy. However, in many practical settings, the goal is to adapt to a new domain in which there is a different in data distribution between training and testing processes. This poses a major challenge for standard natural language systems due to both the intrinsic variation of linguistics (e.g., lexical shift, semantic shift) as well as the extrinsic factors such as how textual datasets are collected and annotated. For example, a model trained to predict news events may easily recognize, from medical domain, "died" as an event, but would not be able to detect obvious events such as "mutation" or "cancer". Such a model may even fail to generalize to closer adaptation settings (e.g. news from different times and sources).
The majority of existing unsupervised domain adaptatiopn (UDA) approaches combined various training objectives to align different aspects of domain-specific extracted features. In particular, the most prominent approach is domain-adversarial neural network (DANN) (Ganin et al., 2016) that employs a domain-adversarial training procedure between a domain classifier and the network's feature extractor to learn a discriminative and domaininvariant joint feature representation. The simplicity of DANN allows researchers to incorporate it with multiple other objectives such as semisupervised learning (SSL) regularizers (Shu et al., 2018), discrepancy metrics (Long et al., 2015), cotraining (Kumar et al., 2018), and auxiliary tasks (Bousmalis et al., 2016). Each of them plays an important role in enhancing domain adaptation ability of models in the current state-of-the-art methods. However, it is not trivial to apply these techniques to textual tasks, where large transformer-based language models are essential to achieve top performance, because of the time and resource required to fine-tune and balance the effects of these terms for multiple different adaptation scenarios.
Meta-learning (ML) framework is an effective solution for the problem of hyperparameter optimization (Franceschi et al., 2018;Behl et al., 2019). Furthermore, it has been widely applied by recent works on Domain Generalization (DG) Dou et al., 2019), in which a learning procedure similar to that of Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017) is leveraged to simulate the domain shift in train-test datasets by a virtual meta train-test set created from data drawn only from source domains. Though DG and UDA share close similarities, the final goal of each learning setting is different. More importantly, the MAML procedure is not applicable for UDA problem because of the lack of a clean validation dataset for meta-test step.
To this end, we propose to dynamically partition the training source data into a low-loss meta-source domain and a high-loss meta-target domain, inspired by self-paced learning (SPL) approach (Kumar et al., 2010). Our framework, called Meta Self-Paced Domain Adaptation (MSP-DA), employs a neural-SPL module to control the data selection process for meta train-test set using a learnable age hyperparameter as threshold while also introducing optimized weighting mechanisms for each of the combined loss' terms, including instancewise weighting for the main classification task and layer-wise weighting for domain alignment losses. The weighted objectives on meta-source domain are minimized in meta-train step in a direction such that also leading to improvement in model's predictions on meta-target domain. During the learning process, parameters and age threshold of the neural-SPL module are updated based on model's evaluation performance in meta-test step, resulting in tuned weighting coefficients and learning schedules similar to that of a standard hyperparameter tuning process. To our knowledge, this is the first work to devise a neural network-based SPL method, in which both the sample weightings/selections and the age hyperparameter are dynamically optimized, generalizing previous works which require heuristic age schedule and complicated mathematical derivation for the corresponding instance weighting.
While the meta-target set does not contain samples from the true target domain, we argue that our formulation is beneficial for UDA because of the two following reasons. First, the proposed partition can result in two virtual domains with a significant discrepancy, and through learning to address in this hard setting that the model would gain the ability to adapt to other, possibly easier, domains. Another reason is based on the cluster assumption from SSL methods (Chapelle et al., 2006), which states that data points of the same class should concentrate around the same cluster, effectively forming a high-density low-loss region. In case of adapting between two highly dissimilar domains, these regions may get shifted significantly, as a consequence low-loss regions of target domain may contain considerable intersection with high-loss regions of source domain, as illustrated in Fig. 1. In other words, by learning to adapt the high-loss meta-target domain, the model would also be able to generalize to a significant portion of the true target domain.
We provide extensively evaluation of the proposed framework on the standard UDA benchmarks -FDU-MTL dataset for sentiment analysis task, along with additional results for event detection task on ACE-05 dataset, which is a much harder adaptation setting. Ablation studies and detailed analyses are conducted to validate each main component of our model and provide insights for future researches.

Related Work
Unsupervised Domain Adaptation for Text Classification The main line of research on UDA focuses on learning domain-invariant, which is either achieved by explicitly reducing the distance between source and target feature space measured by some distribution discrepancy metric (Long et al., 2015;Zellinger et al., 2017), or by adversarial training in which the feature extractor is trained to fool a domain classifier, both are jointly optimized to arrive at an aligned feature space (Ganin et al., 2016). We focus on applying the latter in transformerbased model (BERT) (Devlin et al., 2019) for textual tasks. Previous works have provided empirical results on different domains (Wright and Augenstein, 2020;Lin et al., 2020), different tasks (Naik and Rosé, 2020;Du et al., 2020), most of which presented little to no improvement following the standard domain adversarial training framework.
We further verify this point in our baseline performances.
Sample Weighting There are two main research directions to adaptively output weight of a sample during training process: addressing class imbalance by monotonically increasing function that imposes larger weights to ones with larger loss values (Sun et al., 2007;Lin et al., 2017), and suppressing the effect of noisy labels using monotonically decreasing function which focus on low-loss easy samples (Kumar et al., 2010;Jiang et al., 2014). Although straightforward to apply, the above methods are limited in that they all need a pre-specified closed-form weighting function, while their respective hyperparameters are sensitive to the change of training data such that careful tuning is required.

Meta-Learning
There are three main categories of modern ML algorithms: learning a metric space to measure distance or similarity among data (Vinyals et al., 2016;Sung et al., 2018), learning an optimizer which updates all of model's parameters in a latent parameter space (Andrychowicz et al., 2016;Chen et al., 2018), and learning an initialization that is good for all tasks and able to fast adapt to unseen tasks (Finn et al., 2017;Jamal and Qi, 2019). Our approach falls into the last category, where the learning process follows MAML, more specifically its variant for DG problem in . (green) Adapter layers, bottleneck outputs of which are then fed into domain classifier heads (red). The neural-SPL module consists of instance-wise weighting head (purple) for main task classification (orange) and a layer-wise balancing head (blue) for domain adversarial training.

Model
We denote the source dataset consisted of N s samples and an unlabeled set of N t samples T = x t i N t i=1 drawn from target domain. Label space Y = {1, 2, · · · , K} of K classes is shared across domains.
Our model's feature encoder is a fixed pretrained BERT encoder with hidden dimension R d h , augmented by adapters with bottleneck representation of size R da . We refer to the main model learnable parameters as θ = (θ a , θ c , θ d ), which includes the parameters of adapters, the main classification head, and the DANN heads. Following prior work (Ngo et al., 2021), low dimensional output from each layer's adapter is used by a separate DANN head for domain adversarial training. Our neural-SPL module consists of two weighting mechanisms: an instance-wise f v (θ v ) : R → R which weighs the contribution v i of each example based on the its classification loss and a learnable age parameter λ a ; and a layer-wise f w (θ w ) : R da → 1 that takes adapter representation of each layer and outputs the relative "magnitude" w l of which the corresponding layer l should be aligned. We refer to the set of source samples whose losses are less than λ a as meta-source domain S tr while the rest is meta-target domain S ts . The latter acts, in meta-test step, as a validation set used to evaluate the model after meta-train step and provide learning signals to tune the "hyperparameters" from the neural-SPL module. The overall architecture is presented in Fig. 2.

Meta Self-Paced Learning
Self-Paced Learning Kumar et al. (2010) devised Self-Paced Learning method that extends Curriculum Learning (Bengio et al., 2009) to jointly learn the model and its curriculum, circumventing the need for an ad-hoc implementation of easiness based on some predetermined heuristics. Specifically, SPL employs an age hyperparameter λ a that represents the current learning pace of the model. The objective is then reformulated as a weighted loss where each instance's contribution is thresholded by λ a as follow: where l i is the corresponding loss of i-th training sample. Intuitively, λ a is the "age" of the model which is set to gradually grow as training proceed. Thus, only easy samples are considered at the initial learning stage while samples with larger losses will be slowly added to the model's curriculum as it progresses.
Adaptive SPL via Meta-Learning The advantage of incorporating SPL into a ML framework is two-fold. First, ML provides a way to adaptively tune the highly sensitive λ a , alleviating the need for manually devising an age scheduler. At the same time, SPL helps address the lack of clean validation data, by splitting the source domain instances of the current mini-batch into two disjoint sets based on the age value λ a . The easy samples are used for meta-train step, in which the objective consists of a domain adversarial loss and a SPL-weighted classification loss: is the loss of each sample and L d is the weighted domain adversarial objective that is explained in the following section.
f v is a small feed-forward network with sigmoid as final activation function to guarantee the resulting weights located in the interval of [0, 1], and with no bias so that the 0-valued inputs will also correspond to outputs of the same value. Typically, k gradient steps are applied to approximate the optimal solution that minimizes the current meta-train objective. Because of the sizeable transformer encoder, a high value of k will cost serious computation overhead. Thus, we decide to use k = 1, from which we observe no significant performance loss: where α is meta-train learning rate. Next, the meta-test objective is the standard crossentropy loss on samples in meta-target domain S ts with loss values higher than λ a : This acts as a hard, distinct domain that provides tuning signals for guiding model updates of both model's parameters in θ and hyperparameters v i and λ a .

Balancing domain adversarial objectives
The survey presented by (Rogers et al., 2020) provides a detailed probing and understanding of how the different layer-block of BERT encodes different types of information. Accordingly, each layer should contain a different amount of discrepancy between source and target domains.
To align these representation spaces between the two domains, we employ multiple do-main classifiers at the bottleneck of every adapter: where each L l d is an adversarial term of a different DANN , taking adapter representations z l d of layer lth and domain labels y d as inputs. These losses are weighted by a set of coefficients {w l } that corresponds to how important it is for the representations at the respective layer to be aligned. Following standard learning procedure, they would be hyperparameters that required careful tuning for each specific domain, which would be impractical (in our setting, there would be a total of 12 hyperparameters). To address the above issue, we employ a small feed-forward network f w with a final softmax layer to output the relative layer-wise weights: where Z d ∈ R L×da is a set of layer representations, each element of which is the sum of all adapter representations of the corresponding layer with respect to the current mini-batch. As θ w is updated throughout the ML process, W is dynamically tuned to maintain high performance on meta-test set while domain-adversarial training makes representations across layers domain-invariant.

Meta
Optimization Following MLDG, meta-train and meta-test losses are combined in the final objective as follow: argmin θw,θv,λa where β is meta-test balancing term. The second term in Eq. 9 is the result of passing the weights computed by neural-SPL module in Eq. 3 and 7 into Eq. 2 as pre-determined values, not learnable variables.

Self-training by incorporating Pseudo Label
Pseudo-labeling is an effective method to improve target domain performance by leveraging the predictions of previous step on unlabeled target data as additional learning signals for the main downstream task. We use the pseudo-labeled target data only for L ce from Eq. 2 in meta-train step, in which they are weighted and thresholded by neural-SPL module using the same λ a as source data: L ce S tr , T = the set of target samples with losses lower than λ a . To alleviate the confirmation bias in pseudolabeling,  provided strong regularizations and data augmentations to prevent model from propagating its own inaccuracy throughout the training process. In our case, neural-SPL module would ensure that only high confident pseudo labels are used, while meta-test step explicitly improves model's performance in low-density neighborhood of target domain. This is consistent with the expansion assumption proposed by Wei et al. (2021) on how self-training denoises pseudo-labels by bootstrapping an incorrectly pseudo-labeled example with its correctly pseudo-labeled neighbors. Thus, our framework is able to effectively leverage self-training by suppressing the noises and providing a robust training for the model. In addition, as we will discuss later section, the gradient updates of these pseudo-labeled samples are also regularized by the ML framework, forcing them to be consistent with meta-target domain.   We evaluate the proposed model on the standard multi-domain sentiment analysis (SA) task. In ad-dition, we also demonstrate the effectiveness of our framework when addressing the label-shift by applying MSP-DA to ED task with significant more classes in UDA setting.
FDU-MTL (Liu et al., 2017) A dataset included reviews from 16 domains for binary sentiment classification task. In each adaptation setting, a single domain is assigned as the target with unlabeled data while the other 15 are labeled source. Given the contextual sequence computed by models from a review, we use the first token [CLS] as the feature to predict its positive or negative sentiment.
ACE-05 (Walker et al., 2005) A densely annotated corpus collected from 5 different domains. Two of which are used as source data, while each of the rest is a target domain for an adaptation setting. Given a trigger word in the context of an event mention, the model is required to perform a multi-class classification task that assigns a predicted label into one of the pre-defined 34 event types (including 1 negative type).

Data Settings
We provide statistics of each domain in UDA setting for ACE-05 and FDU-MTL in Table 1 and    (Yuan et al., 2021) is the state-of-the-art approach that learns to explicitly mask domain-related words from text, resulting in domain-agnostic sentences.

ED baselines
For ED task, we also compare MSP-DA to other functional weighting schemes that trying to balance the learning process to address the label shift. In particular, Uniform treats each sample's loss equally, Focal Loss downweights well-classified instance exponentially (Lin et al., 2017), and Class-Balanced uses a weighting factor that is inversely proportional to the number of samples (Cui et al., 2019) Noted that these model employ both adapter-based fine-tuning and adversarial training procedure.
Implementation details All models are implemented in Pytorch. We leverage pre-trained BERTbase models and checkpoints from Huggingface repository. (Wolf et al., 2020). We inject adapter layers after every feed-forward sub-blocks have bottleneck feed-forward architecture with downsampled dimension chosen among [48,96,128]. All of the downstream heads are implemented as feed-forward networks with activation functions between layers. Each weighting net of neural-SPL module is a feed-forward network with 2 or 3 layers with hidden vectors of size [100,50]

Main Results
Sentiment Analysis SA results are presented in Table 3. While simple model using contextual embedding BERT outperforms all previous LSTMbased methods, we observe little to no improvement applying domain adversarial training naively with it. In particular, BERT+DANN actually has negative effect on about half of the domains, indicating that the standard baseline approach being unable to adjust to each specific adaptation setting. In contrast, our framework achieves the best performance for 11 review domains overall, surpassing the current state-of-the-art method BertMasker by 1.5 points on average. This demonstrates both the effectiveness and the robustness of MSP-DA to each domain.
Event Dectection UDA performances for ED task are presented in Table 4. Again, we observe that BERT+DANN only provides slight improvement for domain bc compare to BERT, while significantly degrades model's performances on the other two resulting in almost 3 points drop in average out-of-domain F1 score. Similarly, applying DANN for the adapter-based model without any weighting mechanism, as in Uniform, also has adverse effects on out-of-domain performances.
Class-Balanced's in-domain results are slightly higher than other models due to its ability to balance the training process, which addresses the extreme negative-skewed label distribution of the given source data. In contrast, its domain adaptation ability is actually the lowest because of the change in data distribution across domains. Focal Loss performs generally better in out-of-domain settings as they generate weighting coefficients adaptively based on the current losses, without involving any domain-specific statistics. Finally, MSP-DA provides consistent improvements when adapting to any new domain, even achieving on average 5 points higher in F1 score compared to the best baseline method.

Ablation Study
In the first row-block of Table 5, we conduct an ablation study to validate the effectiveness of each of our main components by investigating the performance of the following variations of our model: MSP-DA-mSPL follows the normal SPL process to produce the weighting coefficients and train-test datasets for ML; MSP-DA-DANN trains only on source domain without utilizing unlabeled target data for domain adversarial objective; and MSP-DA-PL in which no pseudo-labels are leveraged for training. In general, our full model outperforms all variants across domains, even in the in-domain setting, which confirms the superiority and flexibility provided by the jointly optimized pacing and weights from our neural-SPL module. Especially for wl domain, domain adversarial training in MSP-DA manages to improve more than 8 F1 points.

Meta-test Selection
To examine the correctness of our assumption, we augment the data selection process for meta domains in Random and Reverse variants. The former randomly selects training samples for each meta domain, whereas the latter implements the opposite hypothesis by choosing hard and easy instances for meta-train and meta-test sets, respectively. Both variants result in a considerable decline in domain adaptation results as shown in 5. Notably, the significant performance drop in the in-domain setting of Random indicates that simply constructing train-test sets without any appropriate condition can do more harm than good for the ML process. These empirical observations further confirm our initial assumption on how domain shift correlates well with the easy meta-train and hard meta-test sets.

The Values of Age Hyperparameter
Age hyperparameter λ a is usually the hardest to tune in a SPL system due to the fact that aside from the initial value, determining how λ a changes throughout the training process also has a major impact on the final performance. Several prior works (Li and Gong, 2017;Ren et al., 2017) have proposed alternative age schedulers in place of the naive strategy which adds/multiples λ a with a constant at each epoch. However, the value of λ a in these methods still follows a predefined sequence, implying the need for a meticulous tuning process. In contrast, our neural-SPL module updates λ a based on optimization signals from meta-test set, thus always able to create an appropriate dynamic curriculum regardless of different learning tasks and datasets. In Table 6, we examine how different values and schedules of age hyperparameter affect performances on bc and wl domains. The Fixed (p) settings with p ∈ [25,50,75] are variations of our model with λ a values always corresponding to the unchanged p-th percentile of the current mini-batch's sample losses; or in other words, the number of samples in meta-train set is always a constant p percent that of the current mini-batch. Additionally, we evaluate the case in which p is linearly increased as training proceeds, similar to the standard SPL process, in Linear Incrs setting. The results show that the lower p is, the worse model performs, indicating that with too few meta-train data, the model will not be able to adapt to the hard meta-test domain. Surprisingly, the gradual rising scheduler of Linear Incrs is not as effective as the other Fixed variants. This means that the easy-to-hard assumption of prior SPL systems is not suitable for our ML framework.
λ a Visualization To gain more insight into how age hyperparameter changes throughout the training process of each domain, we plot the values of λ a in source-losses percentile against the number of update steps for 10 epochs in the right subplot of Fig. 3. While λ a quickly follows the standard incremental trend initially, it starts to plateau within the 60-70 percentile range until eventually starting to decrease. Notably, behavior of λ a diverges across domains in subsequent steps. Whereas λ a continues the to decline in bc and cts domains, it experiences a complete trend reversal at the end of the training of wl domain. We hypothesis that this drastic change of λ a is because of the gradients' dot product term that the objective in Eq. 9 implies, which we will delve deeper into in the discussion section below. The shape of λ a correlates with the term's value as the model maximizes it to align the gradient directions between the meta train-test domains, going from negative initially as the training started, to 0 which causing the plateau, then gradually becoming positive as the model was able to adjust the updates of meta-train set to be consistent with that of meta-test set. However, for hard adaptation such as wl domain, too few data in metatrain set can cause a major disparity between the two meta domains again, thus the resulting trend reversal at the last few steps. We also visualize the same plot for targetpseudo-losses percentile, which leads to an interesting observation: Initially, the model followed its own pseudo labels without any constraint and the high value of λ a percentile represents model's incorrect overconfidence. However, these pseudolabel updates will cause discrepancies with meta-test domain, thus the ML framework will gradually fix the corresponding predictions, allowing only quality pseudo samples to be included in meta-train set. Eventually, the target trend converges with the source ones, suggesting that model's predictions on pseudo labels are then as consistent as on clean training labels.

Balancing Domain Adversarial Losses
Previous works have observed that the weight of DANN in the combined objective has a significant impact on the overall adaptation performance of the model. We further validate this point by investigating how different domain adversarial weighting schemes affect the results on bc and wl domains. Specifically, we evaluate 3 types of layerwise weighting: (i) Constant -all layers share the same w l value, (ii) Anneal Upw l slowly increases from lower to higher layers, and (iii) Anneal Downw l is highest for the first layer and gradually declines for subsequent layers. The results are present in Table 5, in which none of the schemes is better than the others in both domains. In contrast, the meta-learned coefficients of our framework manage to boost model's performances in every adaptation setting, especially for the hard wl domain where domain adversarial training matters the most.
We further visualize how each layer's weight changes during the learning process across domains in the left subplot of Fig. 3. In particular, we partition 12 layers of BERT-base model into 3 groups of 4 sequential layers, each of which is known to contain a different type of information that is important for a different type of task as described in the previous section. We can observe from the graphs a certain pattern: the higher level the group is, the more volatile its layers' coefficients are. However, there is no specific rule shared among all domains regarding the value of each layer's weight. This affirms the sensitivity of domain adversarial balancing term to each individual domain and further justifies the effectiveness of the jointly optimized weighting in our framework.

Discussion
Following the analysis of MLDG framework presented in , we decompose the meta-test loss, given thatθ = θ − αL ′ tr (θ), using the first order Taylor expansion: Denoting G = ∂Lts(θ) ∂θ · ∂Ltr(θ) ∂θ and plugging Eq. 10 into the final objective to update main model's parameters from Eq. 8 results in the following optimization problem: The third term in Eq. 11 is a gradient-based regularization that penalizes inconsistency between parameter updates of meta-train and meta-test domains. By enforcing loss gradients of the two domains to follow a similar direction, Eq. 11 prevents the model from over-fitting to a single domain, effectively improves model's adaptation capacity provided that meta-test set is 'close' to target domain. We further examine how the ML framework affects the values of neural-SPL module's parameters (θ w , θ v , λ a ) in our model. Plugging Eq. 10 into the gradient of λ a , we have: From Eq. 12, we see that the multiplicative factor G also controls how the value of λ a changes throughout the ML process. When there is a significant discrepancy between meta-train and meta-test domain, G would have a negative value, which would in effect push λ a higher and allow more samples into meta-train set for easier adaptation to meta-test set. Conversely, a positive G would imply that the model is good enough to align the current meta domains, thus gradually pulling λ a down to make the task harder. This behavior is clearly illustrated in Fig. 3. Similar arguments can be made for the meta-learned weighting coefficients, where G would encourage samples whose gradients are similar across domains while decreasing the contribution of those whose gradients are not. These understanding are also presented in (Shu et al., 2019) and closely related to how MAML works (Nichol et al., 2018;Raghu et al., 2019) 6 Conclusion We present a novel ML framework for UDA setting that achieves state-of-the-art performance on ED task. In particular, a neural-SPL module is employed to adaptively partition source domain into meta-train and meta-test set, while simultaneously learns the instance-wise and layer-wise weights for the loss terms of downstream task and domain adversarial task respectively. The proposed model significantly improves domain adaptation performances against various baselines on every domain without domain-specific hyperparameter tuning. In the future, we intend to apply our approach to the several direction: (1) We will extend our work to multilingual problems (Pouran Ben Veyseh et al., 2022), or other domains and tasks (Lu et al., 2021); (2) We will incorporate different novel domain adaptation regularization methods ; (3) We will adapt our framework to more general multi-source domain adaptation setting.