{'fraction_non_alphanumeric': 0.040779973399201974, 'fraction_numerical': 0.008288998669960098, 'mean_word_length': 4.699106659447753, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Multi-Domain Generalization through Domain Re-labeling

Domain generalization (DG) methods aim to develop models that generalize to settings where the test distribution is different from the training data. In this paper, we focus on the challenging problem of multi-source zero shot DG, where labeled training data from multiple source domains is available but with no access to data from the target domain. Though this problem has become an important topic of research, surprisingly, the simple solution of pooling all source data together and training a single classifier is highly competitive on standard benchmarks. More importantly, even sophisticated approaches that explicitly optimize for invariance across different domains do not necessarily provide non-trivial gains over ERM. In this paper, for the first time, we study the important link between pre-specified domain labels and the generalization performance. Using a motivating case-study and a new variant of a distributional robust optimization algorithm, GroupDRO++, we first demonstrate how inferring custom domain groups can lead to consistent improvements over the original domain labels that come with the dataset. Subsequently, we introduce a general approach for multi-domain generalization, MulDEns, that uses an ERM-based deep ensembling backbone and performs implicit domain re-labeling through a meta-optimization algorithm. Using empirical studies on multiple standard benchmarks, we show that MulDEns does not require tailoring the augmentation strategy or the training process specific to a dataset, consistently outperforms ERM by significant margins, and produces stateof-the-art generalization performance, even when compared to existing methods that exploit the domain labels.


I. INTRODUCTION
Supervised machine learning models are developed with the assumption that the training and testing data are independent and identically distributed (i.i.d.). As a result, such models can fail drastically when tested on non i.i.d. test data [1]. This severe drop in performance indicates poor generalization capabilities of the learned models, and addressing this fundamental challenge has become an important topic of research [2], [3], [4]. Over the last few years, several classes of solutions, ranging from unsupervised adaptation approaches [5], new formalisms for domain-invariant learning [6], [7] to data augmentation techniques [8] and novel regularization strategies [9], have been proposed to improve generalization under covariate shifts. Furthermore, given the inherent limitation of using data from a single source domain to generalize under real-world shifts, methods that leverage multiple domains have also emerged [10]. Commonly referred to as zero-shot, multidomain generalization (ZS-MDG), this formulation assumes that labeled data from multiple source domains is available but with no access to the target domain.
The simplest solution to this problem is the vanilla empirical risk minimization (ERM) [11], which minimizes an average loss computed on data pooled together from all available source domains. The inability of this approach to exploit statistical discrepancies between domains has motivated the design of multi-domain learning techniques [12]. However, recently Gulrajani et al. [13] reported that a powerful feature extractor coupled with effective model selection can make ERMs highly competitive on standard benchmarks. Since then, there is renewed interest in better understanding and improving the performance of vanilla ERM.
In this context, approaches that enforce ERM-based models to be consistent under appropriate data augmentations have become popular [8], [14]. Despite their effectiveness, choosing the most appropriate augmentation for a given dataset is challenging, and in practice, even advanced strategies, e.g., random convolutions, can provide varying degrees of performance gains across datasets. Ensembling methods [15] form another important class of approaches for MDG [16], [17], which combine domain-specific models (that can optionally share parameters), though our empirical study shows that they do not consistently provide non-trivial gains over ERM.

A. Proposed approach
In this paper, we take a different perspective for improving ZS-MDG, and for the first time, study the important link between domain-aware MDG solutions and the domain labels that come with the dataset. For example, in the WILDS Camelyon-17 dataset [18], each domain corresponds to a different hospital site, and one source for large covariate shifts is the variation in the scanning protocol adopted at each site. However, when severity of covariate shifts across domains are low (similar scanning protocols), when compared to variations within a domain (diverse patient demographics), extensively optimizing for invariance across domains can be insufficient. We argue that, since domain labels play a significant role in the success of ZS-MDG methods, through a more meaningful re-grouping of data samples, one can learn invariances that are the most beneficial for out-of-domain generalization.
To this end, we first present a motivating case-study based on a recent distributional robust optimization algo- Fig. 1: Approach Overview. We compare our approach to vanilla ERM and domain-aware ensembling [16], [17] methods. While the ERM formulation ignores the domain labels, the latter combines domain-specific models. In contrast, MulDEns uses a deep ensemble backbone but automatically infers the "optimal" domain groups for MDG. rithm (GroupDRO [19]), wherein we develop a new variant GroupDRO++ that jointly infers optimal "domain" groups and performs multi-source model training with GroupDRO. Our analysis shows that, GroupDRO++ outperforms both the vanilla ERM and a standard GroupDRO implementation in zero-shot generalization, thus validating our hypothesis. Note that, by design, GroupDRO++ requires integration of an auxiliary clustering step (e.g., K-means), which identifies the domain groupings, to the GroupDRO algorithm, and it is nontrivial to extend this to other MDG approaches. Furthermore, the need for explicit clustering once every few training epochs can pose a computational bottleneck for large datasets.
Hence, we introduce MulDEns (Multi-domain Deep Ensembles), that uses a simple ERM-based ensembling technique as the backbone MDG algorithm, and employs a meta learning approach, along with a novel gradient-matching strategy, for implicitly reorganizing the data samples into more meaningful domain groupings. In other words, MulDEns enables effective end-to-end training for this challenging optimization problem and avoids the need to learn an explicit domain labeler. The choice of deep ensembles as the backbone MDG algorithm (even though it is sub-optimal compared to other sophisticated ensembling methods) is motivated by its simplicity while being able to produce well-calibrated models under distribution shifts [20]. Figure 1 illustrates the proposed approach in contrast to the vanilla ERM and existing domain-aware ensembling methods. Using extensive empirical studies on a large suite of multi-domain benchmarks, for the first time, we find that domain re-labeling leads to significant performance gains over the ERM baseline, and more importantly, relaxes the need to tailor data augmentation or regularization strategies for any given dataset.

B. Contributions
We make the following contributions and key findings in regard to ZS-MDG: • We introduce GroupDRO++ to demonstrate the need for inferring "optimal" domain labels in MDG; • Next, we propose MulDEns, a general approach for relabeling in MDG with a deep ensemble backbone; • Gradient-matching based model assignment that we show to be empirically superior to other design choices; • We investigate two model selection strategies based on only source domain validation data; • Using extensive empirical studies with DomainBed [13], we show that both GroupDRO++ and MulDEns significantly improve over ERM as well as the vanilla GroupDRO. • MulDEns works effectively on all datasets without the need to tweak the data augmentation strategy. • MulDEns produces SoTA results on PACS (+0.71%), OfficeHome (+1.36%) and Camelyon17-WILDS (∼ 2%) benchmarks. On the more challenging Terra-Incognita dataset, MulDEns matches the best reported performance from SagNet [21].

A. Problem Setup
Given access to K labeled source domains {D 1 , . . . , XY is the k th domain comprising N k image-label pairs, the goal is to generalize to any novel test domain D † , without requiring any labeled or unlabeled examples. In this paper, we consider the homogeneous MDG setting i.e., observed and unobserved domains share the same label space.

B. Vanilla ERM
The simplest approach to this problem of ZS-MDG is to train a deep classifier on all available training data from multiple source domains combined. The objective of this baseline is to find a function f : x → y that maps samples x to labels y by minimizing the empirical risk on the pooled data given by where is the loss function that measures predictive error, e.g., cross-entropy, and |D| denotes the size of the pooled dataset. Since this simple baseline method does not leverage the inherent discrepancies between the sources domains, one might expect this to be ineffective in practice. Surprisingly, this naïve solution can produce competitive performance to stateof-the-art methods on standard domain generalization benchmarks [13]. However, insights from the prediction calibration literature show that ERM models are not reliable under complex distribution shifts as they can produce poorly calibrated predictions. As a result, this baseline is not guaranteed to be effective in all cases.

C. Error Bounds for MDG
Theoretical results on bounding the average risk estimation error for MDG in a simple, binary classification setting were first reported in [10], and subsequently extended to the muticlass classification setting in [22]. In order to motivate our study, we first state the result for the binary classification case from [10]. In our setup where no prior knowledge about the target domain is available, it is common to measure the average risk over all possible target domains. To this end, let us begin by defining a hyper-distributionP on (x, y) from which all source and target distributions are drawn from. In other words, each of the source distributions P (k) XY ∼P , k = 1, · · · , K and any target distribution P † XY ∼P . In order to train a classifier f that generalizes to any possible target domain, we can leverage the domain information during training, i.e., y = f (P X , x). In such a case, the average risk can be estimated as follows: In practice, we approximate these expectations using a finite number of realizations, i.e., K different domains with {P (k) where The discrepancy between E(f ) and the estimateÊ(f ) can be measured using an appropriate divergence in some hypothesis space for f . For example, Blanchard et al. [10] considered the space of f to be a reproducing kernel Hilbert space (RKHS), where the inducing kernel is of the form k((P (1) X , x 2 j )). The following theorem provides an upper bound on the discrepancy |Ê(f ) − E(f )|: Theorem 1 (Average risk estimation error bound for binary classification [10], [12]). Assume that the loss function is L -Lipschitz in its first argument and is bounded by B . Assume also that the kernels k X , k X and κ are bounded by B 2 k , B 2 k ≥ 1 and B 2 κ , respectively, and the canonical feature map Φ κ : v ∈ H k X → κ(v, ·) ∈ H κ of κ is L κ -Hölder of order α ∈ (0, 1] on the closed ball B H k X (B k ) 1 . Then for any r > 0 and δ ∈ (0, 1), with probability at least 1 − δ, it holds that: where C is a constant.
For simplicity, assuming that ∀N k = N , this upper bound becomes larger when (K, N ) is replaced with (1, KN ) thus indicating that using domain-wise datasets is better than pooling them into one single dataset [12]. Under the light of this result, the success of ERM over methods that use domainspecific datasets is very surprising and warrants attention. We argue that the domain labels themselves play an important role in determining the efficacy of any MDG algorithm, and when the discrepancy between the domain groups is less severe compared to variations within a group, optimizing for domain invariance does not produce apparent gains over the naïve ERM method. Hence, this paper focuses on designing MDG algorithms that can automatically infer custom domain groups so as to improve generalization on unseen target domains.

III. THE BENEFIT OF RE-LABELING: A MOTIVATING CASE-STUDY WITH GROUPDRO
In this section, we provide a motivating case-study with GroupDRO [19], a recent approach for MDG, to demonstrate the importance of re-labeling domains in improving multidomain generalization.

A. Improving ERM by Maximizing Worst-case Performance
A major drawback of the formulation in (1) is that it treats all samples from all domains/groups equally and thus decreases the loss in an average sense. An insight from distributional robust optimization is that increasing the worst-group performance can lead to better generalization. Hence, a natural extension of ERM is to decrease a weighted mean of the group-level losses with an adaptive domain-level weighting, such that large weights are assigned to groups with poor performance. Formally, let g i = k ∈ {1, · · · , K} denote the group to which a sample (x i , y i ) belongs to, where K is the total number of groups, and q k is the weight assigned to a particular group. The risk now becomes GroupDRO [19] uses an update rule for q g across iterations, such that a group with larger error is assigned a higher weight. Note that, this approach is known to be effective for certain datasets, though the improvements over ERM are not consistent across all benchmarks.

B. GroupDRO++
As GroupDRO requires explicit domain labels g i for each sample to maximize the worst-case performance, appropriate domain labels are critical for its success. We argue that this is the case for all MDG methods that explicitly leverage domain labels in the dataset [23], [24], [19], [9], [3]. We thus make a first attempt at studying the important link between choosing the appropriate group labels and generalization performance. To this end, we ignore the domain-labels that come with the dataset (e.g., photos, sketch etc. in PACS [25] dataset) and attempt to re-categorize the data into custom domain groups during training. Our proposed solution, GroupDRO++, alternatively performs unsupervised clustering to re-group samples and minimizes the objective in Eq (6). We also introduce a novel regularization strategy to make the latent space amenable for meaningful clustering.
a) Inferring Domain Groups: While one can design a variety of approaches to group samples, we adopt a formulation similar to DeepCluster [26], which is a popular deep learning-based clustering solution that iteratively clusters latent representations via K-means and uses the cluster-labels as pseudo-labels to train the network with a classification loss.
However, there is a key difference in the notion of clusters -i.e., our clusters could contain samples from many classes instead of the semantic grouping achieved by DeepCluster. b) Algorithm: We begin by first extracting the latent representations of data from all source domains via a pre-trained feature extractor h. We represent it by Z : where N denotes total number of samples pooled from all training domains. Subsequently. Z is clustered using K-Means to form M groups (can be equal to or different than K) and each sample can now be represented as a tuple (x i , y i , g i ). The feature extractor h and the classifier c are trained with a GroupDRO style optimization i.e., with adaptive weights for each group for a pre-specified number of iterations T , following which, we re-compute the latent representations for data using the updated model and perform clustering to refine the group labels.
Since the clustering algorithm is disconnected from the model training except when utilizing the updated features, it is important to regularize the training so that suitable domain groups can be created. Note that, ERM aggregates losses at the sample-level, while GroupDRO operates at the group level, and there is a need to enable a finer control between these two strategies. To illustrate this intuitively, consider these nondesirable cases: (i) An individual sample can have a high loss while its group has been assigned a smaller weight. In this case, the update to the model f = h • c via GroupDRO will not have the desired effect as the weight is low; (ii) On the other hand, a sample having a low loss value when its group has a larger weight. Due to this wrong group association, that sample would still be updated during SGD.
To combat these issues, we introduce a novel regularization term that balances both the group-level weighting and samplelevel weighting: is the sample level mis-classification error, q gi is the weight assigned to the group that the sample belongs to and 0 < γ < 1 controls the sharpness of the regularizer. Thus the final objective is given by (f (z i ), y i ) + λR, where hyper-parameter λ is used to weight the regularization term when attempting to maximize worst-case performance. Our algorithm is listed in Table 1.

C. Empirical Findings
We evaluate GroupDRO++ using four standard visual multidomain generalization benchmarks -PACS [25], VLCS [27], Office-Home [28] and Terra Incognita [29] datasets (details can be found in Section V). Following standard practice in MDG, for each dataset, we run experiments by leaving out one of the K domains for testing while using the K − 1 domains for training. To enable a fair comparison, we implemented GroupDRO++ using the DomainBed framework 2 and used the ResNet-50 [30], pre-trained on ImageNet [31], as the feature extractor for all experiments. Across experiments we use the following hyper-parameters: (i) batch size of 32 per domain; (ii) learning rate α as 5e − 5; (iii) Adam optimizer [32], (iv) number of training iterations n iter set to 5000 and T as 300, (v) number of groups K set to 4, and (vi) λ, γ, η q are set to 0.1, 0.3 and 0.2 respectively. All experiments were repeated for three trials, and the means/standard deviations are reported. For the GroupDRO++ implementation, we set the number of groups M = 4 (except for Terra Incognita, where M was set to 5).
From the results in Tables I and II, we first notice that GroupDRO, a domain-aware method, does not provide any improvements over vanilla ERM on both datasets. In contrast, GroupDRO++ improves upon both these baselines, through domain re-labeling. For example, on the PACS dataset, Group-DRO++ achieves gains of around 2.2% over GroupDRO and 1.1% over ERM. Similarly, in the case of VLCS, we observe a boost of 3.1% compared to GroupDRO and around 2% compared to ERM. These experiments provide empirical evidence to our hypothesis that re-grouping samples into optimal domain groups can lead to more generalizable solutions.
While this case-study reveals that the domain labels themselves play a crucial role, using an auxiliary label generator (K-Means clustering in our study) is limiting in practice. The need to recompute the clusters for the entire dataset (once every few epochs) is a computational bottleneck for large datasets. Furthermore, it is challenging to perform end-to-end training with such an auxiliary module and requires non-trivial modifications to the MDG backbone to perform this joint optimization. To address these crucial limitations, in the next section, we propose MulDEns, which uses a simple ERMbased deep ensemble as the MDG backbone and performs implicit re-labeling through a meta-optimization strategy.

IV. MULDENS: A GENERAL APPROACH FOR DOMAIN RE-LABELING IN MDG A. MDG Backbone Design
A popular solution in the deep learning literature to improve the calibration of ERM-based models is to construct ensembles [15], and in practice, deep ensembles can even boost the prediction performance (mean prediction from the ensemble) in addition to producing low empirical calibration error.
Existing works that construct deep ensembles for multidomain generalization typically combine models independently trained on each of the source domains [33], [34], [17], [16], [35]. For example, [17] uses a common model across the members of the ensemble, while allowing domain-specific normalization layers. On the other hand, [16] proposed to build domain-specific masks over neurons and optimized the masks for minimizing cross-domain feature overlap. Furthermore, other existing approaches have also focused on building an additional model to determine the weights to be used while aggregating the predictions from different models in an ensemble at test time [36]. While this approach could be seen as having a manager from the Mixture of Experts literature, by design, we do not employ such an approach when generalizing to an unseen domain in the zero-shot setting. This is due to the fact that in ZS-MDG problems, we can neither assume access to a collection of samples from the unseen domain, nor can we expect the manager model to automatically withstand the complex distribution shifts. In contrast to existing ensembling solutions for MDG, MulDEns uses a simple, ERM-based ensembling and does not require any parameter sharing between the constituent models.

B. Meta-Optimization for MulDEns
Formally, we represent an ensemble using a set of models {f θm } M m=1 , where M is the ensemble size and all the models are initialized randomly with random seeds. Upon training, the predictions for samples in a novel target domain D † can be obtained as an unweighted averaging of predictions, i.e., 1 M M m=1 f θm (x i ), ∀x i ∈ D † . Labeled data from each observed domain D k is divided into three disjoint sets -train set D t k , meta-validation set D v k and held-out validation set D hv k . Note, in contrast to existing ensembling methods in ZS-MDG [34], M can be different from the number of domains in the dataset K.
MulDEns has two main stages, both operating at the minibatch level -(i) a meta-train stage, where we obtain ERMstyle gradients for each constituent member of the ensemble using the collection k D t k from all K source domains; (ii) a meta-test stage, where a model relevance score (MRS) is used to determine the most appropriate model f θm from the ensemble, to apply for each of the meta-validation sets D v k and subsequently f θm is updated only using meta-gradients from the subset of D v k 's assigned to this model. This step enables the implicit re-organization of the validation mini-batches into different domain groups. In addition, we also explore the use of synthetic augmentations to create additional meta-validation batches for exposing the models to more diverse variations of the observed data. Finally, the held-out validation sets {D hv k } are used for model selection (following standard practice). Algorithm 2 lists the steps involved in our algorithm.
1) Meta-train Stage: : In every iteration, K mini-batches {B t k } are randomly sampled from the K training sets {D t k }, which are then pooled to form the data batch B := k B t k ⊂ D t k and passed as input to all M models. The empirical risk and the corresponding gradients for each of the models ∇ θm (L θm ) are computed independently w.r.t. B. Akin to any MAML [37] style algorithm, MulDEns takes one gradient step for each of the models f θm to obtain f θ m , i.e., with a pre-specified learning rate α.
2) Meta-test Stage: While the train stage is similar to conventional ERM, our goal is to build ensembles that implicitly identify optimal domain groups for improved generalization. To achieve this goal, we systematically regulate the gradient flow from the meta-validation data batches to each of the constituent models based on a model relevance score.
We denote a generic, model relevance scoring function by S : V k × f θm → R + [0, 1] which scores the model f θm for a mini-batch V k ⊂ D v k from the meta-validation dataset. We denote by β km the resulting score i.e., We now describe the different design choices of S and explain in detail our proposed gradient-matching based MRS in IV-D.
, · · · B K ] // pool the minibatches from {D t k } // Compute empirical risk L θm w.r.t. B, using eq. (7); //inner gradient update// Update θ m = θ m − α∇ θm L θm (θ m ) end //model relevance score// for k in 1 · · · K do for f θm in f θ1 · · · f θ M do sample a mini-batch V k from D v k ; compute β km using eq. (12); end end Identify γ m , the set of indices of meta-validation batches, assigned to f θm ; Compute meta-test loss G θ m using eq. (10); Perform meta-update using eq. (11) end end Intuitively, when the MRS is high, one expects that taking a gradient step for θ m based on B is highly likely to improve the performance on V k . We compute this relevance score between every pair of meta-validation mini-batches {V 1 , . . . , V K } and the models {f θ1 , . . . , f θ M } to obtain the matrix β ∈ R K×M . By identifying the model with the largest β km value, one can assign the most relevant model for each validation mini-batch V k . The final step is to compute the meta-gradients for θ m , ∀m w.r.t. to their "relevant" domains and perform a gradient-through-gradient update. We denote the indices of meta-validation batches that are assigned to f θm by γ m = {j ∈ (1, · · · , K)}, such that for each j, model f θm provides the largest MRS. The meta-validation loss G θ m of f θ m using the relevant validation batches, γ m , can be written as where the definition of θ m comes from eq. 8. The final metaupdate of f θm using a gradient-through-gradient optimization can be written as follows: As a consequence of computing this assignment for metaupdate in every iteration: (i) different parts of data from one source domain or mini-batches from different source domains could get assigned to different models, thus producing an ensemble that is guided by the inferred domain groups; (ii) in cases where none of the validation batches are assigned to a model, this meta-update reduces to simple ERM update from eq. (8), which is still a strong baseline model.

C. Augmenting Meta-Validation Data
Our empirical study shows that by including synthetically augmented versions of the meta-validation batches, one can better leverage the intra-diversity in the source domains and further enhance the performance of MulDEns. Formally, we use standard image augmentation strategies (details in Section V) on the meta-validation batches {V 1 , . . . , V K } to produce {V K+1 , . . . , VK}, whereK − K is the number of additional batches. Note, through ablation studies, we demonstrate what role this augmentation for meta-validation data plays alongside the standard practice of training data augmentation adopted in the meta-train stage.

D. MRS Design
In this section, we discuss several design choices for implementing MRS. As outlined above through the MRS scoring function S, we determine assignment of a model for a given meta-validation set to perform the meta-test update. It is to be noted that S operates at the level of mini-batches thus enabling us to leverage discrepancies within and across domains. We have considered the following choices for S: 1) Random Assignment: In this case, S assigns randomly assigns each mini-batch to one of the models; 2) All-to-All assignment: Here, every validation mini-batch V k is assigned to all members of members; 3) Loss-based assignment: A more sophisticated choice is to use the empirical loss to determine the member assignment i.e., β km = 1 − 1 Gradient-matching based assignment: While empirical loss-based assignment is a reasonable choice, we propose to design MRS through gradient-matching. Though gradient-matching has been used in different contexts -for example, model alignment in MAML [37], task affinity in multi-task learning [38], diversity in active sample selection [39] etc., we are the first to find that gradient-based domain grouping is more superior to lossbased grouping, though the latter metric is routinely used for model selection in MDG.
where L θm and G k θm are the empirical risks (eq. 1) computed using the model f θm on the meta-train (B) and meta-validation (V k ) batches respectively.
The summation is over all parameters in θ m , and this score computes the dot product between parameter sensitivities of θ m w.r.t. the train and validation batches.

E. Model Selection Strategies
A crucial component of any ZS-MDG algorithm is the specification of a model selection criterion. Here, model selection mainly refers to the selection of appropriate training checkpoints to evaluate on the unobserved domain D † . It was found in [13] that different model selection criteria lead to drastically different performance for the same method, thus making benchmarking of ZS-MDG approaches challenging.
In the context of MDG with ensemble-based approaches, the choice of model selection strategy has not been studied before. Note that, with MulDEns, we perform inference for a test sample by averaging the predictions from all M models in the ensemble, x ∈ D † ,ŷ = 1 M M m=1 f θm (x). Defining the accuracy function as A : x × y → R + [0, 1], we can compute the performance of an individual model for a domain k as A(D hv k ; f θm ) and that of an ensemble as A(D hv k ; {f θm }). We investigate two model selection strategies in this study: Overall Avg: In this strategy, we choose the checkpoint in which each individual model f θm produces high accuracy on each of the K domains, on average. In other words, arg max where c indicates the training checkpoint index.
Overall Ens: In this case, we choose the checkpoint in which the ensemble produces the highest accuracy for each of the K domains, on average.
arg max

A. Dataset Description
We evaluate MulDEns using six standard visual MDG benchmarks (i) PACS [25] dataset comprising 4 domains, namely photos, art, cartoon and sketches, with images belonging to 7 different classes; (ii) VLCS [27] dataset, which is also comprised of 4 domains corresponding to the four benchmark image datasets (Caltech101, LabelMe, SUN09 and VOC2007) and contains images from 5 classes; (iii) OfficeHome [28] dataset containing images from 65 classes, where the images represent 4 different domains, namely art, clipart, product and real respectively; (iv) Terra Incognita [29] comprised of camera trap images of wild animals obtained from four different camera angles (i.e., domains) and 10 different wildlife categories; and (v) Camelyon17-WILDS [43], [18] consisting roughly 400k images of potentially cancer cells taken at different hospitals and scanners.

B. Experimental Setup
Following standard practice in ZS-MDG, for every dataset except for Camelyon17-WILDS, we run experiments by leaving out one of K domains for testing while using the K −1 domains for training. For Camelyon17-WILDS [43], [18], we use the standard protocol of using data from first three hospitals as training domains and use data from fourth and fifth hospitals as validation and testing domains. To enable a fair comparison with the state-of-the-art, we use ResNet-50 [30], pre-trained on ImageNet [31] as the backbone feature extractor for all experiments. For MulDEns, we use a random 80-20 split from each of the source domains to obtain the train and validation sets, while the train set itself is further subdivided (80-20) to construct meta-train and meta-validation data. We report the mean and standard deviation of performance, obtained across three trials with different random seeds, for each experiment similar to [13]. Across all experiments we use the following hyper-parameters: (i) batch size of 32 per domain; (ii) learning rate of 5e − 5 (for both α and λ); (iii) Adam optimizer [32], (iv) number of training iterations set to 5000 and (v) ensemble size M is set to 3. We implement MulDEns into the publicly available DomainBed framework. In MulDEns, the training mini-batches are augmented using a composition of the following augmentation choices: random horizontal flip, random  (left) When compared to approaches that utilize advanced data augmentation strategies to improve generalization, MulDEns eliminates the need for tailoring the augmentation strategy and is consistently effective for all benchmarks. In contrast, even sophisticated approaches such as RandConv (RC) or its combination with RandAug (RA) provide varying degrees of improvements over ERM (RA) across different datasets; (b) When compared against different design choices for the MRS function S, we find that the proposed gradient-matching performs the best.
color jitter and grayscaling with 10% probability, which we refer to as RandAug (RA). As described in Section IV-B2, we also create additional meta-validation batches by augmenting each batch V k using subsets of augmentations used during training. We set η in eq. (11) to 1.0 and study the sensitivity of this hyper-parameter as part of our ablation study. We report results for both the proposed model selection strategies and our rigorous empirical study shows that the Avg. strategy provides a small margin of improvement over Ens.

C. Key Findings
Re-labeling leads to significant improvements over ERM. As can be seen from Figure 4 (left), Tables I, II and III, using custom domain groups is highly beneficial. Overall, across the benchmarks, both GroupDRO++ and MulDEns improve over ERM by significant margins, in terms of average generalization performance, regardless of the level of cross-domain gap inherent to each of the datasets. Furthermore, similar to our findings in Section III, GroupDRO++ is consistently better than the vanilla GroupDRO implementation. Through the use of a deep ensemble backbone, MulDEns produces the best performance among the two proposed methods. For example, datasets such as TerraIncognita, and Camelyon17 are known to contain much higher cross-domain discrepancies compared to VLCS, thus making zero-shot generalization more challenging. However, we find that MulDEns with M = 3 achieves large performance gains over ERM and importantly, the gap with ERM widens as the severity of domain shift increases. In the challenging Camelyon17-WILDS dataset, MulDEns provides > 10% gains over ERM. In order to gain insights into the behavior of MulDEns, in Figure 6, we visualize examples images assigned to the three models (M = 3) in the ensemble. For this illustration, we trained MulDEns using the art (A), cartoon (C) and photo (P) domains from the PACS benchmark. This clearly shows that images from the same input domain can be assigned to different groups, thus enabling improved generalization. Interestingly, we observe that each model evolves to specialize for different styles (in terms of image statistics) and semantic concepts (e.g., images corresponding to the person class are strongly representative of group 3).
MulDEns outperforms existing ensembling methods with the same complexity: We perform a comparative analysis of MulDEns to a state-of-the-art multi-domain ensembling method, DMG [16], which infers individual masks over the neurons for each of the source domains. Note that both DMG and MulDEns are implemented using the DomainBed framework, and hence use the same experiment protocol, I: Summary performance of popular ZS-MDG baselines obtained using GroupDRO++ and MulDEns. While the proposed GroupDRO++ improves over ERM and vanilla GroupDRO methods, MulDEns with Overall (Avg.) model selection consistently achieves the best generalization performance. † denotes results from our implementation of baselines. We highlight the best performing method with bold face and next best with bold italics.
i.e., architecture, augmentations etc. In Figure 4(right), the superiority of MulDEns over DMG is clearly evident across all benchmarks, with improvements as high as 4.5% on the Office-Home benchmark. Through our ERM style update in the meta-train stage and by taking into account the relevance of a member w.r.t. to a domain during the meta-test stage, we effectively re-label the samples into optimal domain groups.
Detailed results across datasets are reported in Table I. MulDEns provides non-trivial improvements over SoTA DG methods: Despite the effectiveness of ERM as a baseline, its inability to leverage domain discrepancies implies that there is a non-trivial performance gap between ERM and stateof-the-art MDG methods. Using rigorous comparisons with benchmarks created by [13], and compiled in Table I, we find that MulDEns is highly competitive with the SoTA methods, which rely on a variety of strategies to leverage cross-domain discrepancies. Interestingly, our approach produces state-ofthe-art results on PACS (+0.71%), Office-Home (+1.36%) and matches the performance of the state-of-the-art SagNet on the TerraIncognita dataset. Similarly, as showed in III, on Camelyon17-WILDS MulDEns achieves an overall accuracy of 94.6%, outperforming best-performing approaches such as SagNet and CORAL by a margin of ∼ 2%. Choice of MRS function S in MulDEns: As explained in section IV-D, we experimented with four different choices of the scoring function S. In Figure 5(right), we show the effect of these choices on the performance using the VLCS dataset. We observe that the proposed gradient-matching based MRS performs the best, considerably outperforming lossbased model assignment.

D. Ablation Study for MulDEns
In this section, we present crucial observations from our ablation study designed to test the impact of different design choices.  1) Impact of meta-validation and train augmentation: In Table IV, we show how the performance of MulDEns varies when the augmentation protocol is changed. As discussed earlier, we divide data from each observed domain into three disjoint sets -train, meta-validation and held-out validation. We explore whether adding synthetic augmentations to create additional meta-validation batches can lead to improved implicit grouping. Note that, in this setup, we applied standard augmentation (following DomainBed) to the meta-train batches. It is evident from the table that, removing meta-validation augmentation leads upto 1.5% decrease in accuracy in the cases of VLCS and Office Home datasets.
We performed another ablation by not including any augmentation to the meta-train batches, while still considering synthetically augmented meta-validation batches. We observed that, for benchmarks with larger cross-domain gap such as, TerraIncognita [29], removing augmentation during training has significant effect on performance with upto 2.8% drop and unsurprisingly the performance in this case of no train augmentation is lower than ERM with train augmentation. In contrast, with benchmarks such as VLCS, not applying train augmentation does not show any apparent impact.
2) Choice of η: Next, we studied the sensitivity of the penalty η, a hyper-parameter that controls the penalty for the meta-validation loss in eq. (11) in terms of the performance of MulDEns using the OfficeHome [28] dataset. As evidenced in Table V, for values greater than 0.2 the performance of MulDEns is stable with respect to changes in η.
3) Impact of Ensemble Size.: The ensemble size M is a standard hyper-parameter in any ensembling approach. The complexity (size) of the ensemble is controlled such that a simple averaging strategy can still work for unseen test domains. When M becomes large and models become diverse, one might require a "manager" module to select a specific IV: Impact of augmenting meta-train and meta-validation sets. Without meta-validation augmentation and only train augmentation, we observed reduced performance across all benchmarks. On the other hand, omitting train augmentation and including only meta-validation leads to bigger performance drops on more challenging datasets such as TerraIncognita.   model from the ensemble. However, the ensemble size is not necessarily connected to the number of domains and for simplicity, we fixed M = 3 for all cases (identified using parameter search on PACS, VLCS). Table VI shows the performance of MulDEns on VLCS [27] at different ensemble sizes (M = 2, 3, 4). We observe that while a three-model ensemble outperforms two-model ensemble by almost 2%, there is no significant improvement achieved beyond M = 3.
We note that at this time the choice of M is empirical, and extending this framework to automatically identify the required number of models, such that simple averaging of predictions from the ensemble can still be effective, is part of our future work.

VI. DISCUSSION
In this work, we explored the benefits of re-labeling domains in zero-shot multi-domain generalization. First, we showed that, one can design solutions that are specifically tailored for a given MDG algorithm, for example, GroupDRO++, in order to jointly infer an explicit domain labeler and a classifier that achieves invariance to the domain shifts. However, through MulDEns, we introduced a generic approach that uses a simple deep ensemble backbone and an implicit re-labeling strategy to improve multi-source generalization performance. Interestingly, we find that, neither the augmentation technique nor the training strategy (e.g., additional regularization) needs to be modified for use with any dataset. Using rigorous empirical studies on standard benchmarks, we showed that both Group-DRO++ and MulDEns produce consistent performance gains over the vanilla ERM training, and on challenging datasets, even outperforms state-of-the-art approaches that exploit the domain labels. Future extensions to this work include automatically identifying the desired model complexity (ensemble size) for MulDEns, deriving achievable bounds for average risk through re-labeling and finally, investigating the role of different diversity-promoting augmentation strategies in this framework.