{'fraction_non_alphanumeric': 0.030280649926144758, 'fraction_numerical': 0.009601181683899557, 'mean_word_length': 4.606443142799434, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning

Unsupervised domain adaptation (UDA) enables cross-domain learning without target domain labels by transferring knowledge from a labeled source domain whose distribution differs from that of the target. However, UDA is not always successful and several accounts of `negative transfer' have been reported in the literature. In this work, we prove a simple lower bound on the target domain error that complements the existing upper bound. Our bound shows the insufficiency of minimizing source domain error and marginal distribution mismatch for a guaranteed reduction in the target domain error, due to the possible increase of induced labeling function mismatch. This insufficiency is further illustrated through simple distributions for which the same UDA approach succeeds, fails, and may succeed or fail with an equal chance. Motivated from this, we propose novel data poisoning attacks to fool UDA methods into learning representations that produce large target domain errors. We evaluate the effect of these attacks on popular UDA methods using benchmark datasets where they have been previously shown to be successful. Our results show that poisoning can significantly decrease the target domain accuracy, dropping it to almost 0% in some cases, with the addition of only 10% poisoned data in the source domain. The failure of these UDA methods demonstrates their limitations at guaranteeing cross-domain generalization consistent with our lower bound. Thus, evaluating UDA methods in adversarial settings such as data poisoning provides a better sense of their robustness to data distributions unfavorable for UDA.


Introduction
The problem of domain adaptation (DA) arises when the training and the test data distributions are different, violating the common assumption of supervised learning. In this paper, we focus on unsupervised DA (UDA), which is a special case of DA when no labeled information from the target domain is available. This setting is useful for applications where obtaining large-scale well-curated datasets is both time-consuming and costly. The seminal works [1,2] proved an upper bound on a classifier's target domain error in the UDA setting leading to several algorithms for learning in this setting. Many of these algorithms rely on learning a domain invariant representation by minimizing the error on the source domain and a divergence measure between the marginal feature distributions of the source and target domains. Popular divergence measures include total variation distance, Jensen-Shannon divergence [9,32,37], Wasserstein distance [29,15,7], and maximum mean discrepancy [19,17,20]. The success of these algorithms is argued in terms of minimization of the upper bound proposed in [1] along with an improvement in the target domain accuracy on benchmark UDA tasks.
Despite this success on benchmark datasets, some works [9,16,36,33] have presented evidence of the failure of these methods in different scenarios. Recent works have explained this apparent failure of UDA methods by proposing new upper bounds [36,6,14,34]  others have demonstrated the cause of failure using experiments showing that learning a domain invariant representation that minimizes the source domain error can cause an increase in the error of the ideal joint hypothesis [16]. To provably explain this apparent failure of learning in the UDA setting we propose a lower bound on the target domain error. Our lower bound provides a necessary condition for successful learning in the UDA setting, complementing the existing upper bound of [1] and is dependent on the difference between the labeling functions of source and target domain data induced by the representation map. For cases where the induced labeling functions match on the source and the target domain data (i.e., favorable case), the success of UDA is explained using the upper bounds proposed by previous works [1,36,14]. For a representation that aligns the source and the target domain data and minimizes the error on the source but induces labeling functions that don't agree on the source and the target domain data (i.e., unfavorable case), our lower bound explains the failure of UDA. Our analysis brings to light yet another case (i.e., ambiguous case) of data distributions where success and failure of UDA are equally likely. This happens due to the lack of label information from the target domain. This case opens doors for adversarial attacks against UDA methods since a small amount of misinformation about the target domain labels can lead the UDA methods into producing a representation similar to the unfavorable case, incurring a significant increase in the target domain error.
Motivated from this analysis of UDA methods under different data distributions, we evaluate the extent to which the performance of current UDA methods can suffer in presence of a small amount of adversarially crafted data. For this purpose, we propose novel data poisoning attacks, using mislabeled and clean-label points. We evaluate the effect of our poisoning attacks on popular UDA methods using benchmark datasets, where they were previously shown to be very effective. We find that our poisoning attacks cause UDA methods to either align incorrect classes from the two domains or prevent correct classes from being very close in the representation space. Both of these lead to the failure of UDA methods at reducing target domain error. With just 10% poison data in the source domain, target domain accuracy for current UDA methods is significantly reduced, dropping to almost 0% in some cases. This dramatic failure of UDA methods demonstrates their limits and suggests that the future UDA methods must be evaluated in adversarial settings along with evaluation on benchmark datasets to truly gauge their effectiveness at learning under the UDA setting.
Our main contributions are summarized as follows: • We prove a lower bound on the target domain error that provides a necessary condition for successful learning in the UDA setting. Our bound shows the failure of learning a domain invariant representation while minimizing the source domain error at guaranteeing target generalization.
• We present example data distributions where UDA succeeds, fails, and where success and failure are equally likely. This sensitivity of UDA methods to the data distribution brings to light a new vulnerability of UDA methods to adversarial attacks such as data poisoning.
• To concretely understand the extent of this vulnerability of UDA methods, we propose novel data poisoning attacks using clean-label and mislabeled data. Our results show a dramatic failure of current UDA methods at target generalization in presence of poisoned data. Thus, our poisoning attacks can provide better insights into the robustness of UDA methods than those obtained from performance evaluation on benchmark datasets.

Background and related work
Analysis of unsupervised domain adaptation: Several previous works have studied the problem of UDA and have provided conditions under which UDA is possible [2,1,21,22,8,3,34]. [1,2] proposed an upper bound on the target domain error which has inspired many UDA algorithms. Recent works [36,6,14] have improved the upper bounds on target domain error and have proposed a lower bound dependent on the labeling functions in the input space. Using this lower bound, the failure of learning in the UDA setting was explained in the case when marginal label distributions are different between the two domains. Our lower bound, on the other hand, is dependent on the labeling functions for the source and target domains, induced by the representation. This suggests that if a representation induces labeling functions that disagree on source and target domains then UDA provably fails even if the representation is domain invariant and minimizes error on the source domain. Thus our lower bound directly explains the observations of failure of UDA in many previous works [9,16,33,36,34]. A detailed comparison of our work with other works analyzing the failure of learning in the UDA setting is present in Appendix G.
Algorithms for UDA: Many algorithms [9,32,18,37,30,10] for UDA learn a domain invariant representation while minimizing error on the source domain. A popular adversarial approach to domain adaptation is DANN [9] which uses a discriminator to distinguish points from source and target domains based on their representations. Another popular method CDAN [18], uses classifier output along with representations to identify the domains of the points. IW-DAN and IW-CDAN [6] were recently proposed as extensions of the original DANN and CDAN with an importance weighting scheme to minimize the mismatch between the labeling distributions of the two domains. A different approach MCD [27], makes use of two task-specific classifiers as discriminators to align the two domains. This method adversarially trains the representation to minimize the disagreement between the two classifiers on the target domain data (classifier discrepancy) while training the classifiers to maximize this discrepancy. Another recent approach, SSL [35] uses self-supervised learning tasks (e.g. rotation angle prediction) to better align the two domains. In this work, we study the effect of poisoning on these methods as they have been shown to be effective at various UDA tasks.
Data poisoning: Data poisoning [4,25,12,5,13,31,24] is a training time attack where the attacker has access to the data which will be used by the victim for training. Most works [38,26,28,23,11] have considered data poisoning in a fully supervised setting where train and test sets are drawn from the same underlying data distribution (single domain setting). These works either target the classification of a single test point by adding a large number of poisoned data or require modifying all points in a class to affect the model's performance on that class after retraining. Our work studies the effect of poisoning on the entire target domain in the UDA setting using popular UDA algorithms for training. The success of our poisoning attacks at significantly reducing the target domain accuracy with a small amount of poisoned data shows the ease of poisoning in the UDA setting. In contrast, poisoning leads to a small decrease in the overall test accuracy in the single domain setting, especially when using state-of-the-art classifiers such as deep neural networks.
3 When does learning fail in the unsupervised domain adaption setting?
Notations and settings: Let X be the data domain and D be the distribution over X with the corresponding pdf p(x). We assume there is a deterministic labeling function f : X → [0, 1] for the given binary classification task. The f (x) can be interpreted as P r[y = 1|x]. Let g : X → Z denote the representation map that maps an input instance x to its features where Z is called feature or representation space. Let h : Z → [0, 1] be a hypothesis for binary classification on the representation space. Note that the representation map g induces a distribution over Z denoted by P rD[B] := P r D [g −1 (B)] and the corresponding density functionp(z) on Z [2]. The g also induces the labeling functionf for any B such that g −1 (B) is D-measurable. The misclassification error e(h) w.r.t. the induced labeling function is The distributionsp and the labeling functionsf for the source and the target domains will be written as p S ,p T ,f S andf T , respectively. The total variation distance is D 1 (p,p ) = Z |p(z) −p (z)|dz.

Lower bound on the target domain error
Most adversarial DA methods learn a domain invariant representation g : X → Z by minimizing error on the source domain and penalizing the mismatch between the marginal source and target distributions since the conditional distributionp T (z|y) for target domain is unavailable in the UDA setting. Some works [33,36,16] have shown this to be insufficient at guaranteeing target generalization. Recent works have proposed a new upper bound [36] or argued failure in terms of the upper bound in [1,2] However, a large upper bound does not guarantee failure. Thus, we prove a simple lower bound on the target domain error which shows the necessary condition for the success of learning in the UDA setting and also explains the failure of current UDA methods at guaranteeing target generalization. In UDA, one can observe the source data (blue blobs), the source labels (class + and -), and the target data (green blobs) but not the target labels. The optimal decision boundary (red dotted line) and linear representation g(x) = u T x from the input space in R 2 to the feature space in R (dashed line) minimizing the source and the alignment losses in Eq. 3 can be computed accurately. Depending on the data distribution, domain adaptation can be successful (Case 1), fail (Case 2), or be undetermined (Case 3). Case 3 has two global minima with drastically different target domain performance, even though representations in both cases minimize the error on the source distribution as well as align the marginal distributions of the two domains.

Theorem 1.
Let H be the hypothesis class and G be the class representation maps. Then, for all h ∈ H and g ∈ G, The proof is in Appendix A. Even though our bound depends on the total variation distance which is a strict measure of distance and is difficult to estimate, the bound still explains the limitations of UDA.
Extending the bound to different divergence metrics is left as future work. The following corollary is immediate from Theorem 1.
This is due to the Pinsker's inequality: D1(p, p ) ≤ 0.5DKL(p||p ). The interpretation of Eq. 2 is as follows. Since f T is not observable, the goal of UDA is to minimize the observable source classification error e S (h) and the domain mismatch D 1 (p S ,p T ) (or other metrics such as d H∆H (p s ,p T )): This leads to maximization of the second and the third term in the RHS of Eq. 2. With overparameterized models and large datasets, minimizing the empirical estimates of the quantities in Eq. 3 can drive e S (h) 0 and D 1 (p S ,p T ) 0. Consequently, e T (h) ≥ max{e S (f S ,f T ), e T (f S ,f T )}. Iff S and f T disagree on the source and target domains in the representation space, target domain error e T (h) will be provably large. Corollary 1.2. For all h ∈ H and g ∈ G, This is obtained by combining the upper and the lower bounds (Appendix A). Thus a UDA method can only guarantee the target error e T (h) to be close to the labeling function mismatch e(f S ,f T ).
Whether e(f S ,f T ) becomes larger or smaller after solving Eq. 3 is data/model dependent. For concreteness, we analyze the sensitivity of the performance of UDA methods to different data distributions using the following illustrative examples.

Illustrative examples showing the sensitivity of UDA methods to data distributions
Assume mixture-of-Gaussian distributions p S (x) and p T (x) for the source and the target domains in the input space, shown as blue and green blobs in Fig. 1. We consider a linear representation map g(x) = u T x from the input space X ⊂ R 2 to the feature space Z ⊂ R (dashed line) that minimizes the source classification loss plus the marginal mismatch loss in Eq. 3. The optimal solution g to the minimization problem can be found accurately using a mix of analytical and numerical optimizations (detailed in Appendix B). We demonstrate the performance of UDA on three different data distributions. In Case 1 (favorable case), the true labeling function for source and target is f S ((x 1 , x 2 )) = f T ((x 1 , x 2 )) = I[x 2 ≥ 0], that is, f is 1 in the upper halfspace and 0 in the bottom halfspace. One can verify (Appendix B) that the optimal u is the vertical direction (u = [0, 1] T ) and the best hypothesis is h(z) = I[z ≥ 0], in which case e S (h) = 0 and D 1 (p S ,p T ) = 0. That is, perfect source classification and a perfect alignment of the marginals are achieved. Furthermore, the true labeling function in Z isf S (z) =f T (z) = I[z ≥ 0] (from Eq. 1) and therefore e(f S ,f T ) = 0 as well as the target loss e T (h) = 0. In other words, the representation that minimizes Eq. 3 simultaneously minimizes e(f S ,f T ), achieving the goal of reducing e T (h). However in Case 2 (unfavorable case), the true labeling function for target is upside- Since UDA does not use the target label nor the true labeling function, the optimal g is exactly the same as Case 1 (u = [0, 1] T ), in which case we still have e S (h) = 0 and D 1 (p S ,p T ) = 0, but the labeling function mismatch becomes e(f S ,f T ) = 1 as well as e T (h) = 1 which is the worst case. In other words, the representation that minimizes Eq. 3 simultaneously maximizes e(f S ,f T ), totally failing at the goal of reducing e T (h). Case 3 exemplifies an ambiguous case. The optimal projection minimizing e S is still the vertical direction u = [0, 1] T but the optimal projection minimizing the alignment loss D 1 (p S ,p T ) can be either of the ±45 with no preference of one over the other. Therefore the optimal solution u for Eq. 3 that trades off the source error and the mismatch loss has two equal-valued global solutions [±u 1 , u 2 ] T for some u 1 , u 2 . One solution ( Fig. 1, Case 3, left) yields a small e(f S ,f T ) and the other (Fig. 1, Case 3, right) yields a large e(f S ,f T ). As can be intuitively seen, UDA is successful in the former but fails in the latter, and which equal-valued solution will be chosen is undetermined. A similar example of the failure of UDA methods due to the presence of two global optima was presented in [14]. Unlike their example, we use mixture-of-Gaussian distributions and analytically compute the representation that will be obtained from the minimization of Eq. 3. Using the obtained representations, we evaluate the quantities appearing in our lower bound and show that it is predictive of the performance of UDA methods on different data distributions. Details of the analysis and the results are in Appendix B.
Motivated from this extreme sensitivity of the performance of UDA methods on simple data distributions, we set out to explore the effects of small changes in real-world data distributions (via data poisoning) on state-of-the-art UDA methods. The results of our poisoning attacks in the next section suggest that slight changes in the data distributions could have a drastic impact on the target domain generalization performance of state-of-the-art UDA methods. Our findings highlight the importance of using adversarial settings (such as evaluating the performance of UDA methods in presence of poisoned data) to gauge the effectiveness of future UDA methods at learning in the UDA setting.

Breaking unsupervised domain adaptation methods with data poisoning
In this section, we present our novel data poisoning attacks to evaluate the ease with which current UDA methods can be fooled into producing a representation that leads to a large error on the target domain 1 . We propose three methods to generate poisoned data which will be added to the clean source domain data. The first poisoning attack uses mislabeled data as poisons. Under this attack, we evaluate two approaches (a) adding mislabeled source domain data (wrong-label correct-domain poisoning) and (b) adding mislabeled target domain data (wrong-label incorrect-domain poisoning) as poisons. The second attack adds images from the source domain watermarked with images from the target domain with incorrect labels (watermarking attack) as poisons. The last poisoning attack uses poisoned data with clean labels. We evaluate two approaches for this attack (a) using source domain data (clean-label correct-domain poisoning) and (b) using target domain data (clean-label wrong-domain poisoning) to initialize the poison data. The intuitive pictures of how these poisoning attacks hurt UDA methods are shown in Figs. 4, 5, and 6 (details in Appendix C). We evaluate the effect of poisoning on popular UDA methods namely, DANN [9], CDAN [18], MCD [27], SSL [35] (with rotation-angle prediction task), IW-DAN [6], and IW-CDAN [6]. We compare the difference in the target accuracy attainable by these methods when using clean versus poisoned data. Two benchmark datasets are used in our experiments, namely Digits and Office-31. We evaluate four tasks Without poisoning, correct classes (data from source class 2 is zoomed in) from two domains are aligned ((a) and (c)). The presence of poisoned data fools the methods into aligning incorrect classes from the two domains ((b) and (d)). The mismatch between the source and target classes is dependent on the labels of the poison data (due to which the target class 1 is aligned to the source class 2).
performance of our poisoning attacks against other popular UDA methods are present in Appendix F. For all experiments, we train UDA algorithms using neural networks whose architectures are similar to those used by previous works (see Appendix H for details).

Poisoning using mislabeled source and target domain data
In this experiment, mislabeled data is added to the clean source domain as poison. The labels provided by the attacker to the poison data are chosen to fool UDA methods into learning a representation that incurs large errors on the target domain. We use two simple and effective labeling functions for this. For the Digits dataset, we use a labeling function that systematically labels the poison data to the class next to their true class (e.g. poison points with true class one are labeled as two, points with true class two are labeled as three, and so on). For the Office-31 dataset, we use a labeling function that assigns the poison data the label of the closest (in the representation space learned using the clean source domain data) incorrect source domain class. Here the attacker is limited to adding only 10% poisoned data with respect to the size of the available target domain data (for experiments with different poison percentages see Appendix D). In wrong-label correct-domain poisoning, mislabeled source domain data are used as poisons. The results in rows marked with Poison source in Tables 1 and 2, show that UDA methods suffer only a minor decrease in target domain accuracy with this approach. This happens because the presence of a large amount of correctly labeled source domain data prevents the small amount of poisoned data from affecting the performance of UDA methods. A similar effect is observed when a small amount of mislabeled data is used for poisoning in the traditional single domain setting [23,26]. However, if the relative size of poisoned data is larger or comparable to the size of clean source domain data, poisoning can be effective. This is observed in Table 2 when Amazon is the target data. The Amazon dataset is roughly 5 times bigger than both DSLR and Webcam datasets. Due to this, the permissible amount of poisoned data (10% of the target domain) makes the size of clean and poisoned data comparable, leading to successful poisoning. Thus, this attack causes UDA methods to fail in presence of a large amount of poisoned data.
In the wrong-label wrong-domain approach, mislabeled target domain data is used for poisoning. The effect of poisoning on discriminator-based methods [9,18] is shown in Fig. 4. The domain discriminator used in these methods is maximally confused when marginal distributions of the source and target domains are aligned. However, alignment of the marginal distributions does not ensure alignment of the conditional distributions [36,18]. The objective of achieving  low source domain error pushes the source domain classifier to correctly classify the poison data. Thus placing the poison and source data with the same labels close in the representation space. Wrong-label incorrect-domain poisoning causes discriminator-based UDA approaches to align wrong classes (+ to -) from the two domains, leading to a significant decline in the target domain accuracy.
Since the poison data is mislabeled target domain data, poisoning makes UDA methods align wrong source and target domain classes. This leads to a significant decline in the target domain accuracy. This is also evident from the t-SNE embedding for DANN and CDAN methods in Fig. 2. In absence of poisoning, correct source and target classes are aligned ( Fig. 2 (a) and (c)), whereas in presence of poisoning wrong classes from the two domains are closer ( Fig. 2 (b) and (d)). For MCD [27], which uses use classifier discrepancy to detect and align source and target domains, our poisoned data prevents the method from detecting target examples. This happens because the term that minimizes the error on the poisoned source domain reduces the discrepancy of the classifiers on poison data, which are from the target domain. Thus, both the generator (common representation) and discriminator (in the form of two classifiers) become optimal and there is no signal for the generator to align the two domains. The t-SNE Table 2: Decrease in the target domain accuracy for UDA methods trained on poisoned source domain data (with poisons sampled from source/target domains) compared to accuracy attained with clean data on the Office tasks (mean±s.d. of 3 trials). embedding in Fig. 3 shows this effect. In presence of poisoned data Fig. 3 (b), we see twenty distinct clusters rather than just ten (we have ten classes in the Digits) as seen in the absence of poisoning in Fig. 3 (a). In SSL [35], the generator must work well on the main task, i.e., should correctly classify all data in the poisoned source domain data. The auxiliary task ensures that representations of the source and target domains become similar as seen on clean data ( Fig. 3 (c)). But in presence of poisoned data, similar representations of correct source and target domain classes leads to a drop in the accuracy of the main task on the poisoned data. This creates a conflict between the main and auxiliary tasks due to which correct source and target domain classes cannot be aligned (Fig. 3 (d)). The objective of making the main task accurate on poisoned data leads to target domain data being assigned the labels of the poisoned points. Thereby leading to a significant drop in the target-domain accuracy. The results of wrong-label wrong-domain poisoning present in rows marked with Poison target in Tables 1 and 2 show a significant reduction in the target domain accuracy compared to the accuracy obtained on clean data. On Digits, poisoning makes the target domain accuracy close to 0% on most tasks. On Office-31, poisoning causes at least a 20% reduction in the target domain accuracy in most cases. The reason tasks in Office-31 are less hurt by poisoning is because of the use of a pre-trained representation. Similar to previous works for Office-31, we use all labeled source and unlabeled target domain data for training and fine-tune from a representation pre-trained on the ImageNet dataset. The slowly changing representation pre-trained on a massive dataset weakens the effect of poisoning but cannot eliminate it. For the two tasks D → W and W → D, the fine-tuned representation, trained just on the clean source dataset (Source Only in Table 2) achieves high accuracy indicating the domains are already well aligned in terms of conditional distributions as well. As a result, UDA methods can easily align correct classes and suffer a drop close to 10% which is the amount of poisoned data added. However, this is not an interesting case for the evaluation of UDA methods as domains can be aligned just by training on the source domain data without the need for target domain data. In this experiment, we evaluate the effect of using poisoned data that looks like the source domain data. The poisoned data is generated by superimposing an image from the source domain with an image from the target domain. This method of generating poison data is known as watermarking [28]. To generate watermarked poison data we select an image from the target domain (t) and a base image from the source domain (s) such that it has the same class as the target domain image and lies closest to the target image (in the input space). The poisoned image (p) is obtained by a convex combination of the base and target images i.e., p = αt + (1 − α)s where α ∈ [0, 1]. α is selected such that the target image is not visible in the poison image ensuring the poisoned image looks like the image from the source domain. We use the same labeling function as discussed in the previous section to label the poisoned image and add 10% poison data to the source. The illustrative picture of the effect of poisoning in this scenario is presented in Fig. 5. Successful poisoning, in this case, works just like in the previous experiment i.e., by making the representations of the data from wrong classes in source and target domains similar for DANN/CDAN, by reducing the discrepancy between the classifiers on target data for MCD and inducing a conflict between the supervised and auxiliary task for SSL. The t-SNE embedding showing the effect of poisoning (Fig. 9) and watermarked poison data (Fig. 10) are shown in the Appendix. We evaluate the effectiveness of this method on the Digits dataset for different values of α. The results in Table 3 show a significant decrease in the target domain accuracy even with a small watermarking percentage for all methods except CDAN. This is because the success of CDAN is dependent on the correctness of the pseudolabels on the target domain data (output of the classifier), which are used in the discriminator. Correct pseudo-labels provide CDAN a positive reinforcement to align correct classes from the two domains, leading to a failure of poisoning. However, as we increase the amount of watermarking, the quality of pseudo labels deteriorates. Thus, providing a negative reinforcement to CDAN that causes the alignment of wrong classes from the two domains. In this experiment, correctly labeled data is used for poisoning. To generate clean-label poison data that can affect the performance of UDA methods we must affect the features of the poison data. This requires solving a bilevel optimization problem [11,23,24] which we present in the Appendix E. Due to the high computational complexity involved in solving the bilevel problem, we propose to use a simple alternating optimization to demonstrate the feasibility of a clean label poisoning attack against UDA. We use the setting of previous works [11,28] and consider misclassification of a single target domain test point (x target test , y target test ) rather than affecting the accuracy of the entire target domain as done in the previous two experiments. Let  . The clean-label poison data u is such that when the victim useŝ D source D poison andD target for UDA, the target domain test point (x target test , y target test ) is misclassified.

Poisoning using clean-label source and target domain data
The optimization problem for the clean-label attack is as follows.
The first problem minimizes the distance between the representations of the poison and the target domain test data (first term) while ensuring the poison data is not too far from the base data (second term). The second problem optimizes the parameters of the representation using UDA methods. Attack success is evaluated by solving the second problem in Eq. 4 from scratch and evaluating the classification of x target test . This is illustrated in Fig. 6. The left part shows the case before retraining using the poison data generated from Eq. 4 and the right part shows how poisoning induces misclassification. We use two approaches for poisoning. The first uses source domain data and the second uses target domain data as base data. We add 1% poisoned data and test the effect of poisoning on a two-class (3 vs 8) domain adaptation problem on MNIST → MNIST_M (see Appendix H). The results in Fig. 7 show that using target domain data as base data is significantly more successful under small permissible perturbation ( ). Using base data from the source domain requires larger distortion to keep the poison data close to the target point in the representation space and is hence less successful. This shows the feasibility of clean label attacks against UDA methods. We believe the attack success can be improved by solving the bilevel level problem (Eq. 9 in Appendix E) and is left as future work.

Conclusion
We studied the problem of UDA and highlighted the limitations of learning under this setting. We proposed a simple lower bound on the target domain error for UDA, dependent on the labeling function induced by the representation. The lower bound demonstrated that learning a domain invariant representation while minimizing error on the source domain cannot guarantee good generalization on the target domain. We analyzed a simple model and showed the existence of cases where UDA can naturally succeed or fail. The analysis also highlighted a case where, without access to any labeled target domain data, the success and the failure are equally likely. In such a case, the presence of even a small amount of poisoned data can make the data distribution unfavorable for UDA methods, making them fail dramatically in comparison to the case without poisoning. We proposed novel data poisoning attacks to demonstrate the failure of popular UDA methods with a small amount of poisoned data. Our results suggest that the performance of a UDA method in presence of poisoned data indicates how well the method aligns the conditional distributions across the two domains. Thus, we believe our attacks can be used for evaluating UDA methods, beyond simple benchmark datasets, to reveal their robustness to data distributions inherently unfavorable for UDA.

Acknowledgment
We thank the anonymous reviewers for their insightful comments and suggestions. Proof.
Similarly, we can also show that Combining the two results gives us the statement of the theorem. Proof. From the upper bound we have, From the lower bound (Eq. 2) we have, Combining the results from the upper and the lower bounds gives us the statement of the corollary.

B Illustrative examples of UDA failure
In this section, we provide the details of the analysis of the illustrative cases in the main paper. As described in Sec. 3.2, the input space X is in R 2 and the source and the target distributions are Gaussian mixtures  The representation map g induces the distributionsp(z) over Z as The map g also induces the labeling functionf (z) on Z defined asf (z) = E D [f (x)|g(x) = z] [2]. Computing this quantity can be complex in general but is relatively straightforward for a mixture of Gaussians and a simple half-space labeling function f (x). Following the definition, we havẽ In our example, the integral R 2 · dx can be decomposed into ∞ −∞ ∞ −∞ · dzdw where z and w are the coordinates along the rotated axes u and u ⊥ (see Fig. 8).
We therefore havẽ This integral can be evaluated as where Φ is the cumulative normal distribution and q is the intersection of the two lines v T x = 0 and u T (x − zu) = 0 projected along the u ⊥ direction. More concretely, The UDA minimization problem is where the last term was added to enforce u = 1. For differentiability, we consider the squared loss instead of the absolute loss: The expectation in e S (h) can only be computed numerically due to the complex formula forf (z).
On the other hand, the mismatch loss is which can be computed either numerically or analytically.
The three cases explained in the main paper are as follows: The other shared parameters are σ = 1 and η = 10. The λ determines the optimal tradeoff between E s and D in Eq. 8.
We solve Eq. 8 numerically using scipy.optimize.minimize(method='Nelder-Mead') function which is stable even if the cost function may be non-differentiable. Starting from random initial conditions and running until convergence, the solution u for both Case 1 and Case 2 converges to [0, 1] T .

C Details of the figures explaining the effect of poisoning on UDA methods
As illustrated in Case 3 of Fig. 1 in Sec. 3.2, UDA methods can be fooled into producing a representation that causes a large error on the target domain with a small amount of poisoned data. The simplest successful poisoning attack to fool UDA methods was shown in Sec. 4.1 (wrong-label incorrect-domain poisoning). In this attack, we added mislabeled data (wrong-label) from the target domain into the source data (incorrect-domain). The left part of Fig. 4 shows this setting. The right part of Fig. 4 shows how the representation learned from discriminator-based UDA methods aligns the incorrect classes closer than the correct ones. Due to the lack of target domain labels, the loss of the discriminator is minimized as long as the green and blue blobs align, regardless of their labels. But to minimize the classification loss on the source domain the representation must classify the poison data correctly. This forces the representation of wrong source and target domain classes to be closer than the correct ones. As a result of this, the source classification loss and domain mismatch loss are minimized but the learned representation still incurs a large target domain error. This is exactly what Case 3 (right) of Fig. 1 illustrated. The t-SNE embeddings in Fig. 2 confirm this on real datasets where representations are learned using popular discriminator-based UDA methods.
To make the poisoning attacks harder to detect we used watermarking-based attacks using poison data that has some features of the target data but still looks like the source data (Fig. 10). This setting is illustrated in the left part of Fig. 5. The right part of Fig. 5 shows how discriminatorbased UDA methods are fooled into producing a representation that fails to generalize on the target domain. Similar to the previous case of wrong-label incorrect-domain poisoning discriminator is optimal when the green and blue blobs align. However, since the poison data has incorrect labels the source classification loss prefers to align it with the wrong class (poison data labeled as + is aligned with source class with label +). Due to the presence of target features in the poison data (due to watermarking) the representation moves the target domain data closer to the poison data leading to an alignment of wrong source and target domain classes. As the percentage of watermarking increases the poisoning attack becomes more successful (Table 3) indicating that target domain data is being aligned to wrong source domain classes similar to the poison data. Fig. 9 ((a) and (b)) demonstrate this effect on popular discriminator-based UDA methods.
To make our attack even stealthier, we consider the effect of using correctly labeled poisoned data on the UDA methods. We generate such poison data by solving a computationally efficient version of the bilevel problem (Eq. 9), as shown in Eq. 4. The left part of Fig. 6 illustrates the poison data generated by solving Eq. 4. The poison data specifically targets a particular target domain test point (shown in purple). Since the poison data is close in the representation space to the target domain test point, UDA methods align it closer to the class of the poison data. This leads to misclassification of the test point. Since the poison data is crafted for a specific test point, they don't have much effect on the entire target domain data. The right part of Fig. 6 illustrates this effect and shows that clean labeled poison data can also successfully hurt the performance of UDA methods.

D Effect of poison percentage on attack success with mislabeled poison data
In this section, we evaluate the effect of using different poison percentages on attack success when mislabeled data is used for poisoning. As can be seen in Tables 1 and 2, the success of wrong-label clean-domain poisoning with 10% poisoned data is very limited. Thus, here we only focus on using a smaller poison percentage to study the attack success of wrong-label wrong-domain poisoning. The results of the experiment are summarized in Table 4. For all tasks, the presence of only 6% poison data causes a significant decrease in the target domain accuracy. When the poison percentage is decreased further to only 2% we still see a drop of at least 20% in the target accuracy for all methods except CDAN [18]. The use of a conditional discriminator provides CDAN this robustness. However, the success of CDAN is dependent on the quality of the pseudo-labels from the classifier on the target domain data. Good pseudo-labels provide CDAN a positive reinforcement to align correct source and target domain classes. Thus, leading to a failure of poisoning. However, as the percentage of poisoned data increases, the classifier begins to easily classify the target domain data into labels intended by the attacker, deteriorating the quality of the pseudo-labels. This provides a negative reinforcement to CDAN causing it to align wrong classes from the source and the target domain. As a result, the poisoning attack becomes successful. Thus, for wrong-label wrong-domain poisoning, increasing the percentage of poison data gradually drives UDA methods from the case favorable to UDA to the unfavorable one. . The clean-label poison data u is such that when the victim useŝ D source D poison andD target for UDA, the accuracy onD valtarget is minimized. The bilevel formulation for this attack is as follows: The solution to the lower-level problem θ * are the parameters of the generator and the classifier learned from using a UDA method on the poisoned source domain data and unlabeled target domain data. Solving bilevel optimization problems [11,23,24] to generate clean-label poison data has previously been shown to be effective. We used an alternating optimization to avoid the computational complexity of solving the bilevel optimization (Eq. 4). However, we believe the attack success can be boosted by solving the bilevel formulation proposed in Eq. 9 and is left for future work. We use a simple two moons dataset to illustrate the effect of changing both the poison percentage and the divergence between the two domains. The target domain data is the translated version of the source domain data along the x-axis. The amount of translation is used to measure the divergence between the source and the target domains. For this experiment, we use mislabeled target domain data as our poison points. This is referred to as wrong-label incorrect-domain poisoning in the main paper.
The results of this experiment are present in Table 5. Consistent with the results present in rows marked with Poison_target of Table 1 and 2 of the main paper and Table 4, we see that increasing the divergence between the source and target domains makes it easy to poison UDA methods with small  amounts of poisoned data. Moreover, at a fixed divergence level increases the amount of poisoned data leads to a larger drop in the target domain accuracy.
Effect of poisoning on [17]: Following the original and follow-up works on MMD [6], we also evaluated the effect of our poisoning attacks against this method using the Office-31 dataset. The results of poisoning on DAN (Table 6) with 10% mislabeled target domain data, consistent with the other results shown in Table 2 of the main paper, demonstrate the effectiveness of our poisoning attack against this method. We used the Pytorch version of the implementation for DAN available from the official code [6] to generate these results.
Effect of poisoning on [30]: Here we present the results of our poisoning attack against the DIRT-T method proposed by [30], using the Digits dataset. This method uses virtual adversarial training and conditional entropy minimization whose effectiveness is contingent on the quality of the pseudolabels of the target domain data. In the main paper, we presented results of using a similar method, CDAN [18], which relied on the idea of using pseudo-labels for the target domain data in the discriminator. Consistent with our results of CDAN presented in Table 1, we see our poisoning attacks (Table 7) are effective at reducing the target domain accuracy obtainable by using the DIRT-T method. We used 10% mislabeled target domain data as our poisons (same as that used in Table 1) and used a similar setting to the official implementation of the paper for our evaluation.  Effect of poisoning on [10]: This work uses a Cycle GAN coupled with a semantic loss dependent on the pseudo-labels for the target domain data, to enforce consistency of the two domains in the pixel space as well as in the representation space. When using the poisoned data to train the cycle GAN, we found that it failed to generate good transformations of the source domain. However, it could not be concluded whether the failure of cycle GAN was due to poisoning or due to hyperparameters. Thus, in our poisoning experiments, we omit the training of the cycle GAN and rely on the transformed version of the source domain images provided in the official repository. We treat these transformed images as our original source domain images. We present the results of poisoning only the feature level domain adaptation in Table 8 (assuming cycle GAN is trained on clean data). The high target accuracy of training using only the transformed source domain data (transformed source only) compared to the target accuracy obtained using feature-level domain adaptation on clean data implies a high similarity between the transformed source and the target domains. Due to this high similarity between the transformed source and the target domains, our poisoning attacks (using 10% mislabeled target domain data as poisoned data) have limited effect. For USPS → MNIST, poisoning with 10% of MNIST training data (6000 images) is comparable to the size of USPS training data (7291 images) and thus we see a significant reduction in the target domain accuracy. These results are consistent with the results presented in the main paper, especially on tasks W → D and D → W in Table 2, where poisoning fails since the source and target domains are very similar (indicated by high source only performance).

G Additional related work
Previous work [36], presented an information-theoretic lower bound to explain the failure of learning a domain invariant representation when marginal label distributions differ across the source and target domains. In comparison, our lower bound does not require any assumption on the data distributions. Our bound presents a necessary condition for successful learning in the UDA setting. In particular, our lower bound shows that a UDA method may succeed or fail at target generalization i.e. the term max (e S (f S ,f T ), e T (f S ,f T )) in our lower bound (Eq. 2) can be small or large, even when a domain invariant representation is learnt that minimizes source error.
Unlike the bounds presented by previous works [36,14] which fail to provide insights into the behavior of UDA when marginal label distributions are the same across the two domains, our bound remains tight. In particular, the information-theoretic lower bound of [36] (Theorem 4.3 of [36]) becomes vacuous when d JS (p Y S , p Y T ) = 0. However, our lower bound will be non-vacuous in this scenario as long as UDA methods are used i.e., e S and D 1 (p S ,p T ) are minimized. A concrete example where our bound is tighter than the bound of Theorem 4.3 of [36] is their example in section 4.1 where the true target risk is 1. As discussed in their paper (last paragraph below Corollary 4.1), their lower bound on the example is vacuous and says that the target risk will be greater than 0. In comparison, our lower bound in Theorem 1 says that the target risk is greater or equal to 1 which is tighter and much more informative.
Compared to the previous work of [34] which decomposed the target risk into source risk, representation conditional label divergence, and representation covariate shift and explained the reason for the failure of DANN to be its inability to account for representation conditional label divergence. Our lower bound suggests a similar explanation for the failure of DANN i.e, without access to any labeled target domain data, DANN may learn a representation that induces labeling functions (f S ) and (f T which do not agree with each other on the source and target domains. This difference in the induced labeling functions is captured by max(e S (f S ,f T ), e T (f S ,f T )) term in our lower bound. Our explanation for UDA failure also extends beyond DANN as we illustrated through our data poisoning attacks, which are concrete ways to lead UDA algorithms to learn a representation that incurs high error on the target domain.

H Details of the experiments
All codes are written in Python using Tensorflow/Keras and were run on Intel Xeon(R) W-2123 CPU with 64 GB of RAM and dual NVIDIA TITAN RTX. Dataset details and model architectures used are described below.

H.1 Dataset description
Here we describe the details of the datasets used for the Digits and Office-31 tasks.

H.2 Model architecture
Here we describe the model architectures used for different tasks. To fairly compare the performance of different UDA methods and eliminate the effect of architecture changes in improving the performance of different methods, we make use of similar model architectures for different methods, as described below. The effectiveness of these architectures has also been shown by previous works.
Digits: The architectures used for MNIST → MNIST_M, MNIST → USPS and USPS → MNIST involves a shared convolution neural network. The output of this shared network is fed into a softmax classifier and the discriminator. The architecture of the shared network consists of a convolution layer with a kernel size of 5x5, 20 filters, and ReLU activation, followed by a maxpooling layer of size 2x2. This is followed by another convolution layer with a 5x5 kernel, 50 filters, and ReLU activation followed by similar max pooling and a dropout. Then we have a fully connected layer with ReLU activation of size 500 followed by a dropout layer. For the discriminator, we use two dense layers with 500 units each followed by a ReLU and a dropout layer. This is followed by a 2 unit softmax layer. For MCD, we use the following architecture for the generator on MNIST → MNIST_M task. A convolution layer with a kernel size of 5x5, 32 filters, and ReLU activation, followed by a max-pooling layer of size 2x2. This is followed by another convolution layer with a 5x5 kernel, 48 filters, and ReLU activation followed by a similar max-pooling layer. We use 2 dense layers for the classifier with 100 units followed by ReLU activation and dropout layers. This is followed by the softmax layer. Unlike the original work MCD [27], we do not use batch normalization layers in these tasks to make architectures consistent across different methods. For SVHN → MNIST we use the following architecture for the generator. A convolution layer with a kernel size of 5x5, 64 filters, the stride of 2 followed by batch normalization, dropout, and ReLU activation layer. This is followed by another convolution layer with a kernel size of 5x5, 128 filters, the stride of 2 followed by batch normalization, dropout, and ReLU activation layer. Then another convolution layer with a kernel size of 5x5, 256 filters, the stride of 2 followed by batch normalization, dropout, and ReLU activation layer. This is followed by a dense layer with 512 units followed by batch normalization, ReLU activation, and a dropout layer. We use the softmax (a) Base data chosen from the source domain (b) Base data chosen from the target domain Figure 11: Poison data (top rows) obtained after solving Eq. 4 by using DANN as the UDA method, with base data (bottom rows) initialized from the source domain (left) and the target domain (right). Attack success with poison data initialized from the target is significantly higher than the attack success obtained with poison data initialized from the source, from under the same maximum permissible distortion constraint ( = 0.1 in ∞ norm) as seen in Fig. 7.
layer for classification. For the discriminator, we use two dense layers with 500 units each followed by a ReLU and a dropout layer. This is followed by a 2 unit softmax layer. For MCD, we use the same architecture for the generator except that we use max-pooling instead of convolution layers with stride 2 to downsample the representation. The classifier uses the output of the generator and feeds into a dense layer with 256 units followed by batch normalization and ReLU activation layers. This is followed by a softmax layer.

Office-31:
For office experiments, we use the publicly available code of the work 3 [6] and supply the poisoned data by adding them to the input files being used by the code. We use all default options of the code and use DAN, CDAN, IW-DAN, IW-CDAN algorithms. This is done to eliminate the effect of hyperparameters on the performance of the UDA algorithms on the Office-31 dataset and be able to fairly compare the performance of poisoning. To obtain the representation trained only on the source domain data, we initialize a ResNet50 model with weight pre-trained on Imagenet. We then update the representation by training on respective source domain data for different tasks.

H.3 Clean-label attack on MNIST → MNIST_M
For this experiment, 1% poison data is used to prevent the alignment of a target test point to its correct class. We test the attack on the binary classification problems (3 vs 8). Two approaches to initializing the poison data are evaluated. In the first approach, the poison data is initialized from the source domain data, and in the second approach, it is initialized from the target domain data. In both cases, the poison is picked from the class opposite to the true class of the target test point. Moreover, the poison data is initialized using the points closest in the input space to the target test point. The poison data obtained after solving Eq. 4 is added to the source domain data and UDA methods are retrained from scratch. The attack is considered successful if the target test point is misclassified after this retraining. For the results shown in Fig. 7, we randomly targeted 20 points and obtained poison data corresponding to each UDA method. Attack success is reported after evaluating UDA methods on five random initializations by adding the generated poison data in the source domain. To control the amount of maximum distortion between experiments, we add a constraint on the maximum permissible distortion to poison data using ∞ norm and use a value of = 0.1. The poison data obtained after solving the optimization with base data chosen from the source and target domains with DANN as the UDA method are shown in Fig. 11. To generate poison data that remains effective even after UDA methods are trained from scratch, we make use of multiple randomly initialized networks during poison generation. Following the work [11], we reinitialize the models at different points during optimization. This re-initialization scheme helps train UDA methods with different random initializations and for a different number of epochs making the poison data more resilient to initialization change that can happen at test-time.