{'fraction_non_alphanumeric': 0.0336567222410071, 'fraction_numerical': 0.009857726220511353, 'mean_word_length': 4.651624283404381, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exploiting Inter-pixel Correlations in Unsupervised Domain Adaptation for Semantic Segmentation

"Self-training"has become a dominant method for semantic segmentation via unsupervised domain adaptation (UDA). It creates a set of pseudo labels for the target domain to give explicit supervision. However, the pseudo labels are noisy, sparse and do not provide any information about inter-pixel correlations. We regard inter-pixel correlation quite important because semantic segmentation is a task of predicting highly structured pixel-level outputs. Therefore, in this paper, we propose a method of transferring the inter-pixel correlations from the source domain to the target domain via a self-attention module. The module takes the prediction of the segmentation network as an input and creates a self-attended prediction that correlates similar pixels. The module is trained only on the source domain to learn the domain-invariant inter-pixel correlations, then later, it is used to train the segmentation network on the target domain. The network learns not only from the pseudo labels but also by following the output of the self-attention module which provides additional knowledge about the inter-pixel correlations. Through extensive experiments, we show that our method significantly improves the performance on two standard UDA benchmarks and also can be combined with recent state-of-the-art method to achieve better performance.


Introduction
Unsupervised domain adaptation (UDA) has become one of the primary methods to tackle label-scarce real-world problems such as simulated training for robots and autonomous driving. It transfers the knowledge of a deep neural network learned from the source domain with rich labels to the target domain without any label information. Usually, the source domain is photo-realistic synthetic dataset [22,23] and the target domain is real world images [9]. UDA is especially useful for semantic segmentation since it requires dense pixel-level labeling process which is laborious and time-consuming. Recently, the "self-training" scheme has become a dominant method to enable UDA for semantic segmentation. This paper also tackles the UDA for semantic segmentation based on the "self-training" scheme, yet we introduce a novel method for transferring inter-pixel correlations in the source domain to the target domain, which is also simple and effective.
The idea of self-training [18,44,45] is to give supervision for the unlabeled target domain as well. It generates a set of pseudo labels for the target domain using a trained model and then retrains a new model with the generated pseudo labels. To enhance the performance, it iteratively repeats this process with the retrained model. However, since pseudo labels are based on confident pixels whose predicted confidence surpasses a certain threshold, they are not always correct and sometimes very sparse which means that some pixels have no pseudo labels. To resolve this issue, recent works [24,38,42] have proposed to generate better pseudo labels. They denoise, rectify, and densify the pseudo labels to give more correct supervision. However, these methods are somewhat heuristic and require complex algorithms. Also, pseudo labels provide information only about which class a pixel should be predicted as but not about how each pixel should be correlated or similar to other pixels. We argue that providing guidance about the correlation between pixels is important since the outputs of a semantic segmentation task are highly structured and correlated.
A semantic segmentation task requires a model to predict highly structured pixel-level outputs which contains densely related spatial and local information. For example, the pixels corresponding to the sky are usually located at an upper area of an image, car pixels are over the road pixels and sidewalks and fence pixels are usually on the sides of the road. These inter-pixel relations are domain-invariant properties which are consistent for different domains. Therefore, in this paper, we propose an idea of transferring this domain-invariant knowledge of the inter-pixel correlation from the source domain to the target domain. Our intuition is that providing the extra knowledge of domain-invariant pixel correlations can help to overcome and complement the defects of sparse and incorrect pseudo labels.
To effectively capture these inter-pixel correlations and transfer them to the target domain, we propose a novel selfattention [7,21,27] module. The proposed self-attention module takes predictions of the segmentation network and outputs new self-attended predictions. The module creates an attention map which captures similarities between pixels and uses this map to calculate a weighted sum of the given predictions. Our proposed algorithm consists of two steps. The first step is to train the self-attention module only on the source domain since it has the ground truth labels that can correctly guide about the domain-invariant inter-pixel correlations. The second step is to train the actual segmentation network using the trained self-attention module by transferring its learned knowledge to the target domain. At this phase, the attention module is frozen and the main segmentation network is the only part being trained. The module takes the predictions of the segmentation network as inputs and outputs new self-attended predictions. Then, we train the segmentation network to follow the the new selfattended predictions. We call this the 'self-attention loss'. Overall, the segmentation network is trained with both the self-attention loss and the self-training loss. Detailed training methodology will be explained in Sec. 4.
We conduct extensive experiments on the two standard UDA benchmarks, GTA5→Cityscapes and SYN-THIA→Cityscapes. We show that the network trained with our method outperforms the one trained only with selftraining. Especially, our method shows significant improvements on rare classes which scarcely appear in the datasets. Also, our method does not require a complex humandesigned algorithm, it rather lets the module to decide which knowledge to be learned and transferred, thus our method is simple and needs less human intervention. Moreover, our method can be applied to other state-of-the-art self-training UDA methods and additionally boost the performance. We combined our method with ProDA [38], one of the recent SOTA methods, and could observe a performance improvement which sets new SOTA scores on both UDA benchmarks tested.

Unsupervised Domain Adaptation
Unsupervised domain adaptation has been widely studied to address the domain gap between labeled source data and unlabeled target data. Since domain adaptation is an important problem in real world environments, it has been studied in various tasks such as classification [11] and segmentation [38,44,45]. There are three major lines of domain adaptation approaches: adversarial learning, image translation, and self-training. Based on the theoretical analysis [11], the works [26,29] adapt the segmentation model to the target domain via adversarial learning which tries to increase the domain discrimination loss of the discriminator while reducing the segmentation loss on the source domain. The works [14,37] translate the labeled source domain images into the target domain styles using CycleGAN [43] and Fourier transforms to encourage image-level adaptation.

Self-training
Another line of unsupervised domain adaptation is selftraining which is first introduced in semantic segmentation by [44]. In [44], the authors generate pseudo labels using a model trained on the source domain and retrain the model with the pseudo labeled target domain images to implicitly encourage class-wise feature alignment and enable adaptation to the target domain. However, since pseudo labels are noisy due to domain discrepancy, many methods have been proposed to address this problem. [45] introduces a regularization method to generate soft pseudo labels and smooth the output of the model to avoid overfitting on overconfident labels. [38] and [42] lower the weights for noisy pseudo labels based on the feature distance from the class prototypes and the variance between the outputs of the two segmentation models respectively. In addition, [12] has the same effect of modifying pseudo labels based on prior distribution by introducing a transition probability map between the ground-truth label and the pseudo label. To address the sparsity of confident pseudo labels which is another limitation of self-training, [24] proposes densification of pseudo labels using confident pseudo labels of neighboring pixels. In this paper, we resolve these issues by transferring inter-pixel correlation which is a domain-invariant property.

Self-attention for vision
Self-attention calculates the response at a position in a sequence (e.g., an image) by interacting with all positions and paying more attention to more salient positions. There have been several attempts to introduce attention modules into convolutional networks. Non-local network [32] captures long-range dependencies by computing interactions between any two positions regardless of their positional distance. Another study [34] refines features by using channel and spatial attention for various vision tasks. Recently, transformer [28] based models, which consist of attention mechanisms rather than convolutional architectures, are also being actively studied in vision tasks [2,10].

Preliminary
In the common UDA setting, it is assumed that the dataset in the source domain has labels, {x s , y s }, while that in the target domain is unlabelled, {x t }. The main purpose of UDA for semantic segmentation is to train a segmentation network, G, to perform well on the target domain by transferring the knowledge it has learned from the source domain. The two domains are assumed to share the same Figure 1. The architecture of the proposed self-attention module. Grey shaded area at the center is the self-attention module and refers to a matrix product. The module takes the output of the segmentation network and creates an attention map to apply self-attention to z. z is the new self-attended output of the module which possesses information about inter-pixel correlations.
C classes, hence it is possible to adapt between them. Typically, for the source domain, the categorical cross-entropy loss is used to train the network utilizing the given ground truth labels: Here, p s = G(x s ) and p is 1 if the ground truth class at pixel i is c, otherwise it is 0. However, for the target domain, the labels do not exist, thus pseudo labels {ŷ t } are generated to perform self-training. The pseudo labels are based on the confident pixels predicted by a trained network. Following the process of [18], the pseudo labels are assigned only to pixels with confidence higher than a given threshold: The threshold τ c is set differently for each class c and how it is determined is detailed in the supplementary. Using the generated pseudo labels, the target domain can also be trained in a supervised fashion as well using (3). The overall segmentation loss is the sum of (1) and (3). In this way, the performance of the network on the target domain can be greatly improved, however, in this work, we further enhance this by introducing our novel self-attention module.

Self-attention Module (SAM)
A segmentation network uses an upsampling to enlarge its output to the size of its input. DeepLabV2 [3] uses bilinear interpolation while FCN-8s [19] uses transposed convolution. Therefore, it can be formulated as p = S(U (z)) = G(x) ∈ R H×W ×C where S and U stands for the softmax and the upsampling respectively. The z ∈ R h×w×C is the output of the network right before the upsampling (h, w < H, W ). It is the logit which becomes the class probability distribution after softmax is applied. As shown in Fig. 1, z is inputted to our self-attention module to create a new selfattended output z . We assume that z is flattened in the spatial dimension for simplicity, so its shape is R hw×C . Then, z consists of hw number of logits, z = (z 1 , z 2 , z 3 , . . . z hw ) T where each logit, z j , is a C-dimensional vector. The architecture of the self-attention module is depicted in Fig. 1. First, z is transformed into another space using a 1 × 1 convolutional layer producing Conv(z) ∈ R hw×C . Then we calculate the attention map by outer-producting Conv(z) to itself. More specifically, if we denote Conv(z) as A to keep the notation uncluttered, the attention map is computed as where · is a matrix product, A 2 ∈ R hw×1 refers to the row-wise L2-norm of A along with its channel dimension and the division is done element-wise. Each row of M shows how each pixel is similar to other pixels in cosine similarity, hence its element is normalized between −1 and 1. We apply ReLU [1] activation on M to eliminate any negative similarity and then L1 normalize each row in order to make the sum of all elements in the same row equals 1. We denote M as the attention map after the ReLU activation and the normalization. M [i,j] refers to an element of ith row and jth column of M and is computed as follows: The last step is to matrix product M and the input z to produce the final output z : Now z is the new self-attended output attended by M . Each logit (row) of z is a weighted sum of the logits of z whose weights are the corresponding row of M : Here, z i and z j refer to a logit (row) of z and z respectively. Therefore, z is a set of self-attended logits that captures the information about the inter-pixel correlations based on the similarity between pixels informed by the attention map M . Note that we use z instead of U (z) as our input to the module because using U (z) requires expensive computation and huge memory when computing M . Fig. 2 (a) shows the training of the self-attention module. As mentioned earlier, we train the self-attention module only on the source domain because we want the module to learn the domain-invariant inter-pixel correlations and the source domain has the ground truth labels that can guide this correctly. We add the output of the segmentation network z s to the output of the module z s to create z s = z s + z s . We upsample and softmax z s to produce p s = S(U (z s )) and compute the segmentation loss (1) using it. The segmentation network and the self-attention module are jointly trained by minimizing the source segmentation loss. However, when the training is finished, the trained segmentation network is discarded and only the self-attention module is saved to be used later for the next step which is the actual domain adaptive training of the segmentation network.
When computing the source segmentation loss (1), z s is used instead of z s to make a skip connection which enables the loss signal to directly flow to the segmentation network during back-propagation. If there were no skip connection and the loss were computed on z s , the segmentation network would have been trained to produce output that only performs well after the attention module is applied, eventually producing inaccurate z s . This is not desirable because the attention module has to learn well-represented z s that can perform well on the source domain in order to capture the correct inter-pixel correlation of the source domain. The skip connection prevents this problem and enables the segmentation network to be trained good enough to perform segmentation correctly on the source domain.

Domain adaptive training via SAM
In this section, we explain the domain adaptive training of the segmentation network using the SAM that is pretrained on the source domain. The overall training process is shown in Fig. 2 (b). Please note that the segmentation network in Fig. 2 Fig. 2 (a). We train another separate segmentation network from scratch and this network is the one to be tested for the performance measurement. Since this step is the actual process of domain adaptation, we train the network for both the source and the target doamin datasets. We input images of both domains into the network and get z s and z t respectively. Then, z s and z t are used for two different loss functions. One is the commonly used segmentation loss previously introduced in Sec. 3. More specifically, we upsample and apply softmax on z s and z t to get p s and p t to compute (1) and (3). The other loss function is our proposed selfattention loss. z s and z t are given as inputs to the SAM that has already been pre-trained on the source domain. Inform that the pre-trained SAM is frozen and its weights are not updated during the domain adaptive training of the network. The SAM produces z s and z t which are the self-attended logits. We add z s and z t to them respectively and get z s and z t . The proposed self-attention loss is defined as the L1 loss between z and z or between z and z .

(b) is a different network from the one in
The self-attention loss is defined in two forms because we empirically observe that the self-attention loss shows different performances for different source datasets. The intention of the self-attention loss is to train the segmentation network to learn the self-attended output which contains information about inter-pixel correlations by forcing z to mimic the output of the module, z or z . When computing the self-attention loss, we detach the output of the module, z and z , so the loss signal only flows through z during the back-propagation. The overall objective functions are defined as follows: (11) λ is a hyper-parameter that balances between the segmentation loss and the self-attention loss. We empirically find that for GTA5→Cityscapes task, using (8) and applying selfattention loss for both domains, (10), performs better but for SYNTHIA→Cityscapes, using (9) and applying it only on the target domain, (11), shows better results. This will be further studied in the ablation study in Sec. 5.2. In the inference time, prediction is based only on z, which is the output of the segmentation network without using the SAM.

Datasets and Training Details
Datasets. We conduct UDA experiments using two different source domain datasets, GTA5 [22] and SYN-THIA [23] which are photo-realistic synthetic datasets collected from virtual games scenes. The target domain is the Cityscapes [9] which consists of real scenes under driving scenarios. The GTA5 contains 24,966 images and shares 19 classes with the Cityscapes while the SYNTHIA dataset contains 9,400 images and shares 16 classes in common with the Cityscapes. We additionally report the results of 13 common classes for SYNTHIA. The Cityscapes dataset contains 2,975 training images and 500 validation images. We conduct experiments on the validation set following the standard protocols of previous works.
Training Details. The experiments are mainly conducted using DeepLabV2 [3] with ResNet101 [13] backbone. We also conduct experiment using FCN-8s [19] with VGG16 backbone [25] but its results and training details are in the supplementary. We initialize the backbone of the network with ImageNet pre-trained weights. We use the SGD optimizer with initial learning rate of 2.5×10 −4 and weightdecay of 0.0005. Learning rate is scheduled using 'poly' learning rate policy with a power of 0.9. During the selfattention module training phase, both segmentation network and the SAM are jointly trained by the same optimizer. When domain adaptive training the segmentation network via the self-attention module, we use the pre-trained selfattention module that is saved at the last save point. The network is trained for 120,000 iterations with a batch size of 1. It is tested and saved at every 2,000 iterations. The hyperparameter λ is set to 0.1 empirically. We utilize the source images that are style-transferred into Cityscapes by [43] to further close the domain gap at image level.

Ablation Study
Tab.1 shows our ablation studies. We test our proposed method in different settings and validate which setting works the best for each UDA task. 'No Pseudo' does not employ any self-training via pseudo labels and only trains the network with the style-transferred source domain images. It shows the lowest performance for both UDA tasks. 'Pseudo-Only' refers to applying self-training using the pseudo labels generated by the 'No Pseudo' models. It is trained only by the two loss terms (1) and (3). It clearly shows improved performance on both tasks. 'Ours (8)' and 'Ours (9)' are our proposed method introduced in Sec.4.2. We utilize the same pseudo labels used in 'Pseudo-Only' and additionally employ our proposed self-attention loss to show the efficacy of our method. The performances are significantly improved when our method is applied. However, there are some different tendency between the two UDA tasks. For GTA5→Cityscapes, (8) shows better performance than (9) while (9) results in higher mIoU than (8) on SYNTHIA→Cityscapes. Also, when we apply the self-attention loss only on the target domain ('Ours Only on Target'), employing (11), the performance on SYNTHIA improves but it rather decreases on GTA5. We conjecture this is due to the larger domain gap between SYNTHIA and Cityscapes than GTA5 and Cityscapes. (See supplementary for further discussion about it.) Therefore, we utilize (8) and (10) on GTA5→Cityscapes while (9) and (11)

Iterative Training
Self-training methods [18,[37][38][39] employ iterative training scheme which generates pseudo labels for the target domain from a trained segmentation network and retrains a new network with the generated pseudo labels. Tab. 2 shows the iterative training of 'Pseudo-Only' and 'Ours' on both UDA tasks. 'Gen' stands for a generation which indicates the iteration of the iterative training. For example, Gen1 networks are trained with pseudo labels generated by the 'No Pseudo' models from Tab. 1 and Gen2 networks are trained with pseudo labels generated by Gen1 networks. On GTA5 task, the performance of 'Pseudo-Only' shows the highest at Gen2 and rather decreases after as the generation goes on, while on SYNTHIA task, there is only a marginal improvement between the generations. On the other hand, when training with our proposed self-attention loss, the performance tends to improve better at every generation for both source domains. It implies that our proposed method helps the network not to produce noisy error-prone pseudolabels that can lead to incorrect supervision signals and to generalize better on the target domain by transferring domain invariant inter-pixel correlations from the source domain. Note that the SAM is trained only once on the source domain and the same SAM is reused at every generation for the proposed self-attention loss.

Class-wise comparison between Pseudo-Only and Ours
In Tab. 3 we compare our method with 'Pseudo-Only' by classes to further investigate the effectiveness of our proposed method. As mentioned earlier, the same pseudo labels are utilized for both methods. 'Ours' only additionally employs the proposed self-attention loss along with self-training. For both tasks, the mIoU is increased after applying our method. Our method particularly contributes to the performance gain on rare classes which scarcely exist in the dataset. Wall, fence, pole, light, sign, terrain, rider, truck, bus, train, motor and bike are known rare classes. On GTA5→Cityscapes, IoU of rider, truck, motor, and bike are significantly improved, besides, other rare classes such as fence, pole, sign and bus also show increased IoU. Similarly on SYNTHIA→Cityscapes, 'Ours' shows better performance than 'Pseudo-Only' on pole, light, sign, rider, bus, motor and bike. Other than rare-classes, our method also leads to significant improvement of majors classes such as sky, person and car on both tasks.

Comparisons with state-of-the-art methods
Tab. 4 and Tab. 5 show the comparison experiments with other state-of-the-art methods on both datasets. Our method outperforms other methods except for ProDA [38] which achieved a major breakthrough in the UDA for semantic segmentation. Since our proposed method can be easily applied to other methods by simply adding our self-attention loss term to the overall objective function, we combine our method with ProDA by employing our self-attention loss during the stage3 training phase of ProDA. After applying our method on ProDA, we could observe an additional performance gain for both UDA tasks. On GTA5→Cityscapes, 12 out of 19 classes show increased IoU, especially, the performance on rare classes such as fence, pole, terrain, truck and bus are highly improved. Similar results can be found on SYNTHIA→Cityscapes as well, mIoU is improved along with IoU of some rare-classes. Also, it is noteworthy that when excluding ProDA, there is a more than 1% gap between 'Ours' (49.2%) and the second-best method (47.9%) on SYNTHIA task. We empirically find that λ = 0.005 works the best when combining our method with ProDA. On both UDA benchmarks, our method achieves comparable performance to existing approaches. Moreover, we show that our method can be easily applied to other method and validate its effectiveness by additionally employing it on leading SOTA method and setting a new SOTA performance for both UDA tasks.   Table 5. Comparison results with other methods on SYNTHIA→Cityscapes. The numbers in bold are the best score for each column. mIoU* and mIoU denote mIoU of 13 classes and 16 classes respectively. similarity between the pixel with the highest confidence of the class we want to measure and all the other pixels in the image. The most confident pixel is marked as a blue dot in the visualization. The redder color means higher similarity while the bluer color refers to lower similarity. If the network is trained well, the most confident pixel should have high similarity only with the pixels belonging to the same class. We additionally visualize the results after applying the SAM using z and z ('Ours-z ' & 'Ours-z ') for GTA5 and SYNTHIA respectively.

Qualitative Analysis
When using only the self-training ('Pseudo-Only'), the most confident pixel has high similarity even with irrelevant pixels of different classes, while 'Ours' is more attended well with the relevant pixels corresponding to the same class. The results of SYNTHIA are more noisy due to the larger domain gap. 'Ours-z ' and 'Ours-z ' look a little more noisy than 'Ours' but clearer and more distinct than 'Pseudo-Only'. They give us hint about what knowledge is being transferred via our proposed self-attention loss. We conjecture that their little noisiness provides extra information about inter-pixel correlations to the network. In the prediction visualization, the 3rd and the 4th columns correspond to GTA5 task while the 5th and the 6th columns refer to SYNTHIA task. When comparing only the prediction results, it is hard to see the difference between the two methods, even though 'Ours' look a little more accurate. However, when we visualize the attention by classes and compare them as above, we can clearly observe a distinct difference between the two. It implies that even though their prediction visualization might seem similar, their predicted logits have very different distribution which leads to better performance for 'Ours'. These results are not cherry-picked and more results using different images can be found in the supplementary.   Fig. 4 shows entropy analysis of three different networks on the two UDA tasks. Every 2,000 iteration, we measure the entropy of all the 500 validation images of Cityscapes. For each image, we divide the pixels into two groups, correctly and incorrectly classified, and compute the entropy for each group as follows:

Quantitative Analysis
We plot the average entropy of the 500 images for each pixel group separately in the figure. The solid line is the entropy of correct pixels and the dashed line refers to the incorrect pixels. Each red, blue and green line indicates 'Ours-(8)', 'Ours-(9)' and 'Pseudo-Only' networks respectively.
For both benchmarks and all three networks, incorrect pixels have higher entropy than correct pixels. It is an interesting observation that 'Pseudo-Only' has the lowest entropy while 'Ours-(8)' and 'Ours-(9)' show higher entropy than 'Pseudo-Only' throughout the training. Having higher entropy means that the class distribution of a pixel is not biased towards a certain class (a highly peaky distribution) but rather has probability for some of other classes as well. We conjecture this is due to our self-attention loss providing extra knowledge about inter-pixel correlations which prevents the network from being highly biased and over-confident.

Conclusion
In this paper, we propose a method of transferring domain invariant inter-pixel correlations from the source domain to the target domain by utilizing a self-attention module that is pre-trained on the source domain. Our method is very simple and does not require complex and heuristic algorithm (only one hyper-parameter, λ, to tune) yet very effective. Our method supports self-training by overcoming the defects of noisy pseudo labels with extra knowledge provided by the SAM. Also, we show that our method can be easily used in addition to other methods to further enhance the performance. The proposed method sets a new SOTA scores on both UDA benchmarks when combined with the recent SOTA method [38].  We also conduct experiments using a FCN-8s with VGG16 backbone which is another segmentation network that is widely used in unsupervised domain adaptation for semantic segmentation. We train FCN-8s with VGG16 backbone by ADAM optimizer with an initial learning rate of 1 × 10 −5 and the momentums of 0.9 and 0.99. The learning rate is decayed by 'step' learning rate policy with a step size of 50,000 and a decay rate of 0.1. The hyper-parameter λ is set to 0.001 empirically. Other details are the same as the DeepLabV2 with ResNet101 backbone. Different from DeepLabV2, FCN-8s uses transposed convolution instead of bilinear interpolation for upsampling z.

A.2. Ablation Study
Tab. A.1 shows our ablation study on FCN-8s with VGG16 backbone. As shown in the table, applying our selfattention loss improves the performance for both domains, but its performance gain is somewhat lower than that of DeepLabV2. Moreover, in contrast to DeepLabV2, using (8) instead of (9) and applying our self-attention loss on both domains rather than only on the target domain achieve better performance when using FCN-8s. We conjecture this difference comes from the different architecture of FCN-8s which uses transposed convolution for upsampling z instead of bilinear interpolation used in DeepLabV2. Since our proposed self-attention loss is computed on z and not on U (z), it can not train the transposed convolutional layer of FCN-8s. This could be the possible reason why ours does not show as much performance gain as it shows in the DeepLabV2 experiment.

A.3. Iterative Training
We also conduct iterative training analogous to DeepLabV2. As it can be seen in the Tab. A.2, Both 'Pseudo-Only' and 'Ours' show improved performance as the generation goes on. However, 'Ours' shows better performance improvement than 'Pseudo-Only' between the generations. The performance of 'Pseudo-Only' gets saturated around a certain generation while 'Ours' keeps show-

B. Loss function and Layer Analysis
We conduct experiment of computing the proposed selfattention loss utilizing different loss functions and at different layers of the segmentation network. The experiment is done on GTA5→Cityscapes task using the DeepLabV2 with ResNet101 backbone. 'L1', 'KL-Div' and 'Cosine' refer to L1 loss, Kullback-Leibler divergence loss and cosine similarity loss respectively. 'KL-Div' and 'Cosine' are computed between each logit (row) of z and z . The cosine similarity is calculated between z i ∈ R C and z i ∈ R C which are the logits of z and z respectively. It trains the segmentation network by maximizing the computed cosine similarity or minimizing the negative cosine similarity equivalently. 'KL-Div' loss function tries to minimize the KLdivergence between z i and z i . We first apply softmax on z i and z i and then calculate the KL-divergence between them, D KL (σ(z i )||σ(z i )), here σ refers to the softmax. We train the segmentation network with three different loss functions independently and compare the results. As shown in the Tab. A.5, we could observe that L1 loss shows the best score compared to other two loss functions while 'Cosine' shows the worst score which is even lower than 'Pseudo-Only'.
We also test about on which layer of the segmentation network the self-attention loss would work the best. A segmentation network mainly consists of two parts, a feature extractor, F, and a classification head, H, hence G(x) = H(F(x)). 'Prediction' and 'Feature map' in the Tab. A.5, refer to applying our proposed self-attention loss on z and the feature map, F(x) = f ∈ R h×w×k , where k refers to the number of channels. 'Both' is applying our method on both z and f . We find that computing self-attention loss only on the prediction z achieves the best performance. We conjecture this result is due to the fact that applying our method only on 'Feature map' can not train the classification head and applying on both layers regularizes the network excessively more than necessary. The layer analysis experiments are done using the L1 loss.
C. Selecting the τ c for pseudo label generation τ c is set differently for each class as mentioned in the main paper. We basically follow the pseudo label generation process of [18]. Pseudo labels are generated using a pre-trained segmentation network. The pre-trained network G makes inference on all images in the training set of the target domain to obtain the prediction results. Then, for each class, we collect all the pixels that are classified as the class from the entire predicted results and add the confidence score of each pixel to a list. Therefore, there is one list for each class. We sort each list and choose the median value of the list as the τ c for the corresponding class, hence τ c is set by the confidence score of top 50% of each class. If the chosen median value is higher than 0.9, τ c is set as 0.9.

D. Discussion about why different settings work for different source domains
From the Tab. 1 of the main paper, we could observe that different settings are suitable for different source domains. It is difficult to prove why this tendency happens exactly, but we can conjecture by analysis. Fig. D.1 is the same entropy analysis figure from the main paper (Fig. 3). We think we can get a little hint from it. Ours (8) shows higher entropy than Ours (9) generally on both UDA tasks. However, for SYNTHIA→Cityscapses, Ours (8) show much higher entropy than Ours (9) compared to GTA5→Cityscapses. We can clearly observe the large gap between Ours (8) and Ours (9) on SYNTHIA task. This gap is much larger than that of GTA5 task, especially for incorrect pixels. Ours (8) is defined as minimizing the L1 loss between z and z , where z = z + z . On SYNTHIA task, adding the output of the segmentation network, z to the output of the SAM, z somehow brings noisy and incorrect information and eventually make z corrupted and under-perform.
We guess this is because of the larger domain gap between SYNTHIA and Cityscapes than GTA5 and Cityscapes. In fact, SYNTHIA has very different data distribution from Cityscapses and GTA5. Cityscapses and GTA5 are both collected under driving scenario but SYNTHIA is not, it has more images taken from higher position such as traffic surveillance cameras. Also its number of classes shared with Cityscapses (16 classes) is less than GTA5 (19 classes). Therefore due to this larger domain gap, z itself does not contain well-represented domain-invariant information that can improve the performance but rather deteriorates performance when combined with z . On the other hand z which is refined version of z with the help of the SAM, contains domain-invariant information than can further boost the performance. For this reason, we conjecture that it is better to just follow z instead of z for SYNTHIA task.
We think the same reason applies to why using the selfattention loss only on the target domain works better than using it on both domains. If the self-attention loss is applied on both the source domain and the target domain, the network could be more overfitted and trained towards the source domain. This is not desirable especially when there is a large domain gap between the source and the target domains, such as SYNTHIA and Cityscapes.

E.2. Pixel-wise similarity visualization
In Fig. E.4, we show visualization of pixel-wise similarity. We visualize how each logit of predicted pixel (z ∈ R hw×C ) is similar to other pixels. It is computed as follows: It is basically an attention map of z itself. We take ReLU on the attention map to visualize the difference in positive correlation between pixels more prominently. We visualize this for both 'Ours' and 'Pseudo-Only'. For the ground truth, we use the nearest interpolation to resize the ground truth label to the spatial size of h × w from H × W , hence y t ∈ R h×w×C where each pixel is a C dimensional one-hot vector. We flatten y t in the spatial dimension (y t ∈ R hw×C ) and compute M by inserting y t instead of z in (13). The experiment is conducted using DeepLabV2 with ResNet101 backbone segmentation network that is trained on GTA5→Cityscapses task. We sample several images from validation set of Cityscapses and visualize the pixelwise similarity described as above. In the figure, bluer means higher similarity. Since 'GT' is visualized using the one-hot vectors of y t , its each element is either 0 or 1 while elements of 'Ours' and 'Pseudo-Only' are between 0 and 1.
As shown in the figure, 'Ours' show much similar results to 'GT' compared to 'Pseudo-Only' and clear difference between 'Ours' and 'Pseudo-Only' can be observed. It means that each pixel of z which is a C dimensional logit, is more attended well with the correct pixels corresponding to the same class and dissimilar to other pixels of different classes. On the other hand, pixels of 'Pseudo-Only' are attended with even irrelevant pixels showing high similarity.