{'fraction_non_alphanumeric': 0.04204759578477214, 'fraction_numerical': 0.016784144043548048, 'mean_word_length': 4.776859504132231, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation

Minimizing the discrepancy of feature distributions between different domains is one of the most promising directions in unsupervised domain adaptation. From the perspective of distribution matching, most existing discrepancy-based methods are designed to match the second-order or lower statistics, which however, have limited expression of statistical characteristic for non-Gaussian distributions. In this work, we explore the benefits of using higher-order statistics (mainly refer to third-order and fourth-order statistics) for domain matching. We propose a Higher-order Moment Matching (HoMM) method, and further extend the HoMM into reproducing kernel Hilbert spaces (RKHS). In particular, our proposed HoMM can perform arbitrary-order moment tensor matching, we show that the first-order HoMM is equivalent to Maximum Mean Discrepancy (MMD) and the second-order HoMM is equivalent to Correlation Alignment (CORAL). Moreover, the third-order and the fourth-order moment tensor matching are expected to perform comprehensive domain alignment as higher-order statistics can approximate more complex, non-Gaussian distributions. Besides, we also exploit the pseudo-labeled target samples to learn discriminative representations in the target domain, which further improves the transfer performance. Extensive experiments are conducted, showing that our proposed HoMM consistently outperforms the existing moment matching methods by a large margin. Codes are available at \url{https://github.com/chenchao666/HoMM-Master}


Introduction
Convolutional neural networks (CNNs) have shown promising results on supervised learning tasks. However, the performance of a learned model always degrades severely when dealing with data from the other domains. Considering that constantly annotating massive samples from new domains is expensive and impractical, unsupervised domain adaptation (UDA), therefore, has emerged as a new learning framework to address this problem (Csurka 2017). UDA aims to utilize full-labeled samples in source domain to annotate the completely-unlabeled target domain samples. Thanks to deep CNNs, recent advances in UDA show satisfactory performance in several computer vision tasks (Hoffman et al.  Figure 1: 300 points in R 2 and the level sets of the moment tensor. As observed, higher-order moment tensor captures the shape of the cloud of samples more accurately. 2018). Among them, most methods bridge the source and target domain by learning domain-invariant features. These dominant methods can be further divided into two categories: (1) Learning domain-invariant features by minimizing the discrepancy between different distributions Long et al. 2017).
(2) Encouraging domain confusion by a domain adversarial objectives whereby a discriminator (domain classifier) is trained to distinguish between the source and target representations. (Ganin et al. 2016;Tzeng et al. 2017;Hoffman et al. 2018).
Most existing discrepancy-based methods in UDA are based on Maximum Mean Discrepancy (MMD) (Long et al. 2017) or Correlation Alignment (CORAL) , which are designed to match the first-order (Mean) and second-order (Covariance) statistics of different distributions. However, for the real world applications, the deep features are always a complex, non-Gaussian distribution, which can not be completely characterized by its first-order or second-order statistics (Jia and Darrell 2011;Xu et al. 2016). Therefore, aligning the second-order or lower statistics only guarantees coarse-grained alignment of two distributions. To address this limitation, we propose to perform domain alignment by matching the higher-order (mainly refer to third-and fourth-order) statistics, which contain more discriminative information and can better represent the feature distribution. Inspired by (Pauwels and Lasserre 2016), Fig.1 illustrates the metrics of higher-order moment tensor, where we plot a cloud of points (consists of three different Gaussians) and the level sets of moment tensor with different order. As observed, the higher-order moment tensor characterizes the distribution more accurately.
Our contribution can be concluded as: (1) We propose a Higher-order Moment Matching (HoMM) method to minimize the domain discrepancy, which is expected to perform fine-grained domain alignment. The HoMM integrates the MMD and CORAL into a unified framework and generalizes the first-order and second-order moment matching to higherorder moment tensor matching. Without bells and whistles, the third-and fourth-order moment matching outperform all existing discrepancy-based methods by a large margin. (2) Due to lack of labels in the target domain, we propose to learn discriminative clusters in the target domain by assigning the pseudo-labels for the reliable target samples, which also improves the transfer performance.

Related Work
Learning Domain-Invariant Features To minimize the domain discrepancy and learn domain-invariant features, various distribution discrepancy metrics have been introduced. The representative ones include Maximal Mean Discrepancy (MMD) (Gretton et al. 2012;Long et al. 2017), KLdivergence, Correlation Alignment Chen et al. 2019) and Wasserstein distance (Lee et al. 2019). MMD was first introduced for the two-sample tests problem (Gretton et al. 2012), and is currently the most widely used metric to measure the distance between two feature distributions. Specifically, Long et al. proposed DAN (Long et al. 2015) and JAN (Long et al. 2017) which perform domain matching via multi-kernel MMD or a joint MMD criteria in multiple domain-specific layers across domains. Sun et al. proposed the correlation alignment (CORAL) (Sun, Feng, and Saenko 2016; to align the second order statistics of the source and target distributions. Some recent work also extended the CORAL into reproducing kernel Hilbert spaces (RKHS) (Zhang et al. 2018) or deployed alignment along geodesics by considering the log-Euclidean distance (Morerio, Cavazza, and Murino ). Interestingly, (Li et al. 2017b) theoretically demonstrated that matching the second order statistics is equivalent to minimizing MMD with the second order polynomial kernel. Besides, the approach most relevant to our proposal is the Central Moment Discrepancy (CMD) (Zellinger et al. 2017), which matches the higher order central moments of probability distributions. Both CMD and our HoMM propose to match the higher-order statistics for domain alignment. The CMD matches the higher-order central moment while our HoMM matches the higher-order cumulant tensor. Another fruitful line of work tries to learn the domain-invariant features through adversarial training (Ganin et al. 2016;Tzeng et al. 2017). These efforts encourage domain confusion by a domain adversarial objective whereby a discriminator (domain classifier) is trained to distinguish between the source and target representations. Also, recent work performing pixel-level adaptation by image-to-image transformation (Hoffman et al. 2018) has achieved satisfactory performance and obtained much attention. In this work, we propose a higher-order moment matching method, which shows great superiority over existing domain matching methods. Higher-order Statistics The statistics higher than first-order has been successfully used in many classical and deep learn-ing methods (De Lathauwer, Castaing, and Cardoso 2007;Koniusz et al. 2016;Gou, Camps, and Sznaier 2017). Especially in the field of fine-grained image/video recognition, second-order statistics such as Covariance and Gaussian descriptors, have demonstrated better performance than descriptors exploiting zeroth-or first-order statistics (Li et al. 2017a;Wang, Li, and Zhang 2017). However, using second-order or lower statistical information might not be enough when the feature distribution is non-Gaussian (Gou, Camps, and Sznaier 2017). Therefore, the higherorder (mainly refer to third-order and fourth-order) statistics have been explored in many signal processing problems (Mansour and Jutten 1995;Jakubowski et al. 2002;De Lathauwer, Castaing, and Cardoso 2007;Gou, Camps, and Sznaier 2017). In the field of Blind Source Separation (BSS) (De Lathauwer, Castaing, and Cardoso 2007;Mansour and Jutten 1995), for example, the fourth-order statistics are widely used to identify different signals from mixtures. Gou et al. utilizes the third-order statistics for person ReID (Gou, Camps, and Sznaier 2017). Xu et al. exploits the third-order cumulant for blind image quality assessment (Xu et al. 2016). In (Jakubowski et al. 2002;Koniusz et al. 2016), the authors exploit higher-order statistics for image recognition and detection. Matching the second order statistics can not ensure two distributions inseparable, just as using the second order statistics can not identifies different signals from mixtures (De Lathauwer, Castaing, and Cardoso 2007). That's why we explore higherorder moment tensor for domain matching.

Method
In this work, we consider the unsupervised domain adaptation problem. Let D s = {x i s , y i s } ns i=1 denotes the source domain with n s labeled samples and D t = {x i t } nt i=1 denotes the target domain with n t unlabeled samples. Given D s ∪D t , we aim to train a cross-domain CNN classifier f θ (x) which can minimize the target risks t = E x∈Dt [f θ (x) = y t ].
Here f θ (x) denotes the outputs of the deep neural networks, θ denotes the model parameter to be learned. Following (Long et al. 2017;Chen et al. 2019), we adopt the twostream CNNs architecture for unsupervised deep domain adaptation. As shown in Fig where L s represents the classification loss in the source domain, J(·, ·) represents the cross-entropy loss function. L d represents the domain discrepancy loss and λ d is the trade-off parameter. As aforementioned, most of existing (2) (L) Figure 2: Two-stream CNNs with shared parameters are adopted for unsupervised deep domain adaptation.
discrepancy-based methods are designed to minimize distance of the second-order or lower statistics between different domains. In this work, we propose a higher-order moment matching method, which matches the higher-order statistics of different domains.

Higher-order Moment Matching
To perform fine-grained domain alignment, we propose a higher-order moment matching as where n s = n t = b (b is the batch size) during the training process. φ θ (x) denotes the activation outputs of the adapted layer. As illustrated in Fig. 2 ] ∈ R L denotes the activation outputs of the i-th sample, L is the number of hidden neurons in the adapted layer. Here, u ⊗p denotes the p-level tensor power of the vector u ∈ R c . That is where ⊗ denotes the outer product (or tensor product). We have u ⊗0 = 1, u ⊗1 = u and u ⊗2 = u ⊗ u. The 2-level tensor product u ⊗2 ∈ R c 2 defined as Instantiations According to Eq. (3), when p = 1, the first-order moment matching is equivalent to the linear MMD (Tzeng et al. 2014), which is expressed as When p = 2, the second-order HoMM is formulated as, HoMM(p=2) HoMM(p=3) Figure 3: An illustration of first-order, second-order and third-order moments in the source domain. HoMM matches the higher-order (p ≥ 3) moment across different domains. where Therefore, the second-order HoMM is equivalent to the Gram matrix matching, which is also widely used for cross-domain matching in neural style transfer (Gatys, Ecker, and Bethge 2016;Li et al. 2017b) and knowledge distillation (Yim et al. 2017). Li et al. (Li et al. 2017b) theoretically demonstrate that matching the Gram matrix of feature maps is equivalent to minimize the MMD with the second order polynomial kernel. Besides, when the activation outputs are normalized by subtracting the mean value, the centralized Gram matrix turns into the Covariance matrix. In this respect, the secondorder HoMM is also equivalent to CORAL, which matches the Covariance matrix for domain matching.
As illustrated in Fig. 3, in addition to the first-order moment matching (e.g. MMD) and the second-order moment matching (e.g. CORAL and Gram matrix matching), our proposed HoMM can also perform higher-order moment matching when p ≥ 3. Since higher-order statistics can characterize the non-Gaussian distributions better, applying higher-order moment matching is expected to perform finegrained domain alignment. However, the space complexity of calculating the higher-order tensor u ⊗p (p ≥ 3) reaches O(L p ), which makes the higher-order moment matching infeasible in many real-world applications. Adding bottleneck layers to shrink the length of adaptive layer does not even solve the problem. When L = 128, for example, the dimension of a third-order tensor still reaches O(10 6 ), and the dimension of a fourth-order tensor reaches O(10 8 ), which is absolutely computational-unfriendly. To address this problem, we propose two practical techniques to perform the compact tensor matching.
Group Moment Matching. As the space complexity grows exponentially with the number of neurons L, one practical approach is to divide the hidden neurons in the adapted layer into n g groups, with each group L/n g neurons. Then we can calculate and match the high-level tensor in each group respectively. That is, where h i :,k ∈ R L/ng is the activation outputs of k-th group. In this way, the space complexity can be reduced from O(L p ) to O(n g · L/n g p ). In practice, L/n g ≥ 25 need to be satisfied to ensure satisfactory performance.
Random Sampling Matching. The group moment matching can work well when p = 3 and p = 4, but it tends to fail when p ≥ 5. Therefore, we also propose a random sampling matching strategy which is able to perform arbitrary-order moment matching. Instead of calculating and matching two high-dimensional tensors, we randomly select N values in the high-level tensor, and only calculate and align these N values in the source and target domains. In this respect, the p-order moment matching with random sampling strategy can be formulated as, rnd [k,p] j=rnd [k,1] rnd [k,p] j=rnd [k,1] h i t (j)] 2 (9) where rnd ∈ R N ×p denotes the randomly generated position index matrix, rnd[k, j] ∈ {1, 2, 3, · · · , L}. Therefore, rnd [k,p] j=rnd [k,1] h i s (j) denotes a randomly sampled value in the p-level tensor h i s ⊗p . With the random sampling strategy, we can perform arbitrarily-order moment matching, and the space complexity can be reduced from O(L p ) to O(N ).
In practice, the model can achieve very competitive results even N = 1000.

Higher-order Moment Matching in RKHS
Similar to the KMMD (Long et al. 2017), we generalize the higher-order moment matching into reproducing kernel Hilbert spaces (RKHS) as well. That is,  [k,p] j=rnd [k,1] h i s (j), k = 1, · · · , N. In this respect, the domain matching loss can be formulated as where k(x, y) = exp(−γ x − y 2 ) is the RBF kernel function. Particularly, when p = 1, the kernelized HoMM (KHoMM) is equivalent to the KMMD.

Discriminative Clustering
When the target domain features are well aligned with the source domain features, the unsupervised domain adaptation turns into the semi-supervised classification problem, where the discriminative clustering in the unlabeled data is always encouraged (Grandvalet and Bengio 2005;Xie, Girshick, and Farhadi 2016). There have been a lot of work trying to learn the discriminative clusters in the target domain (Shu et al. 2018;Morerio, Cavazza, and Murino ), most of which minimize the conditional entropy to ensure the decision boundaries do not cross high-density data regions, where c is the number of classes, p j is the softmax output of j-th node in the output layer. We find that the entropy regularization works well when the target domain has high test accuracy, but it helps little or even downgrades the accuracy when the test accuracy is unsatisfactory. The reason can be drawn that the classifier may be misled as a result of entropy regularization enforcing over-confident probability on some misclassified samples. Instead of clustering in the output layer by minimizing the conditional entropy, we propose to cluster in the shared feature space. First, we pick up highly confident predicted target samples whose predicted probabilities are greater than a given threshold η, and assign pseudo-labels to these reliable samples. Then, we penalize the distance of each pseudo-labeled sample to its class center. The discriminative clustering loss can be given as whereŷ i t is the assigned pseudo-label of x i t , cŷi t ∈ R L denotes its estimated class center. As we perform update based on mini-batch, the centers can not be accurately estimated by a small size of samples. Therefore, we update the class center in each iteration via moving average method. That is, where α is the learning rate of the center. c t j is the class center of j-th class in t-th iteration. δ(ŷ i t = j) = 1 if x i t belongs to j-th class, otherwise it should be 0.

Full Objective Function
Based on the aforementioned analysis, to enable effective unsupervised domain adaptation, we propose a holistic approach with an integration of (1) source domain loss minimization, (2) domain alignment with the higher-order moment matching and (3) discriminative clustering in the target domain. The full objective function is as follows, where L s is the classification loss in the source domain, L d is the domain discrepancy loss measured by the higherorder moment matching, and L dc denotes the discriminative clustering loss. Note that in order to obtain reliable pseudolabels for discriminative clustering, we set λ dc = 0 during the initial iterations, and enable the clustering loss L dc after the total loss tends to be stable.

Experiments Setup
Dataset. We conduct experiments on three public visual adaptation datasets: digits recognition dataset, Office-31 dataset, and Office-Home dataset. The digits recognition dataset includes four widely used benchmarks: MNIST, USPS, Street View House Numbers (SVHN), and SYN (synthetic digits dataset). We evaluate our proposal across three typical transfer tasks, including: SVHN→MNIST, USPS→MNIST and SYN→MNIST. The details of this dataset can be seen in (Chen et al. 2019). Office-31 is another commonly used dataset for real-world domain adaptation scenario, which contains 31 categories acquired from the office environment in three distinct image domains: Amazon (product images download from amazon.com), Webcam (low-resolution images taken by a webcam) and Dslr (high-resolution images taken by a digital SLR camera). The office-31 dataset contains 4110 images in total, with 2817 images in A domain, 795 images in W domain and 498 images in D domain. We evaluate our method on all the six transfer tasks as (Long et al. 2017). The Office-Home dataset (Venkateswara et al. 2017) is a more challenging dataset for domain adaptation, which consists of images from four different domains: Artistic images (A), Clip Art images (C), Product images (P) and Real-world images (R). The dataset contains around 15500 images in total from 65 object categories in office and home scenes. Baseline Methods. We compare our proposal with the following methods, which are most related to our work: In our experiments on digits recognition dataset, we utilize the modified LeNet whereby a bottleneck layer with 90 hidden neurons is added before the output layer. Since the image size is different across different domains, we resize all the images to 32 × 32 and convert the RGB images to grayscale. For the experiments on Office-31, we use ResNet-50 pretrained on ImageNet as our backbone networks. And we add a bottleneck layer with 180 hidden nodes before the output layer for domain matching. It is worth noting that the relu activation function can not be applied to the adapted layer, as relu activation function will make most of the values in the high-level tensor h i s ⊗p to be zero, which will make our HoMM fail. Therefore, we adopt tanh activation function in the adapted layer. Due to the small samples size of Office-31 and Office-Home We denote SVHN, MNIST, USPS as SN, MT and US respectively.
datasets, we only update the weights of the full-connected layers (fc) as well as the final block (scale5/block3), and fix other parameters pretrained on ImageNet. Follow the standard protocol of (Long et al. 2017), we use all the labeled source domain samples and all the unlabeled target domain samples for training. All the comparison methods are based on the same CNN architecture for a fair comparison. For DDC, DAN, CORAL and CMD, we embed the official implementation code into our model and carefully select the trade-off parameters to get the best results. When training with ADDA, our adversarial discriminator consists of 3 fully connected layers: two layers with 500 hidden units followed by the final discriminator output. For other compared methods, we report the results in the original paper directly. Parameters. Our model is trained with Adam Optimizer based on Tensorflow. Regarding the optimal hyperparameters, they are determined by applying multiple experiments using grid search strategy. The optimal hyperparameters may be distinct across different transfer tasks. Specifically, the trade-off parameters are selected from λ d = {1, 10, 10 2 , · · · , 10 8 }, λ dc ∈ {0.01, 0.03, 0.1, 0.3, 1.0}. For the digits recognition tasks, the hyper-parameter λ d is set to 10 4 for third-order HoMM and set to 10 7 for fourth-order HoMM. For the experiments on Office-31 and Office-Home, λ d is set to 300 for the third-order HoMM and set to 3000 for the fourth-order HoMM. Besides, the hyper-parameter γ in RBF kernel is set to 1e-4 across the experiments, the learning rate of the centers is set to α = 0.5 for digits dataset and set to α = 0.3 for Office-31 and Office-Home dataset. The threshold η of the predicted probability is chosen from {0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95}, and the best results are reported. The parameter sensitivity can be seen in Fig. 5.

Experimental results
Digits Dataset For the experiments on digits recognition dataset, we set the batch size as 128 for each domain and  set the learning rate as 1e-4 throughout the experiments. Table 1 shows the adaptation performance on three typical transfer tasks based on the modified LeNet. As can be seen, our proposed HoMM yields notable improvement over the comparison methods on all of the transfer tasks. In particular, our method improves the adaption performance significantly in the hard transfer tasks SVHN→MNIST. Without bells and whistles, the proposed third-order KHoMM achieve 97.2% accuracy, improving the second-order moment matching (CORAL) by +8%. Besides, the results also indicate that the third-order HoMM outperforms the fourthorder HoMM and slightly underperforms the KHoMM. Office-31 Table 2 lists the test accuracies on Office-31 dataset. We set the batchsize as 70 for each domain. The learning rate of the fc layer parameters is set as 3e-4 and the learning rate of the conv layer (scale5/block3) parameters is set as 3e-5. As observed, the fourth-order HoMM outperforms the third-order HoMM and achieves the best results among all the moment-matching based methods. Besides, it is worth noting that the fourth-order HoMM outperforms the second-order statistics matching (CORAL) by more than 10% on several representative transfer tasks A→W, A→D and D→A, which demonstrates the merits of our proposed higher-order moment matching.
Office-Home Table 3 gives the results on the challenged Office-Home dataset. The parameter settings are the same as in Office-31. We only evaluate our method on 5 out of 12 representative transfer tasks due to the space limitation. As we can see, on all the five transfer tasks, the HoMM outperforms the DAN, CORAL, DANN by a large margin and also outperforms the JAN by 3%-5%. Note that the experimental results of the compared methods are reported from (Wang et al. 2019) directly. The results in Table 1, Table 2 and Table 3 reveal several interesting observations: (1) All the domain adaptation methods outperform the source only model by a large margin, which demonstrates that minimizing the domain discrepancy contributes to learning more transferable representations.
(2) Our proposed HoMM significantly outperforms the discrepancy-based methods (DDC, CORAL, CMD), and the adversarial training based methods (DANN, ADDA and CyCADA), which reveals the advantages of matching the higher-order statistics for domain adaptation. (3) The JAN performs slightly better than the third-order HoMM on several transfer tasks, but it's always not as good as the fourthorder HoMM in spite of aligning the joint distributions of multiple domain-specific layers across domains. The performance of our HoMM will be improved as well if we utilize such a strategy. (4) The kernelized HoMM (KHoMM) consistently outperforms the plain HoMM, but the improvement seems limited. We believe the reason is that, the higherorder statistics are originally the high-dimensional features, which conceals the advantages of embedding the features into RKHS. (5) In all transfer tasks, the performance increases consistently by employing the discriminative clustering in target domain. In contrast, entropy regularization improves the transfer performance when the test accuracy is high, but it helps little or even downgrades the performance  We denote SVHN and MNIST as SN and MT respectively.

Analysis
Feature Visualization We utilize t-SNE to visualize the deep features on the tasks SVHN→MNIST by ResNet-50, KMMD, CORAL, HoMM(p=3) and the Full Loss model. As shown in Fig. 4, the feature distributions of the source only model in (a) suggests that the domain shift between SVNH and MNIST is significant, which demonstrates the necessity of performing domain adaptation. Besides, the global distributions of the source and target samples are well aligned with the KMMD (b) and CORAL (c), but there are still many samples being misclassified. With our proposed HoMM, the source and target samples are aligned better and categories are discriminated better as well.
First/Second-order versus Higher-order We also provide the performance of different order moment matching on three typical transfer tasks. As shown in table 4, the order is chosen from p ∈ {1, 2, 3, 4, 5, 6, 10}. The results show that the third-order and fourth-order moment matching significantly outperform the other order moment matching. When p ≤ 3, the higher the order, the higher the accuracy. When p ≥ 4, the accuracy will decrease as the order increases. Regarding why the fifth-order and above perform worse than the fourth-order, one reason we believe is that the fifth-order and above moments can't be accurately estimated due to the small sample size problem (Raudys and Jain 1991).
Parameter Sensitivity and Convergence We conduct empirical parameter sensitivity on SVHN→MNIST and A→W in Fig. 5(a)-(d). The evaluated parameters include two tradeoff parameters λ c , λ dc , the number of selected values in Random Sampling Matching N , and the threshold η of the predicted probability. As we can see, our model is quite sensitive to the change of λ dc and the bellshaped curve illustrates the regularization effect of λ d and λ dc . The convergence performance is provided in Fig. 5(e), which shows that our proposal converges fastest compared with other methods. It is worth noting that, the test error of the Full Loss model has a obvious mutation at the 2.0 × 10 4 iteration where we enable the clustering loss L dc , which also demonstrates the effectiveness of the proposed discriminative clustering loss.

Conclusion
Minimizing statistic distance between source and target distributions is an important line of work for domain adaptation. Unlike previous methods that utilize the second-order or lower statistics for domain alignment, this paper exploits the higher-order statistics for domain alignment. Specifically, a higher-order moment matching is presented, which integrates the MMD and CORAL into a unified framework and generalizes the existing first-and second-order moment matching to arbitrary-order moment matching. We experimentally demonstrate that the third-and fourth-order moment matching significantly outperform the existing moment matching methods. Besides, we also extend the HoMM into RKHS and learn the discriminative clusters in the target domain, which further improves the adaptation performance. The proposed HoMM can be easily integrated into other domain adaptation model, and it is also expected to benefit the knowledge distillation and image style transfer.