{'fraction_non_alphanumeric': 0.0357355568790947, 'fraction_numerical': 0.009498134854706748, 'mean_word_length': 4.802473626773372, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Privacy-Preserving Unsupervised Domain Adaptation Framework for Clinical Text Analysis

Unsupervised domain adaptation (UDA) generally aligns the unlabeled target domain data to the distribution of the source domain to mitigate the distribution shift problem. The standard UDA requires sharing the source data with the target, having potential data privacy leaking risks. To protect the source data's privacy, we first propose to share the source feature distribution instead of the source data. However, sharing only the source feature distribution may still suffer from the membership inference attack who can infer an individual's membership by the black-box access to the source model. To resolve this privacy issue, we further study the under-explored problem of privacy-preserving domain adaptation and propose a method with a novel differential privacy training strategy to protect the source data privacy. We model the source feature distribution by Gaussian Mixture Models (GMMs) under the differential privacy setting and send it to the target client for adaptation. The target client resamples differentially private source features from GMMs and adapts on target data with several state-of-art UDA backbones. With our proposed method, the source data provider could avoid leaking source data privacy during domain adaptation as well as reserve the utility. To evaluate our proposed method's utility and privacy loss, we apply our model on a medical report disease label classification task using two noisy challenging clinical text datasets. The results show that our proposed method can preserve source data's privacy with a minor performance influence on the text classification task.


I. INTRODUCTION
Unsupervised domain adaptation (UDA) has been explored broadly to address the data domain-wise distribution shift and transfer the source model to unlabelled target with impressive results in a wide variety of areas, including computer vision [23], [24], [44], [47], natural language processing (NLP) [5], [41], [49], and others [14], [34]. The general idea of unsupervised domain adaptation is to learn a domain-invariant feature space by aligning the target data's feature distribution to the source data's feature distribution. Several different feature alignment approaches were developed including adversarial domain adaptation [14], maximum mean discrepancy [24], [44], contrastive learning guided feature alignment [23], and others [28]. While there are numerous publications about the UDA in computer vision tasks [13], [23], [24], [27], [30], [44], [47], the UDA in text analysis remains an under-explored area. Directly applying the UDA methods on the text data has several difficulties: 1) the text data are discrete so that domains in text data are more ambiguous [8]; 2) pretrained text encoder such as BERT [7] is prone to overfitting among domains due to the large model size [43]. Consequently, designing a UDA method to generalize on the text data requires meticulous attention.
Besides the general problems in text UDA, the data privacy concern is another big issue [16]. In the general UDA setting, the pretrained source model and source data are shared to target clients for training a domain adaptation model. However, this setting not only raises the data privacy concern on source data but sharing the source data, e.g., from some medical institutions, is not even allowed by laws or regulations [20], [21]. Previously, only a few works have explored the source privacy-preserved method to protect the privacy of source data by sharing the source feature's distribution instead of the source data [28], [47], [53]. However, these methods are still vulnerable to the membership inference attack [37], [38], [45], allowing the attackers to reveal an individual's membership in the source dataset from the disclosure of overall statistical information [9], [42], [45]. Figure 1 shows an example of the membership inference attack under the UDA setting when sharing the source feature's distribution to target clients. A hospital database maintains the populations of many different diseases and only provides some features with desensitized information to the outside, e.g., the distribution of each particular disease. Intuitively, the released distribution won't compromise each individual's privacy. However, an adversary could reveal an individual's physical condition by comparing disease distributions before and after that patient's visit, i.e., if the diabetes population grows by one or more generally the feature distribution of the diabetes changes after that person's visit, we can infer this new patient has diabetes.
To deal with the issues described above, we propose a privacy-preserving UDA method. At the algorithm level, we share the high-level source feature's distribution modeled by Gaussian Mixture Models (GMMs) instead of the source data along with the source feature extractor and source classifier to the target client. The target client will resample new source features from the GMM and train the domain adaptation model locally. To further defend against the membership inference attack on the statistic information of source data (source feature's GMM), we further propose to pretrain the source feature extractor with differential privacy (DP) as DP is a [10] gold standard of protecting the training data's privacy against membership inference attack. In the end, the DP-pretrained source feature extractor with GMM can be shared with the target client to facilitate any form of domain adaptions.
Our proposed method, the Privacy-Preserving Unsupervised Domain Adaptation Framework with Differential Privacy is shown in Figure 2, has the following contributions: • To resolve the BERT overfitting issue as mentioned above, we opt for the knowledge distillation (KD) loss by smoothing the features of target data between source and target models [43]. • To protect the source data privacy in UDA, we propose to model source feature's distributions in GMMs and use the resampled source feature in target client for domain adaptation training. To further protect from leaking the individual information from the shared source feature distribution, we design the DP-pretrained source BERT model following [1]. By sharing only the source feature distribution, we (substantially) limit the privacy loss of source data within a predefined privacy budget. Note that DP cannot guarantee the data privacy absolutely but can ensure only a limited amount of risk is exposed by sharing the source feature distribution in our case [9]. • Our proposed method is compatible with many UDA methods. In our implementation, we adopt two stateof-art UDA backbones: Domain-Adversarial Training of Neural Networks (DANN) [13], and Conditional Adversarial Domain Adaptation (CDAN) [30].
• We firstly study a very challenging task: transferring a disease label extraction model to unlabelled clinical text, which has great potential to be applied to improve realworld clinical workflow. Many clinical text datasets are created by rule-based automatic labeling tools to save time and reduce cost as labeling the clinical text is extremely expensive due to requiring expert knowledge and excess labor [22]. However, these auto-labeled clinical text datasets contain a large number of noisy labels for clinical text [18]. • We evaluate our approach on two clinical text diseaselabel extraction datasets. Our method even outperforms the source data available method significantly, suggesting our method is robust to noisy labels.

A. Unsupervised Domain Adaptation with Sharing Source Data
UDA aims to transfer a source model (the feature extractor and the classifier) to the target domain with only unlabelled data by learning a shared domain invariant feature space to resolve the feature distribution shift problem among different domains [29], [31], [50]. It is generally achieved by minimizing the distribution discrepancy between source and target data with the following methods: maximum mean discrepancy (MMD) [23], [29]- [31], [50], matching the distribution statistical moments at different order [3], [48] , minimum global transportation cost [2], [52], adversarial learning [13] and conditional adversarial learning [17], [30], [39]. However, all these domain adaption methods require sharing the source data to target clients, which are not suitable for privacysensitive data. Moreover, sharing source and target data online could lead to potential security issues, and expensive network transmission costs [16].

B. Source Data-Protected Domain Adaptation
Source data-protected domain adaption has been proposed to protect source data privacy with only the source classifier and feature extractor provided to the target client for domain adaption, including two major types: 1) Two-stage shared source and target feature extractor method. It first clusters the source features by class labels, then uses unsupervised clustering methods to align target features to the source feature clusters [4], [27]. However, shared source and target feature extractors are only suitable for the small domain distribution shift problem. 2) Separate source and target feature extractors. [49] firstly proposed to train separate feature extractors on the source and target domain for the significant domain shift problem. Several end-to-end source data-protected domain adaption methods were developed by learning the target domain feature extractor that automatically clusters target features (unsupervised or self-supervised) and maximizes the information maximization (IM) loss on the source classifier to find the optimal target decision boundary [25], [28]. Unfortunately, these source data-protected DA methods are still vulnerable to the membership inference attack who can infer the individual's information from the shared source model and classifier.

C. Domain Adaptation with Privacy Concern
Some previous literature has studied the data privacy issue under the DA setting. [46] protects the data privacy without sharing the sensitive data between source and target domains by aggregating small source models into a central server in the federated setting. [26] proposed randomizing the shared local model weights to protect from gradient leakage attack. [51] performed domain adaptation in an adversarial-training manner. However, [33] has shown that although adversarial training can make it difficult for the adversary to recover the input data, it cannot entirely remove the sensitive information from the data representations. To the best of our knowledge, leveraging differential privacy to protect source data's privacy has not been studied so far. We firstly study the DP-based source privacy-preserved UDA method in this work.
Differential privacy (DP) provides a strong privacy guarantees for algorithms on aggregate databases [1], [11], [12]. In our application, the source dataset consists of many text-label pairs. According to the DP definition, two datasets are adjacent if they differ in one element, i.e., one text-label pair is added or removed from the source dataset.
Definition 1. A randomized mechanism M : D → R maps an entry of D to R. The mechanism M is ( , δ)-differential privacy if for any two adjacent inputs d, d ∈ D and any outputs S ∈ R it holds that From the definition by Dwork et al. [10], the term quantifies how much we tolerate the output of a model on one database d to be different from the outputs of the same model on an adjacent database d that only has one different training sample, while δ is the failure rate which we tolerate the guarantee to not hold. Another property of DP is particularly useful for our application is the post-processing property that any computation applied to the output of an ( , δ)-DP algorithm maintains ( , δ)-DP [11], [33]. This property allows us to apply any post-processing algorithms to the DP-pretrained source model without sacrificing its DP property.

D. Cross-Domain Text Classification
Most of the above DA works focused on image classification tasks with models pretrained on Imagenet. The text-based domain adaptation is less explored and has several differences compared to image domain adaptation. Firstly, different imaging domains are easily separable (such as natural images and sketch images), and therefore the feature extractor has less chance to overfit between two domains. The domains of text data are ambiguous since similar text might have different meanings on different domains [8] [14]. To this end, a pretrained deep learning text analysis model, such as BERT, is likely to overfit target and source domains. Yu et al. [43] used the knowledge distillation (KD) loss to prevent the overfitting of BERT on the target data with significant performance improvement. We follow a similar framework using the pretrained BERT and KD loss to address the overfitting issue.

III. METHOD
This section presents the building blocks of our proposed method. We begin by introducing the general UDA framework, then the source GMM resampling, followed by pretraining source feature extractor with DP, next the KD loss to smooth the BERT outputs, and a small trick of information maximization to enhance UDA. In the end, we wrap up the privacypreserving UDA with the DP framework.

A. Unsupervised Domain Adaptation Notations
Most current DA methods assume there is a shared domain invariant feature space Z and they can learn feature extractors E s : X s → Z and E t : X t → Z mapping the source and target data into the shared space Z.
The same classifier C : Z → Y trained on the labeled source data can be applied to the target data in the shared feature space Z. Both E and C are neural networks parameterized by θ E and θ C .
The classic domain adaptation methods can be divided into two categories: 1) to learn a shared feature extractor on both source and target domains, i.e., E t = E s = E, which only works well when the domain shift is negligible [4], [13], [27]. 2) to learn separate feature extractors E s (source) and E t (target) on different domains, which has shown to successfully outperform the previous kind of domain adaptation methods [49] on image classification tasks. The general loss function of the second kind can be written as follows with two parts: where the first term is to minimize the classification error on source data with labels by the cross-entropy loss. When E t = E s = E, it will learn a shared feature extractor for both source and target domains. The second term aims to minimize the feature distribution difference between source and target data through different metrics such as KL-divergence [49], MMD distance [13], and Wasserstein-distance [2], [52]. The above loss function Eq. 2 has been successfully applied to plenty of UDA problems when both source and target data are accessible. In practice, the source feature extractor E s is always used to initialize the target feature extractor E t .

B. Source Data Protected Domain Adaptation
Instead of sharing the source data directly to the target client for adapting, we send the source feature's distribution to the target client and then the target client resamples features from source feature distributions to adapt the target feature extractor where p s (z) represents the high-level source feature distribution learned by BERT on source data (we use GMM to model p s (z), the parameters of p s are denoted as θ ps ). θ ps = [µ s k , Σ s k ∈ R ds×Ks , π s k ∈ R Ks×1 ] are the source distribution's means, co-variances, and the Gaussian mixture parameters respectively. Z s = {z s i } Ns i=1 is the set of resampled source features with N s samples, z i ∈ R ds×1 . d s is the source feature dimension, K s is the number of classes in the source domain. Source model E s , C s are pretrained on source data and fixed. Then we only optimize the target feature extractor E t during adapting.

C. DP-Pretrained Source Protected DA
Although sharing only the source feature's GMM to the target client does not reveal exact data representations, an adversary could still reveal the membership of a sample by comparing the changes from source feature distributions through the membership inference attack as shown in Figure  1. To further protect the source data's privacy, we opt for the differential-privacy training process on pretraining the source feature extractor. Denote the DP-pretrained source feature extractor as E dp s and the source feature's GMM as p s (z) dp , the objective function of adapting the target feature extractor then becomes: L d (θ Et , θ ps ) = L cls (C s (E dp s (X s )), Y s ) + L dz s i ∼ps(z) dp (Z s , E t (X t )).
Since our goal is to protect the source data's privacy while maintaining the pretrained model's utility, we adopt the DP-SGD method [1] and Rényi Differential Privacy (RDP) [35], [36] for the differential privacy-protected pretraining process. The DP-SGD method tracks the privacy loss by moments accountant (MA) to obtain a tighter estimate of the overall privacy loss compared to the strong composition theorem [12].  Randomly sample batch size samples B t .

5:
Compute per example gradient with clip by max norm and add noise: Accumulate gradients without updating: g t ← l g l (x s i ). Compute privacy spent on each iteration by the MA method. 11: end while 12: return θ Es,T and the overall privacy loss ( , δ).
Algorithm 1 outlines our proposed pretraining method to protect the source data privacy as well as provide an accurate accountant of privacy loss. We opt for the DP-SGD algorithm [1] to pretrain the source feature extractor with parameters θ Es on the clinical text classification task by the cross-entropy loss function L(θ Es ) to produce a source model that satisfies ( , δ)-differential privacy. At each step of the backpropagation, we derive the gradient ∇ θ Es L(θ Es , x s i ) for each sample in a randomly sampled mini-batch, clip the l 2 norm of each gradient, add noise to each gradient and compute the average, accumulate a few steps without updating the parameters θ Es , and finally update the parameters θ Es towards reducing the cross-entropy loss L(θ Es ). After each step, we also need to compute the privacy loss based on the moments accountant (MA) method [1] to obtain the final accumulated privacy loss in terms of ( , δ).
According to the DP-SGD's theorem 1 [1], the perexample gradient g l (x s i ) is added by the Gaussian noise of N (0, σ 2 C 2 I). There exists constants c 1 and c 2 so that given the sampling probability q = L/N and the number of steps T , for any < c 1 q 2 T , our proposed algorithm 1 is also ( , δ)-DP for any δ > 0 if we choose σ ≥ c 2 q T log(1/δ) .
From the above theorem proved in [1], we can achieve ( , δ)-DP in protecting the source data's privacy when using the MA to track the privacy loss. From the post-processing property of DP-pretraining [11], [33], the following GMM resampling step will maintain the ( , δ)-DP property.

E. Smoothing the Source and Target BERT Outputs by Knowledge Distillation
The issue in Eq. 3 is that the learned feature extractor E t may overfit the source data and perform poorly on the target data. Yu et al. [43] has shown that adopting the KD loss into the domain adaptation framework with separate feature extractors can mitigate this overfitting issue of the target encoder, and their experiments further confirm this point. However, their original design smoothed the outputs of E s , E t through the source data, where the source data is unavailable in our setting. Therefore, we modify their method by replacing the source data with the target data to adopt the KD loss on the target encoder. Note that this is different from the original version of KD [19] who learns a smaller model using the soft labels inferred from a teacher model. We only utilize the KD loss function to smooth outputs from the target encoder. Thus, our objective function of KD loss becomes this: where t is the temperature hyperparameter measuring the smoothness of the output labels using different feature extractor E s , E t , E represents taking the average of target samples. The temperature t is chosen by the experiments from [43] that t = 20 can balance between maintaining the class feature and smoothing the distributions of class outputs. Our experiment results have validated that this swap has the same effect as using the source data for KD loss.
Since the benefit of adopting the KD loss could enhance the target encoder's generalization, we also modify two other domain adaptation methods, DANN and CDAN, by integrating the KD loss into their adapting process. From our experiments, the KD loss can significantly improve the DANN method's performance on text data. Details can be found in the next section.

F. Information Maximization on Target Data
In the UDA setting, the target data are unlabeled, and therefore we cannot apply the supervised classification signal to train the target feature extractor. To overcome this difficulty, we also adopt the information maximization (IM) loss as Eq. 6 L IM = L ent +L div on the target features from [15] to make the target intra-class features converge and inter-class features diverge.
where L ent means to increase the polarity inside a class, i.e., to make intra-class features converge, and K means different classes. The second term L div aims to increase the inter-class features diverge, where g k denotes the logits of target features.

G. Proposed Framework
Our proposed framework is shown in Figure 2 and the overall objective function combines the pretrain source feature extractor with DP to protect source privacy, KD loss to mitigate BERT's overfitting issue, and IM loss to improve the UDA performance: We adopt two backbone state-of-art UDA methods in our framework: (1) Domain Adversarial Neural Network (DANN). DANN imposes adversarial learning to align the source and target features. It trains a discriminator D to distinguish the source and target features, while the feature extractor aims to generate domain-invariant features to fool the discriminator [13]. DANN uses a shared feature extractor E on two domains with the following objective function: Source-Privacy protected DA with DANN. DANN uses both the source and target data to learn the shared feature extractor E. We sample source features from the DP source feature distribution model (p dp s (z)) and then the encoder E for DANN is optimized by (2) Conditional Adversarial Domain Adaptation (CDAN). CDAN adopts conditional adversarial learning to align the joint distribution of features and prediction of classifiers [30], and it also has a shared feature extractor E on two domains. The CDAN loss function is: Source-Privacy Protected DA with CDAN. For the privacy version of CDAN, we also use separate encoders for source and target domains, and sample source features from the DP source feature distribution p dp s (z). Then the objective function becomes this:

IV. EXPERIMENTS
In this section, we implement our proposed source-privacy preserving UDA method and perform various experiments and visualization to evaluate the effectiveness of our proposed method on two datasets of the medical report label extraction task. We first verify the main task performance of UDA through precision, recall, and F1-score. Then we compare the performance under different privacy losses.

A. Datasets and Hyperparameters
For the pulmonary disease label extraction problem, we select two related datasets: MIMIC-CXR Database (MIMIC) [22] and Open-i collection from Indiana University taken from openi.nlm.nih.gov (Open-i) [6]. We manually select the common disease labels to mitigate the label difference between two datasets and filter the corresponding medical report findings. The common labels we select are cardiomegaly, edema, pneumothorax, no finding, consolidation, pneumonia, fracture, pleural effusion, and atelectasis. Therefore, the main task is to infer correct findings from the clinical report. We select the findings section in the medical report as the input text in both datasets. Note that each medical report may have multiple positive findings. Therefore this task can be viewed as a multilabel classification problem. For the MIMIC dataset, there are three types of labels for negative (0), positive (1), and uncertain (-1). We merge the uncertain labels into the negative class for simplicity purposes. For the Open-i dataset, we extract the labels from the Mesh section. Since the Open-i dataset (1858 samples) is much smaller than the MIMIC dataset (115708 samples) after our filtration, we only apply the adaptation in one direction, i.e., from MIMIC to Open-i.
We opt for precision, recall, and F1-score as the main task performance metrics to evaluate the performance of UDA on the pulmonary disease label extraction problem, as it can be abstracted as a multi-label classification task.
To apply adversarial learning and distinguish the features from source and target domains, the discriminator is implemented as a 3-layer fully connected network. The AdamW optimizer [32] is adopted to update all models (source and target feature extractors, classifier, and discriminator). The learning rate for pretraining the source feature extractor and classifier is 5 × 10 −5 . For adapting the target feature extractor and discriminator, the learning rate is set to 1 × 10 −5 .

B. Pulmonary Disease Label Extraction
Since samples from both datasets (MIMIC and Open-i) may have multiple labels, we cannot simply use the accuracy as the evaluation metric. Instead, we report the results in precision, recall, and F1-score. We mainly compare the performance of UDA on three settings: with source (adapting with source data), without-source (adapting with GMM resampled features), and without-source DP (adapting with DP-GMM resampled features) in this task. We report the evaluation metrics on each common class between two datasets. Table  I summaries the results of pulmonary disease multi-label classification. Note that many results appear to have the same value is due to the rounding issue. We only evaluate the adaptation in one direction: from MIMIC to Open-i because the MIMIC dataset has much more samples than the Openi dataset (115708 compared to 1858), and there are nine classes in total. For each adaptation method, we compare four scenarios: "no adapt" means we directly evaluate the source model on the target data as the baseline; "with source" denotes adaptation with source data; "without source" is our proposed method except the DP pretraining; and "without source+DP" is the complete version of our proposed method. From Table  I, we can observe that our proposed method performs better than the "no adapt" and "with source" version in most cases, especially in the "cardiomegaly" and "no finding" classes. In other classes, our proposed method evens with the rest of the classes in that the baseline results are already saturated and have little space for improvements, e.g., edema, pneumothorax, consolidation, pneumonia, and fracture. In our implementation, the "without-source DP" version's privacy budget is set to 1. The "without-source DP" version is slightly inferior to the without-source version in some classes. This performance is as expected because the DP version introduces some noise in pretraining and leads the GMM to deviate from the source feature distribution.
Interestingly, our proposed method constantly performs better than the with-source version in this pulmonary disease classification task. We hypothesis that the MIMIC dataset is very noisy with many inaccurate labels as pointed out by [18]. Hence, the source feature distributions might be highly irregular and have many outliers. The GMM clustering of source features in our proposed method helps to regularize the source feature distributions. Thus, the resampled features are clustering better than the source features and this phenomenon can assist the target data adaptation.
To verify our hypothesis, we visualize the target features in 3 scenarios: 1) the target encoder before adaptation; 2) the target encoder after CDAN adaptation with source data; 3) the target encoder after CDAN+DP adaptation without source data (out proposed method); as shown in Figure 3, 4, and 5 respectively. We can observe that features within a class converge better than before, and features among different classes become more divergent. This phenomenon confirms the effectiveness of GMM in our proposed method.

C. Performance under Different Privacy Loss
In the DP-SGD method, the privacy budget, denoted as , is to quantify how much we tolerate the output of a model on one database d to be different from the output of the same model on a second database d that only has one different training sample. The smaller the is, the stronger the privacy will be [40]. We evaluate the main task performance of pulmonary disease classification under different privacy budgets. The privacy loss is measured by the moments accountant (MA) from [1]. In our implementation, we adopt the Opacus 1 library to implement the DP-pretrain and privacy loss accountant. We only perturb the last transformer layer in the BERT encoder to perform the DP-pretrain due to the memory limit on our server. We compare the F1-score of each disease with different privacy loss as the F1-score combines the advantage of precision and recall. The non-DP version denotes that we only resample the GMM model of source feature distribution 1 https://opacus.ai/ to adapt the target model, which is equivalent to the "without source" version in Table I. We adopt four different values (2.5, 5, 7.5, 15) of to evaluate adaptation performance. Here, we only report the F1 score since it combines the benefits of precision and recall and better represents the overall performance. In our implementation, the privacy loss is accumulated from the pretraining process through MA. The results are summarized in Table II and Figure 6. From Table II and Figure 6, we can observe that the DP-pretraining does not deteriorate the performance too much, with only some small changes in each class. Applying the DP-pretraining, for classes cardiomegaly, no finding and pleural effusion, the DP-pretrain versions become worse. Other classes are almost the same. This is consistent with our expectations.

D. Discussion of Cluster Number (K) in GMM
Another factor that can influence the performance of our proposed model is the number of clusters K in the GMM. From our experiments based on the without source version of DANN, the factor K does impact the adaptation performance (class no finding) as shown in Figure 7. The highest F1-score 0.8546 appears at K = 6, and the performance deteriorates with larger K values indicating that the number of GMM clusters is not the larger, the better. The best results appear around 5 or 6.    We also adopt the same K value in other clinical text-dataset adaptations.

V. CONCLUSION
In this paper, we mainly study how to preserve the source data privacy in the unsupervised domain adaptation on clinical   text data. We discuss how the privacy leaks when sharing the source data to target clients for adaptation and when confined to only sharing the source feature distribution. To defend against this source privacy leakage, we propose a source privacy-preserving unsupervised domain adaptation framework that adopts differential privacy in pretraining the source model to reduce the risk of source-data privacy leakage. Our proposed method is compatible with many existing unsupervised domain adaptation methods. We implement our framework on two clinical text datasets to perform the pul-monary disease multi-label classification task. Our experiment results have verified that our proposed method can protect the source data privacy with little sacrifice on the utility.