{'fraction_non_alphanumeric': 0.03115841951022333, 'fraction_numerical': 0.010071748576818135, 'mean_word_length': 5.295165394402035, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cross-Domain Self-Supervised Deep Learning for Robust Alzheimer's Disease Progression Modeling

Developing successful artificial intelligence systems in practice depends both on robust deep learning models as well as large high quality data. Acquiring and labeling data can become prohibitively expensive and time-consuming in many real-world applications such as clinical disease models. Self-supervised learning has demonstrated great potential in increasing model accuracy and robustness in small data regimes. In addition, many clinical imaging and disease modeling applications rely heavily on regression of continuous quantities. However, the applicability of self-supervised learning for these medical-imaging regression tasks has not been extensively studied. In this study, we develop a cross-domain self-supervised learning approach for disease prognostic modeling as a regression problem using 3D images as input. We demonstrate that self-supervised pre-training can improve the prediction of Alzheimer's Disease progression from brain MRI. We also show that pre-training on extended (but not labeled) brain MRI data outperforms pre-training on natural images. We further observe that the highest performance is achieved when both natural images and extended brain-MRI data are used for pre-training.


Introduction
Developing reliable and robust artificial intelligence systems requires advanced and efficient deep learning techniques, as well as the creation and annotation of large volumes of data for training. However, the construction of a labeled dataset is often time-consuming and expensive, such as in the medical imaging domain, due to the complexity of annotation tasks and the high expertise required for the manual interpretation. To alleviate the lack of annotations in medical imaging, transfer learning from natural images is becoming popular [35,37,38,62]. Although numerous experimental studies indicate the effectiveness of fine-tuning from either supervised or self-supervised ImageNet models [1,3,21,26,28,67], it does not always improve the performance due to domain mismatch problem [48].
In the meantime, self-supervised learning has demonstrated great success in many downstream applications in computer vision, where the labeling process is quite expensive and time-consuming [4,15,20,27,44,46,56,63,65]. Medical research and healthcare are especially well-poised to benefit from self-supervised learning approaches, given the prevalence of unprecedented amounts of medical images generated by hospital and non-hospital settings. Despite demand, the use of self-supervised approaches in the medical image domain has received limited attention. Only few studies have investigated the impact of self-supervised learning in the medical image analysis domain for limited applications including classification [2,25,32,34,53,68] and segmentation [5,9,50,54].
In this study, we focus on developing a self-supervised deep learning model for predicting the progression of Alzheimer's disease (AD) by using high dimensional magnetic resonance imaging (MRI). AD is a slowly progressing disease caused by the degeneration of brain cells, with patients showing clinical symptoms years after the onset of the disease. Therefore, accurate prognosis and treatment of AD in its early stage is critical to prevent non-reversible and fatal brain damage.
Many of the existing methods have focused on the task of classifying patients into coarse categories: Cognitive Normal (CN), Mild Cognitive Impairment (MCI), or Alzheimer's Disease (AD), or to predict conversion from one class to another (e.g. MCI to AD) [45,49,59]. However, applications in clinical trials require a more fine-grained measurement scale, because clinical trial populations are typically narrowly defined (eg. only MCI). Another approach is to instead predict the outcome of cognitive and functional tests such as Clinical Dementia Rating Scale Sum of Boxes (CDRSB), which are measured by continuous numerical values. Prognostic prediction, modeled as a regression task rather than a classification, helps provide more granular estimates of disease progression. Recently, a few deep learning based approaches, including the recurrent neural network (RNN) and convolutional neural networks (CNN) have been proposed for predicting disease progression of AD patients based on MRIs. [43] adapted Mini-malRNN to integrate longitudinal clinical information and cross-sectional tabular imaging features for regressing endpoints. [17] utilized an ensemble model based on stacked CNN and a bidirectional long short-term memory (BiL-STM) to predict the endpoints on the fusion of time series clinical features and derived imaging features. Recently, several methods have started to employ CNN based models to extract features from raw medical imaging. [55] applied CNN with a multi-task interaction layers composed of feature decoupling modules and feature interaction module to predict the disease progression.
Despite demand, little progress has been made because of the difficult design requirements, lack of large-scale, homogeneous datasets that contain early stage AD patients, and noisy endpoints that are potentially hard to predict. Much of the prior work has focused on using image-derived features (using FreeSurfer, ANTS) in order to overcome the complexity and high variability in raw MRI images, and small dataset sizes. Most of the current prognosis models are trained a single dataset(i.e. cohort), which cannot be easily generalized to other cohorts. Moreover, all of them only utilize limited number of annotated images, which may suffer from problems arising from domain shift and heterogeneity. Moreover, a large number of parameters in CNN are involved in training on a small dataset, which could lead to over-fitting [16].
We establish a cross-domain self-supervised transfer learning approach for medical image analysis that leverages self-supervised learning on both unlabeled large-scale natural images and a domain-specific medical imaging from 11 different internal and external studies to learn transferable and generalizable representations for medical images. Such representations can be further fine-tuned and deployed for downstream tasks such as disease progress prediction using limited labeled data from the clinical setting. Our transfer learning strategy benefits from both large-scale self-supervised representations and domain-specific selfsupervised representations in a unified deep learning framework. We develop a detailed and extensive framework to evaluate the performance of our model fine-tuned from different supervised and self-supervised models pre-trained on either natural images or medical images, or both. Our extensive experimental results reveal that (1) self-supervised pretraining on natural images followed by intermediate selfsupervised learning on unlabeled domains-specific medical data outperforms all other transfer learning techniques, which is very promising because replacing supervised approaches with self-supervised learning can significantly reduce the need for data annotation (2) self-supervised models pre-trained on medical images outperform same models pre-trained on natural images indicating that self-supervised learning on medical images yield discriminative feature representations for regression task.

Related works
The recent development and success of self-supervised learning techniques, including contrastive learning [10-12, 22, 23, 39, 61], mutual information reduction [57], clustering [8,31], and redundancy-reduction methods [6,64] in computer vision indicate their effectiveness in improving the performance of AI systems. These methods train models on different pretext tasks to enable the network to learn high-quality representations without label information. SimCLR [12] maximizes agreement between representations of different augmentations of the same image by using a contrastive loss in the latent space. Barlow Twins [64] measure the cross-correlation matrix between the embedding of two identical networks and its goal is to make this cross-correlation close to the identity matrix. SwAV [8] simultaneously clusters the images while enforcing consistency between cluster assignments produced for differently augmented views of the same image, instead of comparing features directly as in contrastive learning.
Subsequently, self-supervised learning has been employed for medical imaging applications including classification and segmentation to learn visual representations of medical images by incorporating unlabeled medical images. While some approaches have designed domainspecific pretext tasks [5,54,68,69]], others have adjusted well-known self-supervised learning methods to medical data [25,30,53,66]. Very recently [2] has applied SimCLR on a combination of unlabeled ImageNet dataset and task specific medical images for medical image classification; their experiments and improved performance suggest that pre-training on ImageNet is complementary to pre-training on unlabeled medical images.
Although aforementioned approaches demonstrate improvement of the performance on challenging medical datasets, all of them are limited to classification and segmentation tasks and their benefits and potential effects for the prognosis prediction tasks, as regression tasks, have not been studied. Formulating progress prediction as a regression rather than a traditional classification problem leads to a more fine-grained measurement scale which is crucial for real-world applications. Therefore, the development of selfsupervised networks is in great demand for efficient datautilization in medical imaging for disease prognosis. To the best of our knowledge, this is the first study of developing a self-supervised deep convolution neural network on medical data images from various cross-domain datasets to predict a granular understanding of disease progression.

Task and Data
In Alzheimer's Disease clinical trials, the most important cognitive test that is performed to assess current patient function and the likelihood of AD progression is Clinical Dementia Rating Scale Sum of Boxes (CDRSB). Our goal is to predict the progression of Alzheimer's Disease 12 months after the first patient visit. Specifically, our model uses 2D slice stacks of an magnetic resonance imaging (MRI) volume (which are collected at the first visit) as an input and predicts the CDRSB value, which is computed at month 12 (i.e., after one year). All individuals included in our analysis are around 8k from seven external datasets, including ADNI [47], BIOFINDER [36], FACEHBI [41], AIBL [18], HABs [13], BIOCARD [40], and WRAP [29], and four internal studies including A, B, C, and D (dataset names are anonymized in order to allow for blind review). We only include the axial view of MR images for this paper analysis. MR volumes are standardized using standard preprocessing steps (in supplementary materials) and are cropped or padded to (224 × 224) to make them compatible with the ResNet-50 input size. Furthermore, using the predefined SynthSeg model [7], we generate segmented images from raw images in order to evaluate the information of different brain parts. The segmentation results only contain the brain region and masked out other details such as skull, etc. The results of segmented images are provided in the supplementary materials.
We provide two datasets of medical images referred to as center-slice and 5-slice datasets. In the 5-slice dataset, for each subject, we prepare a 5-slice stack from its corresponding 3D MRI volume by taking the center slices of the brain sub-volumes and four adjacent slices at intervals equal to 5 (two to the right, two to the left). The centerslice imaging dataset only contains the center slice, while the 5-slice dataset includes the 5-slice stack. The number of unlabeled images in the center-slice dataset is 43, 740, and the number of unlabeled images in the 5-slice dataset equals 218, 700. Both of these datasets are used to train self-supervised models. During the self-supervised learning process, 92% and 8% of the data are assigned for the training and validation, respectively. Following the common practice in self-supervised learning, based on the minimum self-supervised validation loss, we select the best model and transfer its backbone weights to the supervised model. For the purpose of creating labeled datasets, we use MR images of participant's first visits. The participants are selected from five datasets, including ADNI, A, B, C, and D. All datasets were filtered to include patients who are positive for amyloid pathology, have a MMSE larger than 20 and are diagnosed with early stages of AD i.e. prodromal or mild at baseline. Amyloid positivity is detected by ei-ther cerebrospinal fluid (CSF) or amyloid positron emission tomography (PET). For supervised training, we only consider the 2D center slice. The number of labeled data in this dataset is 935 for the train and validation sets, as well as 360 for the independent test set.
3.2. Self-supervised learning platform for progress prediction problem Following are the experiments that comprise our approach. 1) We assign our model with self-supervised pretrained representations on unlabeled natural images in Im-ageNet. 2) We initialize the weights of model randomly then do self-supervised pre-training on medical images (Figure 1(a)). 3) We initialize weights with supervised pre-trained weights on labeled ImageNet and then selfsupervised pre-training on medical images (Figure 1(b)). 4) We initialize the weights with self-supervised pre-trained weights on unlabeled ImageNet and then self-supervised pre-training on all layers on unlabeled medical images (Figure 1(c)). A number of data augmentations have been performed for all the self-supervised learning models discussed in this paper, including random gray scaling, random cropping with resizing, random Gaussian blurring, random rotation, and random flipping. We use a batch size of 50 and epoch of 1000 and implement an early-stop mechanism based on the validation data in order to prevent over-fitting. After we have pre-trained the weights of self-supervised frameworks, we use their CNN backbone (in our experiments, this is ResNet-50) to fine-tune the regression task ( Figure 1(d)) and then calculate the results on an independent set of data (Figure 1(e)). For supervised learning models, we perform data augmentation including random vertical and horizontal rotation, as well as random flipping. For supervised learning, a batch size of 25 and an epoch of 500 are used.
Self-supervised pre-training on natural images There are a wide variety of self-supervised learning (SSL) models that are pre-trained on large-scale natural image datasets, such as ImageNet, that are commonly used for transfer learning. Such SSL pre-training techniques surpass supervised ImageNet models in several computer vision tasks. To exploit these benefits, in this experiment, we initialize the backbone encoder with weights from SSL models trained on ImageNet. While several types of self-supervised learning methods exist, in this work, we focus on three self-supervised learning methods, one contrastive approach including SimCLR, one redundancy reduction approach including BarLow Twins (BLT), and one clustering-based contrastive learning approach including SwAV which have yielded impressive results on natural image benchmarks. There are several other self-supervised learning strategies, but their performance on ImageNet yields comparable results.
Self-supervised pre-training on medical images Although representations learned from natural images perform well on medical imaging tasks [42], they may not be optimal for the medical imaging domain due to the large distribution shift from natural images. Generally, medical images are monochromatic and have similar anatomical structures. This discrepancy could be minimized by further pre-training on medical data, either supervised or unsupervised. Due to the time and cost associated with the annotation of medical data, self-supervised learning can be considered a more practical and realistic alternative. Similar to self-supervised pre-training on natural images, we employ SimCLR, Barlow Twins, and SwAV to learn distinctive representations of unlabeled medical images effectively. These methods have all demonstrated good performance in the classification and segmentation of medical images.
Fine Tuning During fine-tuning, we train the model using the weights of the pre-trained network as initialization for the downstream supervised task. For progress prediction, the standard ResNet-50 backbone [24] is followed by a linear layer head, where the backbone is initialized either randomly or with the pre-trained models. Training ResNet-50 involves freezing backbone parameters and only training the head. The mean square error (MSE) is used as a criterion for calculating model loss.

Results
We propose the first benchmarking analysis to evaluate the efficacy of different pre-training models for disease progression prediction as a regression problem. (1) We examine the transferability of both supervised and selfsupervised ImageNet models pre-trained on natural images for the purpose of assessing their performance on medical tasks; (2) we investigate the efficacy of pre-training models on domain specific medical images for transfer learning performance on medical tasks; and (3) we evaluate cross-domain self-supervised pre-training on both natural and medical datasets to explore the efficacy of the proposed platform for bridging the domain gap between natural and medical images. In order to evaluate the performance of our model, we used the coefficient of determination, which is indicated as R 2 1.
where (e 2 i ) shows that sum of squared residuals and (y i −ȳ) 2 shows total sum squared. R 2 is commonly used in clinical studies to assess how well a model explains and predicts future outcomes [19]. However, most of the current literature report pearson correlation or mini-mum squared error for evaluating their models which alone is not an indicative of "goodness of fit".

Self-supervised models on natural images provide more generalizable representations
Experimental setup. This experiment evaluates the transferability of standard supervised ImageNet models with three popular SSL methods using officially released models including SimCLR, Barlow Twins, and SwAV. All SSL models are pre-trained on the ImageNet dataset and employ a ResNet-50 backbone. To establish a baseline, we train the target model using random initialization (i.e., without pre-training).
Results. Table 1a displays the results, from which we draw the following indications: (1) transfer learning from the supervised ImageNet model does not improve over random initialization. We attribute this performance to the significant domain shift between the pre-training and regression target task. In particular, supervised ImageNet models tend to capture domain-specific semantic features, which can be inefficient if the pre-training and target data distributions differ. This finding is consistent with recent studies on other medical tasks that demonstrate that transfer learning from supervised ImageNet pre-training does not always correlate with performance on either classification or segmentation of medical images [14,28,58]. (2) Transfer learning from self-supervised ImageNet models provides superior performance compared with both random initialization and transfer learning from the supervised ImageNet model. The best self-supervised model (i.e., Barlow Twins) achieves performance improvement of 7% and 8% over random initialization and the supervised ImageNet model, respectively. As opposed to supervised pre-trained models, self-supervised pre-trained models, encode semantic features that are not biased toward any particular task, which can improve generalizbility cross domains.

Self-supervised pre-training on medical images
outperforms ImageNet models Experimental setup. We investigate the impact of pre-training medical datasets on self-supervised learning. We train three SSL methods, SimCLR, Barlow Twins, and SwAV on 11 cross-domain unlabeled medical imaging datasets (referred to as in-domain dataset). All of the SSL models are initialized randomly and later are fine tuned by using our labeled dataset.
Results. Table 1b shows the performance accuracy measured by the R 2 when SSL methods are pre-trained using the center slice dataset. We observe that (1) SimCLR pretraining on in-domain dataset achieves the highest performance and provides 7% and 4% performance boost compared to SwAV and Barlow Twins respectively. This observation may offer the superiority of contrastive learning for   identifying significant MRI features for predicting progression of Alzheimer's disease in terms of CDRSB (2) Sim-CLR pre-training on in-domain dataset yields higher performance than either supervised or self-supervised pre-training on only the ImageNet dataset (as seen in Table 1a). Intuitively, pre-training on in-domain dataset encodes domainspecific features that reflect unique characteristics of medical images.
The Barlow Twins pre-training on in-domain data, however, does not improve the performance over the Barlow Twins pre-training on ImageNet. This performance suggests that features learned by Barlow Twins pre-trained on ImageNet are generalizable enough to medical images; And therefore pre-training on a limited number of unlabeled images compared with the ImageNet dataset (40k vs. 1.3 M) may offer limited performance gain for Barlow Twins method, which is a redundancy reductions based model.

Impact of the size of the unlabeled in-domain dataset
Experimental setup. We conduct further experiments to evaluate the advantages of increasing the number of unlabeled images for self-supervised pre-training. For each one of SSL methods, we train two separate models by pretraining on either center-slice or 5-slice datasets.
Results. As observed in Table 2, all three SSL models pre-training on 5-slice imaging dataset results in an improvement over pre-training on center-slice dataset, indicating that the SSL models can benefit from larger pretraining in-domain unlabeled datasets. Specifically, Barlow Twins pre-training on the 5-slice dataset provides 1% performance improvement over pre-training on center-slice dataset which may confirm our previous assumption that Barlow Twins, in contrast to SimCLR, needs more unlabeled in-domain images to constrain the ImageNet-based features for medical tasks.  Table 2. A comparison of the performance of self-supervised pretraining models in the fine-tuning step with datasets of different sizes.

Cross-domain self-supervised learning bridges the domain gap between natural and medical images
Experimental setup. We investigate the impact of selfsupervised pre-training on both natural images and domainspecific medical images. To do so, we pre-train Sim-CLR on our 5-slice in-domain dataset with two different initialization schemes including (1) Supervised ImageNet (referred to as Labeled ImageNet→In-domain), (2) Barlow Twins on the ImageNet dataset (referred to as Unlabeled ImageNet→In-domain). We select SimCLR and Barlow Twins because as indicated in Tables 1a and 1b, they provide the best results on natural and medical images.
We fine-tune all pre-trained models for the progress prediction task using the labeled 1k images in the training data. Figure 1 shows the cross-domain self-supervised learning employed for the regression problem. In order to show the relation between predicted values and target value we also calculate another metric Pearson correlation coefficient (r) 2 metrics for our best performance model and randomly initialized model.
Results. The results are shown in Table 3a. We observe that the best performance is achieved when both unlabeled ImageNet and in-domain datasets are utilized for pre-training. Specifically, Unlabeled ImageNet→Indomain pre-training surpasses both in-domain and Ima-geNet pre-trained best performing models. It achieve 14%, 7%, and 2% performance boosts compared to Random initialization, pre-training only on ImageNet, and pretraining only on in-domain dataset, respectively. These results, in line with previous studies [3,28], imply that pretraining on ImageNet is complementary to pre-training on in-domain datasets, resulting in more powerful representations for medical applications. We also note that Labeled ImageNet→In-domain pre-training is inferior to Unlabeled ImageNet→In-domain. These observations restate the efficacy of self-supervised models in delivering more generic representations that can be used for target tasks with limited data, resulting in reduced annotation costs. Supplementary materials also includes the analysis of these crossdomain experiments but for center-slice dataset. Figure 2 illustrate a scatter plot comparing the target CDRSB value with the predicted CDRSB value using a supervised model. As shown in these plots, the red line represents the perfectly fitted model for target values, while the blue line represents the actual fitted model for predicted values. In Figure 2a we use randomly initialized pre-training model and the model shows the results of r of 0.29. Figure 2b illustrates results of the pre-training method Barlow-twin on Natural images, which demonstrates a r value of 0.39. In Figure 2c, we use our best-performing pre-training model, Unlabeled ImageNet→In-domain, which shows the results of r of 0.46. This higher number compare to randomly initialized model is also illustrated by the decrease incline of blue line towards the red line.

Performance on test set
Experimental setup. The robustness of our best model, i.e. Barlow Twins→SimCLR on the 5-slice dataset is evaluated and compared with best performing models initialized by either ImageNet or in-domain dataset. Our test set includes 360 unseen patient images.
Results. As presented in Table 3a, the best performance is achieved when both unlabeled ImageNet and indomain dataset are utilized for pre-training. In particular Unlabeled ImageNet→In-domain pre-training yields a significant improvement of 10% and 6% over in-domain and ImageNet pre-trained models, respectively. These results confirm our previous observation that supervised pre-     ences on short-term prognosis. We used the GradCAM [52] method to extract and evaluate the varying importance of each part of brain MRIs using a gradient of final score. In this method, regions of an image are marked with different colors ranging from red to blue. Generally, the closer the color is to the red color, the greater the contribution from that area to the final result based on the current input data (i.e., MRI slice). Results. Figure 3 illustrates several examples of saliency map. These saliency maps reflect the importance of different regions from the MRIs in pixel level. A general observation we can make through these saliency maps is that our best model i.e. Barlow Twins→SimCLR highlights the subcortiacl areas of the brain. Our observation agrees with previous findings, that the earliest stages of AD show accumulation of abnormal tau in entorhinal cortex and subcortical brain regions [33,51]. Therefore, it is understandable that our model i.e. Barlow Twins→SimCLR gives higher attention to those regions of the brain in a population with patients with prodromal and mild Alzheimer's disease at baseline. The model initialized randomly highlights random features all around the MRIs and even in background areas, while in model initialized from unsupervised Ima-geNet, target regions of brain contribute more than other areas (i.e. eye, background) for Alzheimer's disease prognosis.

Discussion
We present a cross-domain self-supervised learning framework for predicting the progression of Alzheimer's disease, which is formulated as a regression problem. This formulation helps to provide more granular estimations of disease progression but can be very noisy and subjective across studies -making it challenging to establish treatment efficacy in small patient populations. The goal of this problem is to predict CDRSB score as an endpoint for defining the stage of Alzheimer's disease. This framework is enforced to use pre-training on natural images followed by pre-training on domain-specific medical images to alleviate domain shift and improve generalization of pre-trained features. Our extensive experiments demonstrate the effectiveness of our approach to combat the lack of large-scale annotated data for training deep models for progression prediction. Our results show an improvement in prediction accuracy and more robust prediction performance for patients with Alzheimer's disease. The proposed framework has the potential to identify patients at higher risk of progressing to AD and help develop better therapies at lower cost to society. Because the number of annotated images is relatively small, our current platform only explores the advantages of 2D MRIs for prognostic prediction which decreases parameters and leads to model simplicity. 3D deep neural networks require many more learnable parameters to solve such a complex problem and do not always yield proper outcomes [60]. However, as future works, we will utilize 3D MRIs to compare their performance versus 2D slices. we also will expand our dataset to include more than 5-slice from each image, as well as other views such as coronals and sagittals, allowing us to introduce more in-domain information to our pre-training model.