{'fraction_non_alphanumeric': 0.03070665988815455, 'fraction_numerical': 0.013777325876970005, 'mean_word_length': 4.845765230312035, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TINC: Temporally Informed Non-Contrastive Learning for Disease Progression Modeling in Retinal OCT Volumes

Recent contrastive learning methods achieved state-of-the-art in low label regimes. However, the training requires large batch sizes and heavy augmentations to create multiple views of an image. With non-contrastive methods, the negatives are implicitly incorporated in the loss, allowing different images and modalities as pairs. Although the meta-information (i.e., age, sex) in medical imaging is abundant, the annotations are noisy and prone to class imbalance. In this work, we exploited already existing temporal information (different visits from a patient) in a longitudinal optical coherence tomography (OCT) dataset using temporally informed non-contrastive loss (TINC) without increasing complexity and need for negative pairs. Moreover, our novel pair-forming scheme can avoid heavy augmentations and implicitly incorporates the temporal information in the pairs. Finally, these representations learned from the pretraining are more successful in predicting disease progression where the temporal information is crucial for the downstream task. More specifically, our model outperforms existing models in predicting the risk of conversion within a time frame from intermediate age-related macular degeneration (AMD) to the late wet-AMD stage.


Introduction
The scarcity of manually annotated labels is a major limitation for the classification tasks in medical image analysis. Self-supervised learning (SSL) showed a great promise in exploiting the availability of unlabeled medical data by outperforming models trained from random weights or pretrained with non-medical images in difficult supervised settings [1]. Traditional SSL methods rely on pretext tasks that are believed to be semantically relevant to the downstream task, such as jigsaw-puzzle solving, and rotation angle prediction.
In recent years, contrastive learning (CL) methods surpassed the pretextbased SSL in unsupervised representation learning. They learn similar representations of two heavily augmented views (positive pairs) of a sample while pushing away the others in the representation space as negatives. The goal is to find representations that are semantically meaningful and robust to image perturbations. Following this, CL methods have also been adapted for medical images. Li et al. [11] addressed the class imbalance problem by sample re-weighting during contrastive training and devised an augmentation scheme for 3D volumes. Chen et al. [7] proposed a sampling strategy by feeding two frames as pairs from an ultrasound video to encode the temporal information for the CL. They found that sampling and augmentation strategies were crucial for the downstream task. Azizi et al. [1] showed that if two different images of a patient included the same pathology, they formed more informative positive pairs than the heavily augmented pairs. Also, they reported that if the supervised and the unsupervised data were mixed for the CL, the downstream task's performance was increased. These methods were based on contrastive InfoNCE loss [12]. However, the success of the CL largely depends on the quality of the negative samples [8]. This introduces two challenges for medical imaging; (i) large batch sizes, and (ii) explicit negatives in a batch. Furthermore, particularly in longitudinal studies, the large batch sizes (over a thousand) are not compatible with the number of patients (a few hundred) and would create negative pairs from the same patient.
The developments in non-contrastive learning methods [6,20,3] avoid the need for explicit negatives and consequently for large batch sizes. They implicitly learn to push the negatives with stop-gradient [6,9], clustering [5], or creating discrepancy between pairs through a specific loss [3,20]. Barlow Twins' [20] loss function achieves that by making the correlation matrix of the embeddings of the two views close to identity matrix and VICReg [3] by calculating a Huber variance loss within a batch of embeddings. Especially, VICReg requires no architectural trick, large batch sizes, or normalization. Both of them use a simple Siamese Network and a multi-layer perceptron (MLP) projector on the representations for the embeddings. They also allow constructing pairs from different images, and unlike contrastive methods, it is even possible to use multiple modalities.
In this paper, we focus on non-contrastive SSL in longitudinal imaging datasets, and we propose a new similarity loss to exploit the temporal meta-information without increasing the complexity of the non-contrastive training. Also, we introduce a new pair-forming strategy by using scans from different visits of a patient as inputs to the model. In this regard, our work is one of the first to build on non-contrastive learning with continuous labels (time difference between visits).
Clinical background The optical coherence tomography (OCT) imaging is widely used in clinical practice to provide a 3D volume of the retina as a series of crosssectional slices called B-scans. Age-related macular degeneration (AMD) is the leading cause of vision loss in the elderly population [4]. It progresses from the early/intermediate stage with few visual symptoms to a late stage with severe vision loss. Conversion to late-stage could take two forms, wet and dry AMD. Wet-AMD is defined by the formation of new vessels. An intravitreal injection can improve the patients' vision, but it is most effective when applied soon upon conversion to wet-AMD. This motivated the medical imaging research communities to develop risk estimation models for conversion to wet-AMD. AMD progression prediction in OCT has been studied using statistical methods [16,18,15] based on biomarkers and genomics. Schmidt-Erfurth et al. [15] approached the conversion prediction as a survival problem and used a cox model. Initial deep learning models works showed the importance of standardized preprocessing in B-scans [14]. Recently, [19] exploited surrogate tasks like retinal biomarker segmentation in OCT volumes to improve the performance. On the other hand, [2] used imaging biomarkers and demographics to train an RNN on sequential data. As an SSL approach, Rivail et al. [13] pre-trained a Siamese network by predicting the time difference between the branches.
Contribution We improve on learning representations such that they (a) capture temporal meta-information in longitudinal imaging data, (b) do not suffer from dimensional collapse (only a small part of representation space is useful [10]). Our contribution is two-folds. First, we proposed a simple yet effective similarity loss called TINC to implicitly embed the temporal information without increasing the non-contrastive loss complexity. We hypothesized that the B-scans acquired closer in time should be close in the representation space. We chose VICReg [3] to use TINC with because its similarity loss explicitly reduces the distance between representations and allows alternatives in its place. Second, instead of aggressive augmentation for pair generation from a single B-scan, we formed pairs by picking two moderately augmented versions of two different B-scans from a patient's OCT volumes acquired at different times. Finally, TINC had increased performance in the difficult task of predicting conversion from the intermediate to wet-AMD within a clinically-relevant time interval of 6 months.

Method
We modified the VICReg's invariance (similarity) term by constraining it with the normalized absolute time difference (no temporal order among the inputs) between the input images. The original invariance term is the mean-squared error (MSE) between two unnormalized embeddings. The time difference acts as a margin on how low the invariance term can get. As the time difference between two visits increases, the similarity between a pair of the respective B-scans should decrease. Thus, the distance measure should lie within a margin, not on a point like the VICReg invariance term does. In other terms, the representations should slightly differ due to the time difference between the scans of a patient.
Given a batch of n patients with multiple visits, let visits v 1 and v 2 be the components of n pair of time points randomly sampled from each available patients' visit dates within a certain time interval. Randomly selected B-scans from the OCT volumes at times v 1 and v 2 , are augmented by random augmentations t 1 and t 2 for the two views X 1 and X 2 . First, the encoder produces n representations Y 1 and Y 2 , then the projector (also called expander) expands the representations to embeddings Z 1 and Z 2 with embedding dimension d (Fig. 1).
The loss terms of the original VICReg works as follows: S (Eq. 1) is the invariance term or the similarity loss, V (Eq. 2) is the variance term to keep a variance margin between different pairs of embeddings (prevents representation collapse), and C (Eq. 3) is the covariance term that forces each component to be as informative as possible (prevents dimensional collapse).
where in V , std is the standard deviation of the z j which is a vector of the jth embedding component values along the batch. For V , we used γ as 1. In covariance term C, Cov is the covariance matrix of an embedding vector. C and V are calculated for Z 1 and Z 2 separately. S is the MSE between two vectors. Finally, the total loss is the weighted sum of these three.
Temporally Informed Non-Contrastive Loss The temporal label is defined as the difference between visit dates v 1 and v 2 . ∆v is the absolute value of the difference scaled to 0-1 using a Min-Max scaler with given v min and v max . We use ∆v as a margin, where the distance between the two embeddings should not be greater than it. Inspired by the epsilon insensitive loss from support vector regression, the invariance term S in VICReg is replaced with our TINC loss: The TINC term forces the distance between the two representations to be within a non-zero margin, set proportionally to the time difference between visits. As ∆v gets close to 1, the margin becomes wider, and the loss does not enforce a strict similarity. On the other hand, when ∆v is close to 0, TINC loss becomes similar to MSE between the two representations. In principle, the values of the embedding components could diminish, resulting in a collapse. Then the distance would be smaller than the margin, not contributing to the overall loss. However, the variance term in VICReg counteracts it by enforcing a standard deviation between different pairs, preventing the components from having infinitesimal values. We kept the variance and covariance losses from VICReg unmodified.

Experiments & Results
Dataset The self-supervised and supervised trainings were performed and evaluated on a longitudinal dataset of OCT volumes of 1,096 patients fellow eyes 1 from a clinical trial 2 studying the efficacy of wet-AMD treatment of the study eyes. Patients had both eyes scanned monthly over two years with Cirrus OCT (Zeiss, Dublin, US) on monthly basis. The volumes consisted of 128 B-scans with a resolution of 512×1024 px covering a volume of 6 × 6 × 2 mm 3 . For the wet-AMD conversion prediction task, we selected fellow eyes that either remained in the intermediate AMD stage throughout the trial, or converted to wet-AMD during the trial, excluding those that had late AMD from baseline or converted to late dry-AMD. The final supervised dataset consisted of 463 eyes and 10,096 OCT scans with 117 converter eyes, and 346 non-converters. The rest of the eyes were included in the unsupervised dataset, which consisted of 541 eyes and 12,494 volumes. Following [17,19,13,2], wet-AMD conversion was defined as a binary classification task, i.e., predicting whether an eye is going to convert to wet-AMD within a clinically-relevant 6 months time-frame. For the supervised training, eyes were split into 60% for training, 20% for validation, and 20% for testing, stratified by the detected conversion. The evaluation is reported on the scan level.
Preprocessing & Augmentations In the supervised setting, we extracted from each OCT volume a set of 6 B-scans covering the central 0.28 mm, whereas 9 B-scans covering the central 0.42 mm were extracted for the SSL. To standardize the view, the retina in each B-scan was flattened with a quadratic fit to the RPE layer (segmentations are obtained with IOWA Reference Algorithm [21]). Then, each B-scan was cropped to 6 × 0.6 mm 2 and resized to 224 × 224 pixels, with intensities normalized between 0-1. All B-scans from an OCT volume were assigned the same conversion label for the supervised training. But during validation and testing, conversion probability of a volume was computed by picking the B-scan with maximum probability among the B-scans.
For the supervised training augmentations, random translation, rotation (max 10 degrees) and horizontal flip were used. When forming the pairs for the SSL, we followed [3,1] but with an increased minimum area ratio for the random cropping from 0.08 to 0.4, because in OCT volumes the noise ratio is higher than in natural images (Fig. 2), which makes small crops uninformative. Also, random grayscaling augmentation is not applicable. We picked the B-scans of the same patient from different visits with the time difference in the range of 90−540 days. The time difference acts an additional augmentation, which makes the task non-trivial. Additionally, two large crops from two B-scans yield similar color histograms, preventing network to memorize color histograms and overfit (Fig. 2(d)). Following the protocol in [1], the supervised and unsupervised data were combined for the SSL. Setup ResNet-50 was chosen as the encoder backbone, and an MLP with two hidden layers with batch normalization as the projector, similar to [3] except the dimensions were chosen as 4096 for all the SSL steps. In SSL, we used AdamW with batch size of 128, learning rate of 5 * 10 −4 and a weight decay of 10 −6 for 400 epochs. Following [3,6], a cosine learning rate scheduler with a warm-up of 10 epochs was used. In VICReg, the coefficient of the invariance term was fixed to 25, and we found improved performance when the coefficients of the variance and the covariance terms were set to 5 and 1, respectively. For the Barlow Twins, the coefficient for the redundancy reduction term was kept at 0.005.
For the downstream task, we provided the results from both the linear evaluation and the fine-tuning. The performance was evaluated with area under the receiver operating curve (AUROC) and the precision-recall curve (PRAUC). The linear evaluation was conducted by training a linear layer on top of the pre-trained & frozen encoder. It is trained with Adam optimizer, batch size of 128, learning rate of 10 −4 , and 5-to-1 class weights in the cross-entropy loss for 10 epochs. Fine-tuning had the same parameters with the addition of weight decay for 100 epochs. When training from scratch, the model was trained for 300 epochs. The best epoch was selected as the one with the highest AUROC score on the validation set. The learning rate was selected between 10 −2 -10 −5 . The weight decay was selected between 10 0 -10 −7 including 0.
Experiments We report results for wet-AMD conversion prediction task from linear evaluation and fine-tuning. We compared TINC against popular noncontrastive learning methods Barlow Twins and VICReg, which can accept different images as input pairs. When testing our new pair-forming scheme, we used the original VICReg and Barlow Twins along with their modified versions. In order to show the performance of TINC, we compared it against ResNet50 trained from scratch, VICReg and Barlow Twins modified with our new input scheme, and VICReg with additional explicit time difference prediction loss term. Although random crop & resize are the most crucial augmentations for noncontrastive learning, small crop area may not be ideal for OCT images due to the loss in contextual information. To verify this, we first tested Barlow Twins and VICReg with their original augmentations and input scheme as baselines followed by training them with the proposed pair-forming scheme and larger random crops. We observed (first section of Table 1) that with vanilla VICReg, its similarity loss quickly reached close to zero, and the representations collapsed. A significant improvement in performance was observed for both the methods with the proposed input scheme. The AUROC of Barlow Twins increased from 0.686 to 0.708 and VICReg achieved 0.685 AUROC score on linear evaluation.
On linear evaluation, TINC loss clearly outperformed both Barlow Twins and VICReg even after modifying them with our novel pair-forming scheme (1). TINC achieved 0.738 AUROC, while modified VICReg and Barlow Twins achieved 0.708 and 0.685 respectively. TINC captures the temporal information better with its temporally induced margin based approach leading to these improvements.
With end-to-end fine-tuning ( Table 2) the performance improvement due to the proposed TINC loss is more apparent, even after optimizing VICReg and Barlow Twins with our input pair scheme. We also compared our results against AMDNet [14], an architecture specifically designed for 2D B-scans. Interestingly, AMDNet could not outperform a ResNet-50 initialized with random weights.
In order to demonstrate that the temporal information is crucial in the conversion prediction task, we added time difference as an additional loss term to VICReg. For this, we concatenated the two embeddings and fed them to an MLP to obtain the time difference predictions. The labels are calculated as v 1 −v 2 w.r.t input order and scaled between -1 and 1. The MSE between the labels and the predictions is added besides the other VICReg loss terms. This clearly improved AUROC (Table 1, line 1-4), but the additional term increased the complexity of VICReg training. Whereas TINC implicitly uses the temporal information with its margin so that the representation capture the anatomical changes due to the disease progression. The experiments demonstrated that the TINC approach performs better in the downstream AMD conversion prediction task than adding the time difference as a separate term.
Additionally, we modified our loss as a squared epsilon insensitive loss to have smoother boundaries, but it degraded the performance. With Barlow Twins, we observed that the fine-tuning AUROC performance was worse than its linear evaluation by 0.016. This can be explained by the fact that in Barlow Twins, the fine-tuning reached the peak validation score within 10 epochs, same as the number of linear training epochs.

Discussion & Conclusion
The temporal information is one of the key factors to correctly model disease progression. However, popular contrastive and non-contrastive methods are not designed specifically to capture that. Additionally, they require strong augmentations to create two views of an image, which are not always applicable to medical images. We proposed TINC as a modified similarity term of the recent non-contrastive method VICReg, without increasing its complexity. Models trained with TINC outperformed the original VICReg and Barlow Twins in the task of predicting conversion to wet-AMD. Also TINC is not task or dataset specific, it is applicable to any longitudinal imaging dataset. Moreover, we proposed a new input pair-forming scheme for OCT volumes from different time points, which replaced the heavy augmentations required in the original VICReg and Barlow Twins and improved the performance.