{'fraction_non_alphanumeric': 0.03345704315072287, 'fraction_numerical': 0.01778097823076497, 'mean_word_length': 4.88887680765467, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Domain generalization in deep learning-based mass detection in mammography: A large-scale multi-center study

Computer-aided detection systems based on deep learning have shown great potential in breast cancer detection. However, the lack of domain generalization of artificial neural networks is an important obstacle to their deployment in changing clinical environments. In this work, we explore the domain generalization of deep learning methods for mass detection in digital mammography and analyze in-depth the sources of domain shift in a large-scale multi-center setting. To this end, we compare the performance of eight state-of-the-art detection methods, including Transformer-based models, trained in a single domain and tested in five unseen domains. Moreover, a single-source mass detection training pipeline is designed to improve the domain generalization without requiring images from the new domain. The results show that our workflow generalizes better than state-of-the-art transfer learning-based approaches in four out of five domains while reducing the domain shift caused by the different acquisition protocols and scanner manufacturers. Subsequently, an extensive analysis is performed to identify the covariate shifts with bigger effects on the detection performance, such as due to differences in patient age, breast density, mass size, and mass malignancy. Ultimately, this comprehensive study provides key insights and best practices for future research on domain generalization in deep learning-based breast cancer detection.


Introduction
Breast cancer is now the most common cancer worldwide, surpassing for the first time lung cancer in 2020 (Sung et al., 2021). It is responsible for almost 30% of all cancers in women and current trends show an increasing incidence (ECIS, 2021). In x-ray mammography, the gold standard imaging technique for early detection used in screening programs, breast cancer can be detected by identifying abnormalities in the breast structures, which could appear in the form of calcifications, architectural distortions, breast asymmetries, or masses. However, in breast cancer screening there is a high percentage of false-positives that may lead to unnecessary biopsies along with a high rate of false negatives or missed cancers (Siu, 2016;Lehman et al., 2017). The overlook or misinterpretation of abnormalities found in mammograms are the most common reasons for missed breast cancers (Bird et al., 1992).
Recently, a large-scale study (Rodriguez-Ruiz et al., 2019) compared the performance of an Artificial Intelligence (AI) system with the interpretation of 101 radiologists, concluding the AI stand-alone achieved a cancer detection accuracy comparable to an average radiologist in the retrospective setting. Similarly, in McKinney et al. (2020) the performance of AI stand-alone solutions for breast cancer screening was evaluated in different clinical settings showing superior cancer prediction rate compared to the double-reader human expert strategy. In contradiction to these findings, a similar study (Schaffter et al., 2020) assessed the performance of AI algorithms from 126 teams and 44 different countries in mammograms from the United States and Sweden and concluded that the top-performing methods did not improve the radiologists' sensitivity.
The contradictions in large-scale studies reassure the importance of external validation in publications involving AI for breast cancer detection in mammography. Most AI detection methods are not tested for out-of-distribution (OOD) generalization using a different domain than the one used during training. The domain shift may lead to an important performance decrease in different clinical settings -i.e. different scanner, imaging protocol, or patient cohort. To further increase the reliability and robustness of novel Computer Aided-Detection (CADe) methods it is urgent to study their generalization power apart from including external validation tests, as recommended by the FUTURE-AI guidelines . Kim  AI for diagnostic analysis of medical images and less than 6% included external validation.
Taking this into account, Domain Generalization (DG) is an active research area that aims to improve OOD generalization of AI solutions . Most DG research in medical imaging has been focused on the multi-source setting, which assumes images from multiple domains are available in the training set. On the other hand, single-source domain generalization (SSDG) assumes training images are homogeneous, coming from a single domain and lacking other domains during training. The goal of SSDG is to train a deep learning model to be robust against domain shifts using data from a single source domain. The SSDG setting is often more appropriate in medical imaging where public datasets are scarce and the data access is restricted.
In this study, we investigate the SSDG in the context of crossdomain breast cancer detection using digital mammography. In particular, we address breast mass detection, which is the most common pathology in public mammography image datasets. To the best of our knowledge this the first study of SSDG in mammography. Our main goal is to develop a CADe system based on deep learning that is robust to domain shifts in digital mammography. As shown in Figure 1, the sources of domain shift are mainly caused by covariate shift and the differences in the acquisition pipelines. In this paper, we address the DG problem where both types of domain shifts are present. To sum up, our contributions are as follows: • Extensive analysis of the mammograms in six different domains, highlighting the differences between domain and dataset shift and their potential effect in DG.
• Comparison of eight state-of-the-art detection methods, including Transformer-based architectures, fine-tuned for the task of mass detection in full-field digital mammograms using a single-source setting. The models' robustness is tested in five unseen domains, corresponding to different scanner manufacturers and datasets.
• Design of a SSDG training pipeline that boosts the breast mass detection performance and reduces the domain shift in unseen domains.
• Performance comparison by mass and breast attributes, highlighting the potential biases of the proposed model.
• Study of the DG after using Transfer Learning on each unseen domain.
We believe that this study will not only shed more light on the domain generalization of deep learning, but also pose as a comparative study of state-of-the-art object detection methods in the challenging task of mass detection in mammography on different clinical environments. We will also highlight the differences between domain and dataset shift in mammography and the possible effects in the detection performance. In the following section, we analyze the recent work on DG in medical imaging and mass detection in breast mammography. Covariate shift -differences in the distribution of masses and breasts. Label shiftintra-observer variability of experts' annotations. Acquisition shift -different scanner manufacturers and imaging protocols. Patient cohort -differences in demography, geographic area, socioeconomic status, and patient comorbidities. Samala et al. (2020) studied the generalization error of a deep convolutional neural network fine-tuned for the task of classifying malignant and benign masses on mammograms. They aimed to balance the learning and memorization power of the network by varying the proportion of corrupted data in the training set. They concluded that training with noisy data, i.e. including 10% of corrupted labels, could increase the generalization error and improve the performance of transfer learning strategies. Wang et al. (2020) outlined the inconsistencies in performance of deep learning models in mammography classification. A total of four datasets from different patient populations were used to evaluate six deep learning architectures. Results showed that the high performance obtained in the training dataset cannot be generalized to unseen external datasets, regardless of the model architecture, training technique or data labeling method. Recently, Li et al. (2021) studied the DG in lesion detection on different vendors using a contrastive learning scheme to extract domain invariant features. The method was trained with mammograms of three different vendors and evaluated on two unseen vendors, showing great generalization power. For the comparison with state-of-the-art generalization methods they only used the mean average precision (mAP) as evaluation metric and no statistical significance tests were performed to confirm the improvement.

Domain Generalization in Medical Imaging
In other fields of medical imaging, like chest X-ray, prior work also found variable generalization performance of deep learning models under the presence of cross-institutional domain shift (Zech et al., 2018). Cohen et al. (2020) studied the generalization performance of chest X-rays prediction models when trained and tested on datasets from different institutions. The conclusion reached was that the shift present in the labels had a much higher impact on the generalization error than the domain shift in the images. In this study, we also discuss the differences between the domain shift, caused by the different image acquisition protocols, and the covariate shift, present in the data of each domain. Recently, Zhang et al. (2021b) benchmarked the performance of eight domain generalization techniques on multi-site clinical time series datasets and chest X-ray images. None of the DG methods achieved significant gains in OOD performance on the chest X-ray imaging data. Opposite to our work, they did not include intensity scale standardization neither other single-source domain generalization techniques used in this study. Moreover, a single classification architecture, a DenseNet-121, was trained using drastically down-sampled images.
In magnetic resonance imaging (MRI), Mårtensson et al. (2020) examined the reliability of a deep learning model in clinical ODD data, being the largest study to date on the effect of domain shift in deep learning models trained with MR images. The conclusions stated that including more heterogeneous data from a wider range of scanners and protocols during training improved the performance in OOD data. Opposite to this, in our study, we focus on how to make models more robust when data from other institutions -i.e. domains-is not available. Also in MRI, Ouyang et al. (2021) proposed a causality-inspired data augmentation approach for single-source domain generalization for medical image segmentation and compared their method to other SSDG techniques showing superior performance. In this study, we include some of the techniques tested in Ouyang et al. (2021) to study their effectiveness in digital mammography.
In cardiac imaging, Zhang et al. (2020b) evaluated a deep stacked transformation data augmentation approach, named Bi-gAug, on three different 3D segmentation tasks covering two medical imaging modalities (MRI and ultrasound) involving eight publicly available challenge datasets. In four different unseen domains, BigAug obtains a comparable performance to the two state-of-the-art methods. Finally, in digital pathology and histopahology, the domain shift effect for deep learning has been studied in Thagaard et al. (2020); Stacke et al. (2019Stacke et al. ( , 2020.

Mass Detection in FFDM using Deep Learning
Detecting and classifying masses in mammograms using deep learning is widely covered in the literature (Abdelrahman et al., 2021). A large variety of deep learning models have been developed to assist radiologists in screening mammography. The models can be split in those that during training as input a single mammogram (Zhu et al., 2017;Ribli et al., 2018;Al-Masni et al., 2018;Wu et al., 2019;Yala et al., 2019;Agarwal et al., 2020), multiple scans (generally both views of the same breast) (Geras et al., 2017;Khan et al., 2019;Zhao et al., 2020), and patch-based approaches, using image patches (Dhungel et al., 2017;Shen et al., 2019;Wu et al., 2020;Ragab et al., 2021).
Most methods in the literature report their performance in the same domain used for training while transfer learning is used afterwards to adapt the model to new domains. Instead, we would like to evaluate the generalization power of models trained in a single-source setting with and without DG techniques and test their performance in unseen domains without using transfer learning. Additionally, we compare the best single-source DG model with transfer learning in five different domains.
Moreover, existing proposals in the literature employ a single well-known Convolutional Neural Networks (CNN) architectures like Faster R-CNN (Ren et al., 2016) or YOLO (Redmon et al., 2016) but the recent Transformers-based detection models (Carion et al., 2020;Zhu et al., 2021;Liu et al., 2021) are not very well explored yet. In this work, we also include these novel Transformer-based detection models and evaluate their generalization power compared to other CNN detection methods.

Robustness of Transformer-based Architectures
The OOD robustness of Transformer architectures has been analysed in recent publications since Transformers became more popular in Computer Vision tasks (Fort et al., 2021;Paul and Chen, 2021;Bai et al., 2021), mainly since Visual Transformers (ViT) were introduced (Dosovitskiy et al., 2020). Most of these papers conclude that due to the intrinsic properties of Transformers, mainly self-attention mechanisms and the lack of strong inductive biases of convolutions, they outperform CNNs in terms of OOD robustness.
As an example, Zhang et al. (2021a) used the most popular data-shift datasets of ImageNet (Deng et al., 2009) and reported a superior performance of the Transformer-based model, a DeiT (Touvron et al., 2021), against a single variant of the popular Big Transfer (BiT) CNN-based model .
However, a more extensive analysis considering most relevant variants of BiT and ViT, concluded that Transformers are not more robust but better calibrated than CNN models. Pinto et al. (2021) also questioned the superior robustness of Transformers solely attributed to their architecture components, e.g. self-attention mechanism and lack of inductive biases. They showed that the impact of pre-training is more important than the lack of self-attention, achieving superior performance than Transformers with a CNN pre-trained with weakly supervised procedures on large amount of data.
In conclusion, a good understanding of why self-attention mechanisms learn better representations in certain settings and how different pre-training strategies dramatically impact the downstream task is still lacking. In this study, we will compare the robustness of CNN-based versus Transformer-based object detection architectures trained in large-scale datasets and finetuned for the specific task of mass detection in a medium size digital mammography dataset (2,864 mammograms included in the training).

Transfer Learning in Breast Cancer Detection
Transfer learning has been used in mammography breast cancer detection to adapt the model to new domains, mainly new scanners and imaging protocols (Ribli et al., 2018;Shen et al., 2019Shen et al., , 2020Shen et al., , 2021. Nevertheless, there are two main drawbacks of transfer learning in medical imaging, the data availability and catastrophic forgetting. Catastrophic forgetting (French, 1999) is a phenomenon of artificial neural networks that occurs when a model is trained sequentially on multiple tasks, abruptly forgetting previously learned information upon learning new one. When fine-tuning the model in a new domain, we take the risk of over-fitting the model to the new test set, which may have less diversity of masses than the original training dataset. Nevertheless, the transfer learning performance can only be assessed if data from the new domain is available, which is not always the case when developing a new CADe system.

Full-Field Digital Mammography (FFDM) Datasets
Several open access X-ray mammography repositories can be found in the literature (Diaz et al., 2021). In this work, three FFDM datasets are used to study the robustness and generalization of the selected methods on different domains: a subset of OMI-DB, also known as OPTIMAM dataset, (Halling-Brown et al., 2021), the INbreast (Moreira et al., 2012) and the Breast Cancer Digital Repository (BCDR) (Moura et al., 2013), comprising a total of 4,352 FFDM (2,382 subjects) from six different domains including different scanner manufacturers and datasets. See Table 1 for a detailed list of the number of cases, mammograms, and annotated masses present on each domain.

OPTIMAM Mammography Image Database
The OPTIMAM dataset (Halling-Brown et al., 2021) is a shareable resource of digital mammography images from breast screening centers in UK including DICOM images, experts' annotations and clinical observations (e.g. pathological reports). The database contains FFDM of women with screendetected cancers and representative samples of normal and benign screening cases.
A subset of OPTIMAM, containing a total of 3,500 malignant and 500 benign cases, was used in this study. Each case in the dataset may contain different studies from the same patient. We made sure to split the training, validation and test sets by cases and not by study. The two most common views of each breast were used as independent inputs: the medio-lateral oblique (MLO) and cranio-caudal (CC) view. The image matrix of the mammograms is of 3328 x 4084 or 2560 x 3328 pixels depending on the vendor and the compression plate used in the acquisition.
OPTIMAM meets the requirements for a multi-center and multi-scanner study as it contains screenings from a total of three different centers and different scanner manufacturers. Among the different scanner manufacturers, only cases with annotated masses were selected. Four different domains were built from OPTIMAM: Hologic Inc., Siemens, GE and Philips (see Table 1), splitting the cases by scanner manufacturer:.

INbreast Dataset
INbreast mammograms (Moreira et al., 2012) were acquired from a single Portuguese center using a FFDM system, the MammoNovation from Siemens. Images, distributed in DI-COM format, have a matrix of 3328 x 4084 or 2560 x 3328 pixels, depending on the compression plate used in the acquisition. This public database consists of a total of 115 cases with different lesion types including masses, calcifications, asymmetries and distortions. Out of 115 cases only 50 contain masses, including a total of 116 annotations. Most INbreast lesions are not biopsy-proven and the malignancy of the mass is classified based on the BI-RADS assessment categories (Orel et al., 1999). It is common to group masses with BI-RADS ∈ {2,3} as benign and masses with BI-RADS ∈ {4,5,6} as malignant. INbreast will be used as single domain in this study.

Breast Cancer Digital Repository (BCDR)
The BCDR dataset (Moura et al., 2013;Arevalo et al., 2016) is a public dataset from 2012, currently discontinued and available by request. The dataset contains both digital (BCDR-DM) and film mammograms (BCDR-FM). In BCDR-DM, the dataset selected, a total of 90 subjects have biopsy proven mass lesion annotations. All images are supplied by the Faculty of Medicine -Centro Hospitalar São João, at University of Porto (FMUP-HSJ) and obtained using a MammoNovation Siemens FFDM scanner. Images have a matrix of 3328 × 4084 or 2560 × 3328 pixels, depending on the compression plate used in the acquisition, and are available only in 8-bit depth TIFF format. BCDR will be used as single domain in this study.

Domain Shift in Mammography
In Figure 1, we introduced the different domain shifts present in digital mammography. It is well-known that the one of the main sources of domain shift is caused by different scanner manufacturers and image acquisition protocols. In Figure 2, there are sample mammograms from each domain used in this work. The most notable differences among domains are the changes in intensity values and the contrast between the fibroglandular tissues and the adipose areas of the breast.
On top of the aquisition shift, medical imaging datasets suffer from additional covariate shift given by the different data distributions among datasets. Covariate shift is difficult to avoid due to the data scarcity and privacy constraints that obstruct the availability of large-scale medical imaging datasets for training. In mammography mass detection, the covariate shift is caused by differences in the masses -i.e. shape, size, malignancy, location -and the biological variations between patients -i.e. age, breast density.
Masses or nodules can appear in any location of the breast, with different shapes and sizes and look benign or malignant. Moreover, other factors like breast density can increase the difficulty of mass detection. In high density breasts, there is a higher probability that the dense tissues (parenchyma) occlude (or even simulate) masses and other breast lesions. For that reason, the overall sensitivity of mammography for breast cancer detection is reduced by more than a 20% in dense breasts (Kolb et al., 2002), even though women with dense breasts have a 4-6 fold increased risk of breast cancer compared to ones with low density breasts (Huo et al., 2014).
In Table 2, the total number of masses in each domain was categorized by mass size, status, patient age and breast density. Age and breast density information was not available on all the domains. The breast densities are split by BI-RADS categories (D'Orsi et al., 2013), being BI-RADS A almost entirely fatty breasts and BI-RADS D extremely dense breasts. Overall, INbreast and BCDR datasets have a much higher percentage of benign masses than the other four domains. BCDR dataset has the largest covariate shift, with 55% of benign masses, 15% of masses with less than 5 millimeter diameter and the youngest patient distribution. In the experiments, we will show the impact of this covariate shift in the domain generalization error of the mass detection system.

Methodology
Our analysis is done in three stages. First, a total of eight state-of-the-art object detection methods pre-trained on COCO dataset (Lin et al., 2014) are fine-tuned on a single domain for the downstream task of mass detection and tested on five unseen domains. Second, we select the most robust method as the baseline and test the generalization error after using different SSDG techniques in the training pipeline. Third, we test the improvement in performance after fine-tuning on each unseen domain and compare it to the performance of the single-source setting.
In the following sections, we describe the deep learning based object detection methods that were included in this analysis as well as the data preparation pipeline and the SSDG techniques that are used.

Object Detection Methods
In this section, the eight state-of-the-art object detection methods compared in this study are explained.

Anchor-based Detectors
Since the development of CNNS, object detection has been dominated by anchor-based detectors. These methods predict objects with predefined scales, aspect ratios and classes over every CNN feature locations in a regular, dense sampling manner. Anchor-based methods are generally divided into one-stage and two-stage methods depending on the times the coordinates of the anchors are refined, affecting both the performance and the computational efficiency. Among the anchor-based methods, one of the most successful approaches both in computer vision and medical imaging is Faster R-CNN.
Faster R-CNN (Ren et al., 2016): Faster R-CNN is a twostage anchor-based method consisting of a separate region proposal network (RPN) and a region-wise prediction network (R-CNN). Since its publication, many articles in object detection have been focused in improving its performance using different strategies -i.e. redesigning the architecture, including attention mechanisms, modifying the training strategy. Agarwal et al. (2020) used a Faster R-CNN for mass detection, also training with OPTIMAM mammograms from a single scanner manufacturer.

Anchor-free Detectors
Anchor-free detection methods became popular with the emergence of FPN (Lin et al., 2017a) and Focal Loss (Lin et al., 2017b). These methods find objects present in the image without preset anchors, eliminating hyperparameters and increasing their generalization ability.
ATTS (Zhang et al., 2019): a new Adaptative Training Sample Selection (ATTS) method to automatically select positive and negative training samples according to statistical characteristics of the object is proposed to bridge the gap between anchor-based and anchor-free methods.
PAA (Kim and Lee, 2020): proposes a new anchor assignment strategy, named Probabilistic Anchor Assignment (PAA), for single-stage detectors rather than the most common strategy of determining positive samples using Intersection-over-Union (IoU).
VariofocalNet (VFNet) (Zhang et al., 2020a): is designed to learn an IoU-Aware Classification Score (IACS) as a joint representation of object confidence and localization accuracy and perform a more accurate ranking of candidate detection bounding boxes. A new loss function, named Variofocal Loss, is introduced to train the dense object detector. Combining these two components and a box refinement branch, a new dense object detector is built based on FCOS+ATTS architecture.  AutoAssign (Zhu et al., 2020): to make positive label assignment fully data-driven and appearance-aware, AutoAssign presents a new sampling strategy to determine positive samples known as label assignment. Authors claim that AutoAssign can automatically adapt to different data distributions and achieves superior performance without any further adjustment.
YOLOF : revisits feature pyramids networks (FPN) for one-stage detectors and claim to achieve comparable results to RetinaNet (Lin et al., 2017b) and DETR (Carion et al., 2020) while being faster.

Transformer-based Detection Models
DETR (Carion et al., 2020): DEtection TRansformer is a query based set-prediction method to eliminate the need of many hand-designed components in object detection. DETR streamlines the training pipeline as a direct prediction problem, adopting an encoder-decoder architecture based on Transformers (Vaswani et al., 2017).
Deformable DETR (Zhu et al., 2021): alleviates the slow convergence and limited feature spatial resolution of DETR. The limitation of transformer attention modules in processing image feature maps has been tackled attending only to a small set of key sampling points around a reference.

Data Preparation and Training
The domain selected for training was the subset of OPTI-MAM images from Hologic Inc. scanner manufacturer. The dataset was split into train, validation and test sets with 70%, 10%, 20% of cases, respectively. The training dataset has a total of 1,924 cases with annotated masses containing a total of 2,864 mammograms. The mass status, either benign or malignant, and their conspicuity -a measure of difficulty of detecting the mass by radiologists -were balanced among the train, validation and test splits.
The image preprocessing pipeline consists of cropping the images to the breast region -discarding the background-resizing them to 1333x800 pixels keeping the aspect ratio and, finally, normalizing to the default mean and standard used in the pre-trained setup. The only data augmentation used was random image flipping, both vertically and horizontally.
In this study, we use MMDetection (v.2.13.0) PyTorch framework (Chen et al., 2019) and the pre-trained models available in their GitHub repository. All models have been pre-trained on the COCO dataset (Lin et al., 2014) using 1333x800 pixel image resolution and fine-tuned for the task of mass lesion detection in FFDM scans. A single GPU (24GB NVIDIA GeForce RTX 3090) was used for fine-tuning the models during a maximum of 50 epochs, using a batch size of 2 and adjusting the learning rate as recommended in the framework. The default learning rates in MMdetection metods were adjusted to train with two samples in a single GPU instead of the default two samples in 8 GPU setting (diving the learning rate by 8). During fine-tuning, the epoch with better mean Average Precision with a 50% bounding box overlap (bbox_mAP_50) on the validation set was selected as the best model. Most methods converged before epoch 20, only PAA and DETR needed more than 30 epochs to reach the convergence. Find additional settings used in the fine-tuning of the eight methods in Table 3. For Faster R-CNN, we follow the recommendation in Agarwal et al. (2020) for the anchor boxes scales and ratios.

Single Source Domain Generalization (SSDG)
In Figure 3 there is an overview of the different SSDG techniques tested in the training setup.

Intensity Scale Standardization
Intensity, as well as texture, is a domain-dependent feature. CNNs are known to be susceptible to shifts in intensity (Jacobsen et al., 2019). There are two main approaches to remove the intensity shift among domains. First, using intensity-based data augmentation during training or, second, standardizing the intensities of the images before feeding them into the model -i.e. data harmonization. In our experiments, we performed intensity scale standardization (Nyúl et al., 2000), which has shown great improvements in domain adaptation in medical imaging (Kushibar et al., 2019).
This technique was originally designed to standardize the intensity scales of MR images and ease the extraction of quantitative information. It is a two-step post-process method that aims to match similar intensities to similar tissue meaning. In the first step, the standardized histogram is learned from the training images, extracting the histogram landmarks. In the second step, the landmarks are used to linearly map the intensities of input images before feeding them into the network.

Data Augmentation Methods for Domain Generalization
Two data augmentation methods are tested, namely Cutout (DeVries and Taylor, 2017) and RandConv (Xu et al., 2020). Cutout enforces the model to be robust to corruptions and missing features by deliberating removing square patches from training images at random locations. In our experiments, we have tried a variety of sizes for the patches and concluded that in order not to miss small masses and important texture information, a patch size of 1 or 2 pixels was the most effective strategy. In the data augmentation pipeline, Cutout is applied with probability p=0.5 and a maximum of 10% of the total pixels is removed.
RandConv is a data augmentation strategy that generates images with random local textures but consistent shapes using linear filtering. The size of the convolution filter k determines the smallest shape it can preserve. As an example, with k=2, 2x2 random convolutions perturb shapes smaller than the filter size, which are considered local texture. Inspired by Augmix (Hendrycks et al., 2019), the authors also propose to blend the original image with the outputs of the RandConv layer via linear combination by a factor α. In our experiments, the best results were obtained using k=(1,3,5) and combining the outputs with the input images using α = 0.5 with probability p=0.5.

Synthesizing Novel Domains with MixStyle
One way of increasing the diversity of source domains to improve OOD generalization is synthesising novel domains using only the training data. MixStyle (Zhou et al., 2021b) is a simple and versatile method inspired by style transfer. Capturing the style information by the bottom layers of a CNN and mixing styles of training instances results in novel domains that increase the diversity and hence the generalization of the trained model. The method mixes the feature statistics of two instances to synthesizes new domains during the mini-batch training.
All the methods tested use a ResNet as the backbone to extract the image features. The authors of MixStyle recommend adding one MixStyle layer after the first residual blocks of the ResNet, typically after block one, two and three, and test which is the best configuration depending on the task.

Transfer Learning on Unseen Domains
In a final experiment, we compare the transfer learning ability of the baseline model and the model trained using different SSDG strategies. To that end, the test datasets are split in train, validation and test using the 80%, 5% and 15% of cases, respectively. The models, previously fine-tuned in OPTIMAM Hologic dataset, are fine-tuned again in the new domain. The fine-tuning settings are the same as in the previous experiments but the convergence was reached before 15 as the data available for fine-tuning is small.

Evaluation Metrics
In breast cancer mass detection, the True Positive Rate (TPR), also known as sensitivity or recall, is commonly used as the metric of reference to evaluate the performance of the CADe systems (Abdelrahman et al., 2021). The TPR penalizes the missed masses and rewards the detected ones. In commercially available CADe systems the TPR is typically reported in a range of (0.75, 0.85) false positives per image (FFPI) Ribli et al. (2018).
The area under the curve (AUC) of the Free-response Receiver Operating Characteristic (FROC) curve (Bandos et al., 2009) is used to compare the methods. The AUC is computed varying the confidence threshold of each bounding in a range of FPPI ∈ [0, 1]. A bounding box is a true positive (TP) when the Intersection-over-Union (IoU) of the prediction and the ground truth is greater than the 10%, as recommended also in (Agarwal et al., 2020). Even if a 10% may seem very low for detectionan IoU of 50% is typically used in general Computer Vision-, we evaluated the TPR versus the IoU threshold in the training dataset and confirmed that increasing the IoU more than a 10% had a negative impact in the TPR.
Following the recommendations of (Demšar, 2006) to compare multiple classifiers over multiple datasets, the Friedman test (Friedman, 1940) was used to reject the null-hypothesis. The Friedman test ranks the algorithms from best to worst on each dataset (domain) with respect to their performances, the AUC in this study. Additionally, a post-hoc test is needed to rank the algorithms from best to worst, comparing all classifiers to each other. In this case (Demšar, 2006) suggests the Nemenyi test (Nemenyi, 1963). In Nemenyi test, the performance of two classifiers is significantly different if the corresponding average ranks differ by at least the critical difference (CD). Table 4 shows, for each domain, the performance of the methods in terms of TPR at 0.75 FPPI, 95% confidence intervals, and AUC. Additionally, Figure 4 contains the FROC curves for the six different domains. Comparing the FROC curves with the metrics on the table, we can see that a higher AUC correlates with the highest TPR at 0.75 FPPI.

Performance Comparison of Mass Detection Models
All methods were able to detect masses in the source domain -the OPTIMAM Hologic -achieving a TPR higher than 90%. The good performance was maintained in OPTIMAM Siemens domain, likely being the dataset with less shift from the source domain. The other four domains had a significant drop in the AUC, being OPTIMAM Philips, OPTIMAM GE and BCDR the most affected ones. Among the five unseen domains, the Table 4: Performance comparison of mass detection methods. The metrics correspond to the True Positive Rate (TPR) at 0.75 false positives per image (FPPI), the TPR 95% confidence interval (CI), and the AUC of the corresponding FROC curves. All the methods have been fine-tuned using OPTIMAM Hologic manufacturer mammograms. The first column corresponds to the performance on the test set and the following ones evaluate the domain generalization performance on the other five unseen domains. The last column corresponds to the average AUC over the six different domains. The methods are ATTS (Zhang et al., 2019), AutoAssign (Zhu et al., 2020), Deformable DETR (Zhu et al., 2021), Faster R-CNN (Ren et al., 2016), DETR (Carion et al., 2020), PAA (Kim and Lee, 2020), VariofocalNet (VFNet) (Zhang et al., 2020a), YOLOF . The models with best performance are shown in bold.  (Zhu et al., 2020), Deformable DETR (Zhu et al., 2021), Faster R-CNN (Ren et al., 2016), DETR (Carion et al., 2020), PAA (Kim and Lee, 2020), VFNet (Zhang et al., 2020a), YOLOF . The eight mass detection methods are trained in OPTIMAM Hologic dataset.
Transformer-based detection methods, DETR and Deformable DETR, were the more robust. The Deformable DETR model showed the greatest generalization power with an average AUC of 0.79. The methods that performed worst in terms of TPR and AUC were PAA and Faster R-CNN.
To further confirm the statistical difference of the tested methods, we ran a Friedman chi-square test using the AUC over all six domains. The test gave a p-value of 5.38e − 06, rejecting the null-hypothesis and confirming that the methods are not equivalent and their mean ranks are different.

Single-Source Domain Generalization Techniques
In this experiment, the most robust method, the Deformable DETR, was selected as the baseline. Then, the different single-source domain generalization techniques were added to the training pipeline sequentially. As shown in Figure 3, Intensity Scale Standardization was applied prior to any data augmentation. Later, Cutout and RandConv were added as data augmentation with 0.5 probability. Finally, three MixStyle layers were included in the backbone network used as the image feature extractor. Table 5 shows the performance of the models, on each domain, in terms of TPR at 0.75 FPPI, its 95% confidence intervals, and AUC.
The Deformable DETR trained with Intensity Scale Standardization showed the major gain in performance among all stand-alone methods tested, increasing the average AUC from 0.79 to 0.86, and boosting the TPR of the worst performing domains except from BCDR. The second approach, adding Cutout  (Zhu et al., 2021) and the model trained with different single-source domain generalization techniques: Intensity Scale Standardization (ISS) (Nyúl et al., 2000), Cutout (CO)(DeVries and Taylor, 2017), RandConv (RC) (Xu et al., 2020) and MixStyle (MS) (Zhou et al., 2021b) and a combination of them. The metrics correspond to the True Positive Rate (TPR) at 0.75 false positives per image (FPPI), the TPR 95% confidence interval (CI), and the AUC of the corresponding FROC curves. All the methods have been fine-tuned using OPTIMAM Hologic manufacturer mammograms. The first column corresponds to the performance on the test set and the following ones evaluate the domain generalization performance on the other five unseen domains. The models with best performance are shown in bold.  data augmentation, also improved the AUC from 0.79 to 0.82. RandConv data augmentation seemed to downgrade the AUC in every domain except that of OPTIMAM GE. Last, MixStyle layers also helped to improve the performance on unseen domains with an average AUC of 0.81. The last two models include a combination of all the SSDG methods with and without RandConv. The model fine-tuned using the combination of Intensity Scale Standardization, Cutout data augmentation and MixStyle layers (ISS + CO + MS) gave the best results, boosting the average AUC by 2%, reaching 0.88. In OPTIMAM GE the AUC improved from 0.71 to 0.91, in OPTIMAM Philips from 0.74 to 0.86 and in INbreast from 0.81 to 0.99. Nevertheless, none of these SSDG techniques seemed to improve the mass detection performance in BCDR.
Additionally, other intensity based data augmentations, such as histogram equalization and inverting intensity values, were tested. However, the final performance downgraded drastically, hence, the results were not included in the paper.

Detection Performance by Mass and Breast Attributes
Following the distribution of the datasets (Table 2), we tested the performance of the best SSDG model (ISS + CO + MS) on the different domains by mass status, mass size, patient age and breast density. The sensitivity (TPR) values can be found in Table 6.

Mass Status
An unbalance between benign and malignant masses is found in the source domain: only 9% of the annotated masses in OP-TIMAM Hologic are benign (see Table 2). When evaluated individually, the TPR for malignant and benign masses were 0.949 and 0.903, correspondingly. In OPTIMAM Siemens and OPTIMAM GE, the percentage of benign masses is 8% and 2%, respectively, and all of them were detected. OPTIMAM Philips has only malignant masses, which were detected with a TPR of 0.897. INbreast and BCDR are the domains with more bias from the source domain in terms of mass status. INbreast's 35% of masses are benign and all of them were detected reaching a 100% TPR. On the other hand, in BCDR, 55% of the total masses are benign and only half of them were detected (TPR of 0.564).

Mass Size
To have a better representation of the detected and missed masses by size, on each domain, we illustrate the bounding box size distribution in Figure 5. For comparison, in Table 6 there is the performance by mass diameter, closely related to the width and the height of the bounding box.
In OPTIMAM Hologic, Figure 5a, most of the masses are between 5 and 25 millimeters of diameter. The range with more samples, 10 − 15 mm mass diameter, was also the range with lower sensitivity (0.922). Masses with less than 10 mm of diameter were correctly detected, confirming that the input size of the images after resizing is enough to detect small masses. Less than 1% of the masses in the dataset are less than 5 millimeters and most of them were missed (0.333 TPR). Masses larger than 30 mm, an 8% of the total, were correctly detected with 0.987 sensitivity.
OPTIMAM Siemens (Figure 5b) and OPTIMAM GE ( Figure  5c), similarly to OPTIMAM Hologic, do not have many masses larger than 30mm diameter and the missed masses are mostly between 5 and 20 mm. In OPTIMAM Philips (Figure 5d), even though the mass size distribution is similar to the source domain, larger masses (> 25mm) were undetected (0.721 TPR). One can notice a high height-to-width ratio of some masses compared to the rest of domains. In INbreast, all masses were detected, even those smaller than 5 millimeters of diameter. In BCDR, 15% of the masses are smaller than 5 millimeters of diameter and most of them were missed (0.276 TPR). The sensitivity in the range of 5 to 15 mm diameter is still low compared to the sensitivity of masses bigger than 15 mm of diameter.

Age
In all OPTIMAM domains, most of the cases are from patients in the range between 50 and 70 years old. Although, in OPTIMAM Hologic, the performance in cases with patients younger than 50 and older than 70 years, was better than the one in the majority group. In OPTIMAM Siemens, the performance is lower in the group of the patients older than 70 years, contrary to OPTIMAM GE where the worse performing group was the one of patients younger than 50. Finally, in OPTIMAM Philips, all masses of patients younger than 50 were detected, and the performance was stable among the other groups. In BCDR, the cases were balanced among the four age groups, opposite to OPTIMAM GE, which performance was better in patients younger than 50 years, compared to the other groups, being this worse but uniform. In INbreast dataset, the age information is unavailable.

Breast Density
Breast density information is not available for all the images in OPTIMAM dataset. In OPTIMAM Hologic, most cases are in the BI-RADS B category, even so, the performance was similar in the four categories. In OPTIMAM Siemens, only 20% of the cases have breast density information and all of them were correctly detected. In OPTIMAM GE, there was a performance drop (0.70 TPR) in BI-RADS C group while in OPTIMAM Philips was in BI-RADS D (0.714 TPR). Finally, the breast density distribution is similar in INBreast and BCDR. Nevertheless, in INbreast, all masses were correctly detected, while in BCDR, the sensitivity was only higher for the BI-RADS B category.

Transfer Learning on Unseen Domains
In the next experiment, transfer learning was used to further adapt the models to every unseen domain. On that account, every domain was randomly split in train, validation and test sets, Table 7: Performance comparison of the baseline and the best performing methods trained in a single-source setting with the models after applying transfer learning (TL) on each domain. The metrics correspond to the True Positive Rate (TPR), or sensitivity, at 0.75 false positives per image (FPPI), the TPR 95% confidence interval (CI), and the AUC of the corresponding FROC curves. The models with best performance are shown in bold. using the 20% of the images for fine-tuning and the 80% for testing. Table 7 shows the performance of the methods before and after transfer learning. The performances before transfer learning were computed again for a fair comparison, as each domain was reduced a 20% for fine-tuning purposes. Before fine-tuning, the ISS + CO + MS model continues having the best performance, with a gain of 9% over the baseline average AUC (0.75).

Method
After fine-tuning on each domain, the baseline performance improved in all domains excluding OPTIMAM Siemens, where the AUC dropped by 4%, from 0.86 to 0.82. The biggest improvements were seen in OPTIMAM Philips and BCDR, where the AUC improved by 10% and 9%, respectively. OPTIMAM GE and INbreast improved their AUC by 6% and 7% each. The Deformable DETR model trained with Intensity Scale Standardization (ISS), only showed improvement in the BCDR domain, while in BCDR, the AUC augmented from 0.64 to 0.76, reaching a sensitivity of 0.831 at 0.75 FPPI. The best performing model -the Deformable DETR trained with Intensity Scale Standardization, Cutout data augmentation and MixStyle layers in the feature extractor -had a similar behavior than the ISS model. The only domain that improved was BCDR. However, the improvement was the largest one showing an increase of the AUC from 0.66 to 0.80 and reaching a sensitivity of 0.873 at 0.75 FPPI.

Discussion
In our first experiment, we compared the performance of eight detection methods fine-tuned for the task of mass detection in digital mammography. The selection comprised state-ofthe-art anchor-based, anchor-free and Transformer-based detection methods. After evaluating their performance on five unseen domains, we concluded that Transformer-based methods were more robust to domain shifts in mammography datasets, being the Deformable DETR the best overall. As discussed in Section 2.2.1, the OOD robustness of Transformers has been pointed out in recent publications. Nevertheless, it can be missleading to conclude that their superior robustness is given by the intrinsic properties of Transformers -i.e. the self-attention mechanism and the lack of inductive biases. Regardless, we can conclude that, in our specific setting, Transformers-based methods learned better representations and generalized better on unseen domains than other detection methods. Then, the Deformable DETR model trained was selected as the baseline for the next experiments.
In our second experiment, four different SSDG techniques were introduced in the training pipeline to improve OOD performance on unseen domains. ISS showed the major gain in performance among all stand-alone methods introduced, supporting that deep learning detection models are highly affected by the intensity distribution of the input images. In this regard, other intensity based data augmentations, such as histogram equalization, were tested to further improve the robustness but were unsuccessful. We believe that the intensity alterations disturbed the data and added noise during training, reducing the final performance. This has also been confirmed when testing the RandConv method, which ultimately can be seen as an intensity based data augmentation method, where training only with the augmented images downgraded the performance, possibly because the intensity shifts looked unrealistic. The MixStyle layers and the Cutout data augmentation also helped improving the robustness among domains. In Cutout, the patch sizes chosen were one and two pixels, which can be argued as adding salt and pepper noise without the salt (bright pixels) to the data augmentation pipeline. For MixStyle, the best results were obtained adding one layer after the first three residual blocks of the ResNet50 used as feature extractor. The combination of Intensity Scale Standardization, Cutout data augmentation and MixStyle layers gave the best results in all unseen domains, except from the BCDR, in terms of average AUC and sensitivity.
To have a better understanding of the model biases caused by different dataset shifts, we evaluated the detection performance by clinical and demographic variables such as mass status, mass size, breast density, and age. In the BCDR domain, there are big disparities in the performance by mass attributes. The performance dropped drastically for benign masses, which are the 55% of the masses in BCDR. In the source domain, the OPTIMAM Hologic dataset, benign masses represent only 9% of the total masses. Therefore, we could have argued that the drop in performance in BCDR benign masses was caused by the class unbalance present in the source domain but in contradiction, in INbreast dataset all benign masses, comprising 35% of the total, were detected. Moreover, all benign masses were detected in the other three unseen domains of OPTIMAM dataset. Additionally, the model also failed to detect small masses in BCDR, which is the domain with the biggest proportion of masses smaller than 5 millimeters of diameter, 15% of the total. Regarding the mass size, OPTIMAM dataset has few masses smaller than 5 millimeters diameter and bigger than 30 millimeters. Still, the performance in OPTIMAM domains is consistent over the different mass size groups except in OP-TIMAM Philips. In OPTIMAM Philips, some of the masses missed in the detection have a higher height-to-width ratio compared to other domains (see Figure 5d). Finally, we did not observe any correlation between the mass detection performance among different age groups and breast densities. From all this observations we can extract that the model seems to have a bias towards masses smaller than 5 millimeters of diameter and bounding boxes with a high height-to-with ratio, presumably because those samples were not representative in the training dataset (see Figure 5a).
As mentioned in Section 3.4, BCDR has the largest dataset shift in terms of mass size and mass status with respect to the training set and that may be the reason why it is the worst performing domain. In our last experiment, we found that Transfer Learning helped to mitigate the dataset shift in the BCDR domain. In Figure 6, we can observe that most of the small masses missed before fine-tuning are correctly detected after fine-tuning using 20% of the mammograms from BCDR. Inspecting the small masses missed before fine-tuning, we found that most of them contained small calcifications inside or surrounding the masses. Our reasoning is that this type of masses were not represented in the original training set and not learned until fine-tuning in BCDR. However, the performance of the best performing model (ISS + CO + MS) decreased on the other domains after fine-tuning. That finding correlates with the risk of suffering catastrophic forgetting, one of the major limitations of applying Transfer Learning on a small dataset.
One limitation in this study is the unbalance of the training set in terms of mass and patient attributes. Adding more samples to the minority classes could help to evaluate better the the detection performance and have a more fair comparison among subgroups. The minority classes in the training set were: benign masses, masses smaller than 5 millimeters of diameter, high height-to-width ratio bounding boxes, patients with high breast density and patients out of the range between 50 and 70 years old.

Conclusion
In this study, we evaluated different methods for mass detection in mammography on six different domains. Our experimental results showed that Transformer-based detection models were more robust to domain changes. Moreover, we highlighted the importance of SSDG techniques to reduce the domain shift and improve the performance in unseen clinical environments. The proposed training pipeline mitigated the domain shift present in four out of the five domains not seen during training. The results demonstrated that in one domain, the dataset shift, given by a higher proportion of small masses, had a bigger impact than the domain shift caused by the acquisition pipeline. Additionally, we found that Transfer Learning helped to mitigate the dataset shift in that domain but decreased the performance on other domains. Transfer Learning is a powerful technique to mitigate the dataset shift, however, as shown in the results, it is not always successful and has to be applied carefully to avoid catastrophic forgetting. Furthermore, we believe that future work should focus on Continual Learning for AI in breast cancer detection. Continual Learning has a great potential -both in a federated or a distributed manner -to allow the CADe systems to avoid issues such as catastrophic forgetting, dataset shifts present and demographic biases.

Acknowledgments
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 952103. A subset of the OPTIMAM database was obtained as part of the data-sharing agreement with the University of Barcelona in 2021. We are also thanks to Volpara Health (Dr Melissa Hill) for agreeing to share the breast density information available of the OPTIMAM subset used.