{'fraction_non_alphanumeric': 0.02835645807208446, 'fraction_numerical': 0.013429877432142712, 'mean_word_length': 4.918601867368925, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Weakly Supervised Attention Model for RV StrainClassification from volumetric CTPA Scans

Pulmonary embolus (PE) refers to obstruction of pulmonary arteries by blood clots. PE accounts for approximately 100,000 deaths per year in the United States alone. The clinical presentation of PE is often nonspecific, making the diagnosis challenging. Thus, rapid and accurate risk stratification is of paramount importance. High-risk PE is caused by right ventricular (RV) dysfunction from acute pressure overload, which in return can help identify which patients require more aggressive therapy. Reconstructed four-chamber views of the heart on chest CT can detect right ventricular enlargement. CT pulmonary angiography (CTPA) is the golden standard in the diagnostic workup of suspected PE. Therefore, it can link between diagnosis and risk stratification strategies. We developed a weakly supervised deep learning algorithm, with an emphasis on a novel attention mechanism, to automatically classify RV strain on CTPA. Our method is a 3D DenseNet model with integrated 3D residual attention blocks. We evaluated our model on a dataset of CTPAs of emergency department (ED) PE patients. This model achieved an area under the receiver operating characteristic curve (AUC) of 0.88 for classifying RV strain. The model showed a sensitivity of 87% and specificity of 83.7%. Our solution outperforms state-of-the-art 3D CNN networks. The proposed design allows for a fully automated network that can be trained easily in an end-to-end manner without requiring computationally intensive and time-consuming preprocessing or strenuous labeling of the data.We infer that unmarked CTPAs can be used for effective RV strain classification. This could be used as a second reader, alerting for high-risk PE patients. To the best of our knowledge, there are no previous deep learning-based studies that attempted to solve this problem.


INTRODUCTION
PE refers to obstruction of pulmonary arteries by blood clots. Among all cardiovascular diseases, PE is the third most common cause of death, after coronary heart disease and stroke. It accounts for approximately 100,000 deaths per year in the United States alone [1]. The clinical presentation of PE is variable and often nonspecific, making the diagnosis challenging [2]. Thus, rapid diagnosis and accurate risk stratification are of paramount importance. High-risk PE is caused by RV dysfunction from acute pressure overload. It is predictive of in-hospital mortality and identifying the need for more aggressive therapy. RV enlargement can be detected via CTPA, which also provides the diagnosis of PE. Therefore, CTPA can link between diagnosis and risk stratification [3,4].
A major limitation of imaging in PE diagnosis and in assessing treatment response is the lack of robust quantitative assessment. Measurements are currently done manually, which is laborious and involves interreader variation. Computerized analysis can support the clinical workflow by enabling quantitative, reproducible evaluation.
Deep learning applications have shown success in various medical imaging tasks such as Chest x-ray classification [5], pancreas segmentation [6] and anomaly segmentation in brain MRI [7]. Our task of automated risk stratification of PE still poses a challenge for researchers. The variability PE appearance, and lack of public datasets, make PE distinction difficult [8,9,10]. Most existing works on PE focus on the detection task. Computer-aided detection (CAD) systems have been suggested to detect the presence of PE. Most used convolutional neural networks (CNNs) for classifying PE candidates that are first extracted from an entire CTPA volume based on voxel-level features using traditional feature engineering methodologies. While partially successful, a few major limitations include the need for manual feature engineering, complex preprocessing, a high number of false positives, and an uncertainty degree in generalization [11,12,13,14,15]. A more recent solution from Yang et al. [16] presented a cascaded two-stage CNN achieving a high sensitivity of 75%. However, the model requires the division of the CTPA into small cubes rather than processing the entire CTPA scan. This approach still involves some preprocessing and suffers from a lack of an "end-to end" solution required for a clinical application. PENet [17] is a 3D CNN that aims to detect the PE in a series of slices from a CTPA study and currently achieves state-of-the-art performance for PE detection. It processes each slice using a 2D CNN and then uses a 3D CNN to aggregate information from many consecutive slices. This solution requires the labeling of each of the CTPA scan slices individually for the training process. The latest publication from Shi et al. [18], improves AUC accuracy compared to PENet by adding supervised attention maps that use pixel-level annotations for a small set of data. Other works monitor RV motion or enlargement in MRI scans [19,20,21], or predict pulmonary hypertension in CTPA scans [22]. In all the papers presented above, the methods are either fully supervised where the datasets contain full segmentation maps or are labeled on a 2D slice-level and require heavy preprocessing of the data.
A frequent problem when applying deep learning methods to medical images is the lack of labeled data. Manual labeling of images is an expensive and time-consuming process. This issue motivates approaches beyond traditional supervised learning by incorporating other data and/or labels that might be available. These approaches include semi-supervised, weakly-supervised, multiple instances, and transfer learning [23]. In the current study, instead of relying on manually produced pixel-level annotations, a single patient-level label is extracted from radiology reports. This distinguishes our design as a fully automated network that can be trained in an end-to-end manner without requiring preprocessing or exhausting labeling of the data.
We developed a weakly supervised deep learning algorithm, with an emphasis on a novel attention mechanism, to automatically classify RV strain from 3D CTPA scans. We demonstrate the implementation of attention modules in various 3D residual and densely connected network architectures baselines. We compare our model to state-of-the-art 3D CNN classification models. The results show that attention consistently improves prediction accuracy across different models. Also, our model performance surpasses all compared classification networks. The main contributions of this paper can be summarized in the following three aspects:

3D DenseNet with Residual Attention Blocks Architecture
The network's backbone is a 3D DenseNet architecture [24], further improved by residual attention blocks. The 3D DenseNet Attention model is inspired by the residual attention network for image classification [25] and hand gesture recognition videos [26]. In these works, attention modules were integrated into the deep residual network's layers to generate attention-aware features. Our solution generalizes this design from 2D to 3D and uses a baseline of densely connected blocks rather than residual ones. We propose soft attention, image-grid based gating that allows attention coefficients to center on local regions. Four 3D residual attention blocks are incorporated into the network. Each attention block captures different types of features extracted from a different layer in the baseline network.
We present two architectures constructed from the same baseline but differ in how the attention blocks are integrated. In the first architecture -Multi-Layer Attention Network (MLANet), we integrate the blocks in a more common practice as first introduced in [27]. This is a multi-layer feature extraction approach. The baseline network is used as a discriminant feature extractor, and several attention blocks are applied on extracted features from different layers throughout the network to create attended feature vectors at different scales. The attended feature vectors are flattened via channel-wise global average pooling and combined for the final prediction. As the network is constrained to classify based on the aggregated vector, it is forced to extract the most salient features for each class. The attention blocks are generated from extracted features from different layers in the baseline network. Therefore, it preserves local information and spatial context or attended features in both coarse and fine scales. The network architecture's block diagram is presented in  For the second network -Stacked Attention Network (SANet), we incorporate the residual attention blocks between the network's layers in a stacked manner. Fig 2 shows the block diagram of the developed system. In this approach, the blocks are added as an integrated part of the network and not as a separate branch of predictions, as in the previous architecture. Due to this fact and the differentiability of the attention blocks, this network can be easily trained in an end-to-end manner. This network adaptively changes attention as the feature changes and layers go deeper. Each attention block learns and captures different types of features (will be presented in Sec. 3.3). Thus, wrong attention predicted by one block can be masked by the other attention blocks. This makes a multiple block network of this sort quite robust to the attention prediction, as will be shown in Sec. 3.5. The two approaches are compared in Sec. 3.1.

3D Residual Attention Block
The 3D attention block is comprised of two units: mask and trunk. The trunk branch is constructed from ResNet residual block units to perform feature processing. The mask is designed of soft attention structure [28] that creates a 3D attention coefficients mask from the 3D features generated by the trunk. Thus, the mask behaves as a feature selector. The output of the attention module A(x) for input x, which gives us the weighted feature structure, is created by the element-wise multiplication of trunk feature-maps T (x) and attention coefficients created by the mask layer M (x), as defined in equation 1: Here: i ∈ {1, ..., H * W } ranges over all spatial positions, H and W are the height and width of the scan, d ∈ {1, ..., D} is the slice index, and c ∈ {1, ..., C} is the index of the channel. The entire attention structure can be trained end-to-end. During back-propagation, the attention mask serves as a gradient update filter due to its property of differentiability. In the soft mask branch, the gradient of the mask for the input features is shown in equation (2).
Where x is the input, φ are trunk layer parameter, and θ are mask layer parameters. The partial derivative of the trunk features T is multiplied by a factor of the mask M . Therefore, if the trunk features are not correct the mask can prevent the features from updating the parameters.
In our presented networks, we stack multiple attention blocks. This is designed for several reasons: First, RV strain classification from volumetric CTPA is a challenging task considering the number of features the network has to learn for a single scan as compared to a single image; The multiple attention network mitigates this problem by learning multiple masks. Second, the use of multiple attention blocks makes the network more robust. It can capture different attention types, focus on different features at each attention block, and correct a wrong prediction made from one block by other blocks.
For attention module stacking in SANet architecture, we need to use attention residual learning [25], rather than a naive multiplication as in equation (1), as described in equation (3): Here, similar to ideas in residual learning, adding "1" to the generated mask M (x) preserves the identity function of the residual network, as the trunk T (x) is not multiplied with the zeros from the mask M (x). This allows us to stack and integrate multiple attention modules without the obvious performance drop of the straightforward multiplication.
The mask is structured from a fully convolutional autoencoder design that resembles the popular U-Net [29] used for image segmentation tasks. However, the difference between this structure and the U-Net is that our mask branch aims to improve trunk branch features rather than solve a complex problem directly. This structure creates a mask that acts as a filter to the trunk layer features. The mask branch contains an encoder which collects global information and a decoder part that combines global information with original feature maps. 3D max-pooling is applied several times for down-sampling after a small number of residual units. After reaching the lowest spatial resolution, the global information is then expanded by a symmetrical architecture. 3D interpolation layers up samples the output after some residual units. The number of 3D interpolation layers is the same as 3D max-pooling layers so that the output of the mask layer will have the same dimensions as the output from the trunk layer. Finally, a sigmoid layer normalizes the output range to [0, 1] after two consecutive 1 x 1 x 1 convolution layers. Skip connections are also added, connecting the encoder and decoder structures, to get the feature information from various scale levels.
We add four attention blocks after each layer in the network. The number of downsampling (and corresponding upsampling) layers performed in each attention block is reduced as we go deeper into the network. The lowest spatial resolution is shared. For example, the attention block added after the first layer has four 3D max-pooling layers, whereas the block added after the second layer has only three layers, and so on. The full module for the second attention block is illustrated in

Dataset and Annotations
We apply our suggested models to the specific challenging problem of RV strain classification from 3D CTPA scans. An institutional review board (IRB) approval was granted to this retrospective study. The IRB committee waived informed consent. We retrieved data of consecutive patients diagnosed with PE in our emergency department (ED). All the patients underwent CTPA between 1/2012 to 12/2018. All the scans were interpreted by board-certified radiologists. We used the CTPA scan report description to extract a single label of "RV strain biomarker" and label the scans as either RV strain positive or RV strain negative. A scan was marked positive if the radiologist specifically indicated that RV strain was present. Selecting one high-level (or series-based) label for the whole 3D scan, with no additional markings or segmentation maps, enforces this computer-aided diagnosis (CAD) task to a weakly supervised solution and the processing of the scans in a 3D manner. Our dataset included 363 CTPAs, 86 of whom (23.91%) were labeled with RV strain. The model was trained and validated on years 2012-2017 data and tested on held-out year 2018 data. Table 1 describes the dataset partitioning.

Evaluation methods
The model's performance evaluation on the test set included AUC, sensitivity (also known as recall), specificity, accuracy, positive predictive value (PPV, also known as precision), and negative predictive value (NPV). The predicted probability threshold for returning a positive finding was determined by Youden's index [30], which finds the model's optimal joined sensitivity and specificity. To measure the variability in these estimates, we calculated 95% DeLong CIs [31] for the AUC of the model.

Image Preprocessing
Contrast-enhanced chest CT series were used as the dataset. Scans were extracted from Digital Imaging and Communications in Medicine (DICOM) format. Each extracted slice was scaled to 128 x 128 pixels. The entire series of 128 slices was saved as a 128 x 128 x 128 array. The scans are fed to the network in a 3D manner (as a whole), which takes advantage of the scan's contextual context and allows for easy end-to-end training. An important task is to focus attention to a volume of interest (VOI) that includes the heart and lung regions to improve the developed system's performance. This enables us to choose a smaller volume to be fed into our network, which is especially important due to the computational costs of 3D processing. We chose the VOI according to the lung area. The lungs can be located in a chest CT image by using Hounsfield unit (HU) values. The heart area is roughly the area between the lungs.

Training and Inference
To evaluate the compared networks fairly, we used the same experimental conditions. We used a batch size of 4 and a constant learning rate of 5e-3. Furthermore, we used Adam optimizer, weight decay of 1e-5, and binary cross-entropy loss L bce : Where Y pred ∈ (0.0, 1.0) represents the the predicted probability, and Y ∈ (0, 1) is the binary ground truth label. We used data augmentations of flip and rotation. Our code is implemented in Pytorch. The network was trained on an NVIDIA 1080Ti GPU.

Model Performance
We evaluated the performance of SANet and MLANet. The comparison between the two model architectures is described in Table 2. As can be seen from the results, adding our self-attention blocks improves performance compared to both approaches' baseline network. In addition, the stacked attention block solution -SANet, outperforms MLANet for all baseline networks. For both models, the DenseNet121-3D baseline attains the best results. SANet with the baseline of 3D-DenseNet-121 achieved the best performance, with an AUC of 0.88 (95% CI: 0.8-0.94) for classifying RV strain. 84.7% of the test cases are correctly predicted according to the report label. Using Youden's index, the model showed a sensitivity of 87% and specificity of 83.7% for predicting RV strain. The solution's precision was 71.4%, and NPV was 93.2%. The SANet DenseNet121-3D baseline outperforms the next best SANet model with a margin of 0.045 AUC and the MLANet DenseNet backbone with a gain of 0.064 AUC.

Interpretation of the Model Prediction
In order to obtain model interpretability, we identified locations and slices in the scan that contributed most to the classification using 3D gradient-weighted class activation mapping (Grad-CAMs) [32]. We expanded the implementation from a 2D slice-based tool to a 3D one. Fig 4 shows the heatmap visualization results obtained from our network. The brightest areas of the heatmap are regions that influence the model prediction the most. We note that the attention gate focuses on the heart area rather than other scan regions, as expected.   Table 3 examines the result of adding our proposed attention blocks to common baseline architectures in a stacked approach. As shown in the table, adding our proposed attention blocks consistently improves performance with a maximum gain of 0.075 in AUC. All improvements are statistically significant except for the ResNet3D-50 (residual block) baseline. Our proposed solution is constructed using a backbone of densely connected blocks, as in the DenseNet3D-121 baseline, which achieves the best results. It outperforms all other models with a margin of 0.045 AUC.

Comparison with state-of-the-art 3D CNN models
We compare SANet to several common 3D CNN architectures as detailed in Table 4. This includes the current state-of-the-art model architecture for Kinetics-600 dataset [36], ResNeXt3D-101, and memory-efficient  DenseNet3D-121, and the classic ResNet3D-50 in both its basic and residual block version. In addition, we compare our network to two promising recently published 3D classification networks as well: Med3D [37], a ResNet3D-50 network-based that is pretrained using many medical image datasets and uses transfer learning to create a robust network for medical domain 3D classification tasks and Models Genesis [38], a U-Net architecture based solution that is pretrained using a pipeline of self-supervision tasks. Our model outperforms all compared models with a wide margin of 0.068 in AUC. P-Value with compare to SANet shows that all results are statistically significant.

Ablation Study On Multiple Attention Blocks
To further analyze the system, we explore altering the number of attention blocks fused into our model. This enables a comparison to the baseline network version which has the same structure but has zero attention blocks. Table 5 presents the results of this experiment. The entries in the table specify the best performance for each number of added attention blocks, where the number in the brackets list the exact blocks that where added: Adding the described attention blocks shows improvement in result metrics. It is evident that four attention blocks perform better than the networks having one or two blocks. This result follows intuition and can be explained by the fact that multiple attention blocks tend to obtain the correct attention and can even compensate over the wrong feature capture one of the attention blocks. Although networks with three and four blocks attain almost the same AUC, all other metrics are improved when adding a fourth block. In addition, when integrating an attention block after the first layer improves the AUC most. Looking at the results, it is only expected that we add more blocks until saturation or degradation in performance. Yet, we are limited by our baseline's depth, DenseNet3D-121, which includes only four basic units. The numbers in the brackets indicate the attention blocks that were added to the DenseNet3D-121 baseline. For example: [1,3] means that two attention blocks were added after the first and third layers.

CONCLUSIONS
In this work, we proposed generalized self-attention blocks that can be incorporated into existing classification architectures. The use of residual networks allows increasing the network's depth while keeping it stable for training, improving performance significantly [35]. Dense connectivity ensures maximum information paths between layers by connecting all layers [34]. The output features of all convolution layers in the dense blocks are concatenated along the channel axis. Attention blocks automatically learn to focus on target structures without additional supervision. They do not introduce significant computational overhead and do not require a large number of model parameters, as in the case of multi-model or cascaded frameworks. In return, the proposed attention blocks improve the model's sensitivity and accuracy for label predictions by suppressing feature activations in irrelevant regions. In this way, the necessity of using an additional external organ localization model can be eliminated while maintaining the high prediction accuracy. All of these motivated us to use these mechanisms for our proposed network.
We present two network variations that utilize our proposed blocks to classify RV strain on CTPA scans: MLANet -The multi-layer attention block architecture in which the attention blocks are aggregated from different layers of the network for prediction and SANet -the stacked attention block architecture where the attention block is incorporated into the network as additional network layers. Our results show that attention consistently improves performance throughout multiple baseline architectures. The stacked attention block model with a baseline of 3D-DenseNet-121 outperforms the multi-layer architecture with an additional AUC margin of 0.064. This network also outperforms current state-of-the-art 3D CNN models: ResNeXt3D-101, ResNet3D-50 and DenseNet3D-121 for the same task. We conclude that the automatic deep learning solution can classify RV strain effectively from 3D CTPA scans.
We selected a single binary label from the radiologist report. This means that readings were done in an acute scenario rather than a prospective research environment. On the positive side, if our system was able to cope with the provided settings it may provide a more robust solution that can be translatable in real-world settings.
Beyond RV strain's specific task, this approach is also applicable to other abnormality classification problems with minimal preprocessing and a single binary scan-level label of the 3D volumetric data. We plan to test these generalization abilities in our future work and specifically use the system proposed here for the broader PE risk stratification task. Such tools may support improved healthcare. For example, improving the ability to direct preventative and health surveillance resources. To the best of our knowledge, there is no prior deep learning-based solution for fully automated classification of RV strain using CTPA, and this is the first work where medical images are used in such an architecture.