{'fraction_non_alphanumeric': 0.029991724452815113, 'fraction_numerical': 0.015238421368033559, 'mean_word_length': 4.731763166503107, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Learning applied to Intracranial Hemorrhage Detection

: Intracranial hemorrhage is a serious medical problem that requires rapid and often intensive medical care. Identifying the location and type of any hemorrhage present is a critical step in the treatment of the patient. Diagnosis requires an urgent procedure, and the detection of hemorrhage is a difﬁcult and time-consuming process for human experts. In this paper, we propose methods based on EfﬁcientDet’s deep-learning technology that can be applied to the diagnosis of hemorrhages and thus become a decision-support system. Our proposal is two-fold. On the one hand, the proposed technique classiﬁes slices of computed tomography scans for the presence hemorrhage or its lack, achieving 92.7% accuracy and 0.978 ROC-AUC. On the other hand, our methodology provides visual explanations of the classiﬁcation chosen using the Grad-CAM methodology. This paper presents a model based on deep learning techniques that can be useful for classifying ICH in patients from computer-tomography (CT) image slices. Our methodology also includes visual explanations to understand model decision making. We begin


Introduction
Spontaneous intracranial hemorrhage (ICH) occurs when a diseased blood vessel within the brain bursts, allowing blood to leak inside the brain. Several factors, like disruption of blood-brain barrier and leakage of fluids and proteins, inflammatory cascade or sudden increase in intracranial pressure after bleeding develop into a brain injury in the areas surrounding the blood spill. Nearly 66 % of all deaths caused by neurological diseases worldwide are related to hemorrhagic stroke [1]. According to [2], the overall incidence of spontaneous ICH worldwide is 24.6 per 100,000 people each year and approximately half of ICH related mortality occurs within the first 24 hours [3].
The severity and outcome of an ICH depend on its cause, location in the brain, size of the bleed, the amount of time that passes between the bleed and treatment, etc. ICH usually occurs in selected parts of the brain, including the basal ganglia, cerebellum, brain stem, or cortex. Damage can be severe and result in physical, mental, and task-based disability. Various types of ICH strike people of all ages. Although ICH is usually associated with older adults, it can also occur in the younger population. If not treated correctly and immediately, an ICH can cause disability or death. As a heterogeneous disease, image characteristics help to determine the cause, the prognosis and how to treat the disease.
This paper presents a model based on deep learning techniques that can be useful for classifying ICH in patients from computer-tomography (CT) image slices. Our methodology also includes visual explanations to understand model decision making. We begin with a short introduction explaining what intracranial bleeding is and present the latest where µ water and µ air are respectively the linear attenuation coefficients of water and air. To understand brain bleeds, it is important to have a basic understanding of the different types based on the location where the bleed occurs (for a detailed introduction to ICH, see, e.g., [8] or [9]). After a stroke, the cause (blood or blood clot) must be determined to initiate appropriate treatment. Short-term medical treatment can help reduce brain damage and improve the likelihood of recovery. Survival and recovery from intracranial haemorrhage are related to the site, mass effect, and intracranial pressure of the underlying hematoma, and by subsequent cerebral edema from perihematomal neurotoxicity or inflammation and complications from long-term neurological dysfunction.
Surgery may be necessary since bleeding (hemorrhage) may require immediate decompression of the brain to release pooled blood and relieve pressure. A craniectomy incision (partial removal of the skull to allow the swelling brain to expand), or a craniotomy (opening of the skull cavity) are practiced. Depending on the type, location, and extent of the brain bleed, many patients do not survive the initial bleeding event. Mortality rate is as high as 50% in the first 30 days [10].
Research indicates several therapeutic avenues of benefit in intracerebral hemorrhage modelled around an active management plan, aggressive supportive care, early blood pressure control, and targeted surgery as a life-preserving strategy [11], [12]. The time between symptoms, arrival in hospital, and diagnosis of bleeding is crucial. The earlier of brain hemorrhage is confirmed and evaluated, the sooner a treatment plan can be made.
The method of class activation (CAM) has already been used to visually explain medical images. In addition, ResNet and EfficientDet have been well described in literature, but the novelty of this paper is the efficient combination of these technologies. From a technical point of view, to the best of our knowledge, it is the first time that ResNet, EfficientDet and Grad-CAM have been used together. Besides, the proposed model is not a simple juxtaposition of techniques as described in the literature, but an integrated model. The combination of these technologies can illuminate future research in this field and be a starting point for other researchers.
Despite the widespread use of deep learning in medical image classification, this paper makes two folded contributions to this field. On the one hand, the accuracy achieved is competitive with the AI techniques most commonly used in ICH classification. This represents an advance over many papers found in the literature. On the other hand, many deep learning tools are black boxes providing classification without explanation. In this paper, we propose to use a visual explanation technology to provide information on the key areas of CT. This additional information complements other tools of deep learning and makes our proposal a useful tool for clinicians.
The paper is organized as follows. In Section 2, some current approaches to the problem of the detection of ICH by deep learning and Grad-CAM techniques are recalled. Section 3, Materials and Methods, is devoted to explaining our proposal. In Section 4, the results obtained are presented. The paper finishes with the discussion of the results (Section 5) and some conclusions (Section 6). Some appendices have also been added with technical details.

Deep Learning for the Detection of Intracranial Hemorrhage
Deep Learning [13] is a series of models, technologies, and architectures based on Artificial Neural Networks which have undergone a revolution in AI in the last years. Its doubtless success in real world problems reaches areas such as face recognition [14], music composition [15] o multilingual translation [16]. In the last decade, thousands of researchers have proposed new models that overtake the achievements of previous architectures, so it is impossible to give a general framework that covers all possible approaches in deep learning. Nevertheless, in general, deep learning models can be described as mathematical functions that generate output from a given input. For example, some models provide the names of people when faces are given, or sentences are translated into different languages from a given one.
From a technical point of view, such mathematical function is obtained by composition of typically thousands, or millions of simple mathematical functions called neurons due to their original biological inspiration. These sets of neurons are the basis of the neural network and the links among the neurons determine the architecture. The basic architecture corresponds to the so-called multilayer perceptron (MLP, see, e.g., Figure 1 (top)), where the neurons are arranged in layers. Each neuron is connected to all neurons in the following layer and there is no connection between neurons of the same layer. As we will see below, from this basic architecture more and more efficient ones can be built. The main feature of such a neural network is that its parameters are self-adjusted by trying to minimize a loss function on a training dataset in a process known as learning. After the training process, the neural network with the parameters obtained can be used to solve problems that were unapproachable some years ago.
Deep neural networks extract many interpretive patterns and characteristics from data and learn very complex and significant representations. The extraction and discovery of these characteristics or patterns is attributed to the depth of the neuron network, as they are more likely to be found in the later layers of the network. The intuition behind this is that these layers gradually learn more complex features. For example, in the case of image recognition, the first layer may learn to detect edges, the second layer may learn to identify textures, and similarly the third layer can learn to detect objects, etc. (see Figure 1 (bottom)). As the nature of problems input to the neural network becomes increasingly difficult, researchers begin to use more and more advanced models to achieve better results, as we will see below.
One of the most interesting areas of deep learning research is digital image processing, especially medical image processing. Among many other applications, we can cite disease classification [19], ROI segmentation [20] or medical object detection [21]. In this paper, we apply Deep Learning techniques to the study of ICH detection.
Recently, some studies have been published on this topic, and many researchers have started to pay attention to it. Among many others, we can cite [22], where the authors use a fully convolutional neural network for classification and segmentation with examination of ICH with computed tomographies; [23] where the InceptionV3 and DenseNet Deep Learning models for dealing with CT; or [24], where the authors combine convolutional neural networks with other machine learning techniques to deal with ICH detection. In [25], the authors use a Dense U-net architecture for the detection of ICH. In [26] A novel deep learning technique based on the Monte Carlo algorithm [27] was applied.
The paper by Voter et al. [28], published in April 2021, deserves special attention. In this paper, the authors have the objective of determining the diagnostic precision of an AI decision support system (DSS), developed by Aidoc [29], in diagnosing intracranial hemorrhage (ICH) in non-contrast head CT and to assess the potential generalizability of an AI DSS. This retrospective study included 3605 consecutive, emergent, non-contrast adult head CT scans.
Each scan was assessed for the ICH by a certified neuroradiologist and Aidoc. The authors determined the diagnostic accuracy of the AI model and performed a failure mode analysis with quantitative CT radiomic image characterization. The results were as follows: Of the 3605 scans, 349 cases of ICH (9.7% of studies) were identified. The neuroradiologist and Aidoc interpretations were consistent in 96.9% of the cases and overall sensitivity, specificity, positive predictive value, and negative predictive value were 92.3%, 97.7%, 81.3% and 99.2%, respectively.
Aidoc was wrong in 101 of 3605 studies, which corresponds to an accuracy of 97.2%. This study revealed decreased diagnostic accuracy of an AI DSS at their institution. Despite its extensive evaluation, they were unable to identify the source of this discrepancy, raising concerns about the generalizability of these tools. These results further highlight the need for a standardized study design to allow a rigorous and reproducible site-to-site comparison of emerging deep learning technologies.

Materials and Methods
The purpose of this study is two-fold. On the one hand, we present a deep learning architecture that, in some sense, improves the SOTA methods for classifying ICH. On the other hand, our method includes visual explanations of the decisions. Technically, our proposal combines two of the most successful Deep Learning architectures with a technique for visual explanations of AI models. Next, we provide a brief introduction to them.

Deep learning architectures
The first basic architecture used in our model is ResNet. The second is EfficientDet [7]. The main motivation for this architecture is to optimize the resources of the deep learning model for the detection of digital image objects. Normally, the big success of deep learning models is obtained only by expensive large computers that can be used by large companies. Currently, many efforts are trying to adapt these models to simpler (and cheap) hardware without losing accuracy. EfficientDet is a clear example of this research line. The main contribution of EfficientDet is two-fold. On the one hand, it uses a novel computing unit as the base of the model, the so-called weighted bidirectional feature pyramid network (BiFPN) and, on the other hand, the authors propose a compound scaling method for scaling up several features of the model (resolution, width, and depth), which leads to a new family of object detectors called EfficientDet family, from D0 (the smallest one) and D7 (the largest one).
In our study, we propose a neural network architecture that combines the EfficientDet [7] and ResNet [6] architectures. Technical details of ResNet and EfficientDet architectures can be found in Appendix A.1.

Dataset
Kaggle 1 is an online community of data scientists and machine learning practitioners that allows users to find and publish datasets, explore, and build models in a web-based data science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges.
The RSNA Intracranial Hemorrhage Competition [30] was a competition hosted by Kaggle at the end of 2019. This competition provides a high number of annotated data, indicating if there is hemorrhage in the slice and if so, indicating one of five subtypes. The competition was divided into two stages. In this work, we only use the data from the second stage (more than 400 GB).
The dataset contains 752,799 scan slices in Digital Imaging and Communications in Medicine (DICOM) format, from 18,938 patients. Thus, a change of one HU represents a change of 0.1% of the attenuation coefficient of water, since the attenuation coefficient of air is nearly zero. Therefore, the Eq. 1 is the definition of HU units for CT scanners that are calibrated with reference to water and air.
The set of patients was divided into three randomly divided patients' groups: train (90%, 17044 patients), validation (5%, 947 patients) and testing (5%, 947 patients) groups to avoid bias since there are various slices from the same patient. However, it is important to note that there is a remarkable class imbalance since the groups contain the following number of slices: To mitigate this class imbalance, we randomly select the same number of non-ICH slices as the ICH-sliced slices. It turns into 195050 slices from 17044 patients for training, 10802 slices from 947 patients for validation, and 10014 slices from 947 patients for testing.
In addition, since our neural network takes three channels as input, the scans with the HU matrices must be pre-processed to obtain three matrices of size 512 × 512 pixels, one for each channel. To obtain such three matrices, three different windows are applied to every slice. Applying a window with X as lower bound and Y as upper bound means that all HU values lower than X are converted to X, all HU values greater than Y are converted to Y and the rest remains the same (see Figure 2).
We build a test dataset with all the slices of the test patients, that is, 5007 slices with ICH and 32758 without ICH (37765 slices in total). However, in this case we evaluate the model performance annotating every patient with ICH if any slice has ICH and noICH otherwise. Following this annotation criteria, 371 of 947 patients have ICH.
In the Kaggle dataset, the slice thickness was uniform for one patient, but there are different thicknesses between patients.
We added a test group of 55 images of clinical cases of control (eight patients), intracranial hemorrhage (ten patients), and subarachnoid hemorrhage (37 patients) provided by HUVR and HUVM hospitals, where two authors are working. The controls were not healthy volunteers, but patients scanned by CT at admission for who any other type of hemorrhage was excluded. The research project in HUVR was overseen and approved by the Ethics Committee of the Hospital Universitario Virgen del Rocio in Seville, Spain (Cod. CEI2012PI/228). Informed consent was obtained from all patients participating in the study or from their close relatives. The images of the patients at the Hospital Virgen Macarena were obtained after they signed the informed consent form for CT in which they accepted the potential use of the images for the purposes of promoting knowledge and research in relation to the disease from which they suffer. The individual slices were 512 × 512 pixels.
The images were evaluated for diagnosis per patient by experienced radiologists, neurologists and neurocritical doctors. They were not evaluated on slice-by-slice basis, which is not done in clinical practice. The mean age of patients was 54 years, range [21..

Grad-CAM
Grad-CAM uses the feature maps produced by a convolutional layer (one of the types of layer used in deep learning to analyze data structured in grids, as computational images). The authors argue that the last convolutional layers have the best compromise between high-level semantics and detailed spatial information. The output of Grad-CAM is a class-discriminative localization map, i.e., a heatmap where the hot part corresponds to the part of the image that has been relevant for the model to perform the classification. Our model provides two different kinds of information: • Firstly, our model provides as output whether the patient has ICH or not. Technically, the output is the probability of having ICH. The model outputs an affirmative answer if such a probability is greater than a fixed threshold • Also, our model also provides a color map on the input image where the red area corresponds to the pixels in the image which have been determinant in the decision. Our code is available online 2 . More details of the framework can be found in Appendix A.3. Let us recall that the CAM techniques for obtaining visual explanations of Deep Learning decision have been previously used by other researchers (see, e.g., [31] or [32]), but, to the best of our knowledge, it is the first time when ResNet, EfficientDet and Grad-CAM are used together. The combination of such technologies is a contribution of this paper which can shed light to future research in this area.

Metrics
The metrics used to evaluate the experiments are the following: We consider that the model predicts yes (slice with ICH) for a certain image slice if the probability is greater than 0.8 to reduce the false positive predictions. Most binary models assume a cut-off value of 0.5, lower than ours. Table 1 shows the results of the evaluation of our model, called EffClass, with RSNA Kaggle competition test data and the application of different thresholds for the prediction of our model (i.e., the model predicts ICH if the probability returned by the model is higher than the threshold). The chosen thresholds of probability are 0.5, 0.7, and 0.9. The highest values are marked in bold.

Results
We cannot compare our results to those of Section 2 because the authors evaluated the images of the patients rather than the images slice by slice. Therefore, we evaluate the patients considering the following additional criteria: The patient is diagnosed with hemorrhage if the model predicts, at least, n slices of the image with ICH being n a percentage (5% or 10%) of the number of slices since the number of slices per patient varies. We do not check for the continuity of the slices position, just the percentage. For example, given two patients with 30 and 40 as the number of slices, respectively, and a threshold of 10%, the first patient would be classified as ICH patient with ICH if the model predicts at least three (3) ICH slices, but the second patient would be classified as ICH patient with ICH if the model predicts at least four (4) slices with ICH. The results of the experiments are shown in Table 2.
Before comparing our results with the results of reference [28], it is very important to note that we cannot make a fair comparison, since the datasets used in our experiments and the Aidoc experiments are different. In the reference [28] dataset, only the 9.7%, 349 of 3065 patients, were positive (i.e., patients with ICH) while in our test dataset, the 39.2%, 371 of 947 patients were positive. In addition, the results are highly influenced by the types of ICH that comprise the dataset and their proportion.
The study presented in [28] shows high accuracy (ACC) and negative predictive value (NPV). The highest accuracy value obtained from our experiments is 95.5%, with the EfficientClass model with threshold = 5% (overlapping confidence intervals). The study [28] shows the same true negative rate (TNR) as EfficientClass with threshold = 5%. With respect to the true positive rate (TPR), the EfficientClass model with threshold = 5% has a higher value (0.949). Finally, with respect to the predictive positive value (PPV), the highest value is 0.951, and it was obtained with the EfficientClass model with threshold = 10%.
As stated in Section 2, a standard study design is needed to enable a rigorous and reproducible site-to-site comparison of new deep learning technologies. Our goal is to classify patients by their CT images if they have ICH or not. This means that we estimate a binary probability distribution in which each image obtains the probability P that the patient has ICH and the probability 1 − P that the patient does not have ICH. To this end, the original Kaggle dataset was pre-processed to train our model. In addition to many other technical details, preprocessing changes the classification label of all input data from the original label to a binary label marking if the image shows if the patient has ICH or not. Bearing in mind that the target of the models presented to the Kaggle competition is to associate to each CT image a label with the type of ICH, not to classify if the patient has ICH or not, a comparison in the accuracy of our model and the ones presented to the competition is not possible.
A typical detected minimum size (width) by Aidoc software is shown in Fig. 2 A of [28] and is about 2 mm. In clinical practice there are rarely less than 25 slices per brain scan. We did not check or train the model for the size of the hemorrhage. Let us note that the model was trained on slices with yes/no for all the slices, and we did not train the model for the location of the hemorrhage. The Grad-CAM map should not be mistaken for the location of the hemorrhage. It can be interpreted as the usual location of the hemorrhages and the pixels considered in the evaluation of the presence of hemorrhages for this slice.

Test results obtained with clinical data
Despite the results that we obtained in the previous section being very promising, we needed to test our models with external data to validate that our models could be used not only with Kaggle data but also with external data.    We applied GRAD-CAM and obtained a heat map with focus in the ICH area. For the Grad-CAM implementation, we used the pytorch-grad-cam library [33]. In addition, in our Grad-CAM experiments, we used enhancement smoothing, and according to the library documentation, it applies a combination of horizontal flips and image multiplications by [0.9, 1.0, 1.1], having the effect of better centering the CAM around the objects.
In Figure 3 (a) a slice with ICH is shown. It is quite difficult to see the ICH, but the EfficientClassification neural network predicts ICH with a probability of 0.9897. Figure  3 (b) and Figure 3 (c) show the next consecutive two slices where ICH is clearly seen and the neural network predicts ICH with a probability of 0.9972 and 0.9997, respectively. Figure 3 (d) shows a slice with ICH, where the hemorrhage is not detected according to our threshold (0.8), and the probability was 0.7805. Figure 3 (e) shows a positive slice where the probability was just above the threshold, 0.8101. On the other hand, in Figure 3 (f) a slice without ICH is shown.
The results of the classification of additional test 55 patients are provided in the last row or Table 2. All patients with ICH (47) were identified correctly. All control patients were identified as negative (8) There were no false positives diagnosed in control patients. NPV and PPV are 100% because there was no false negatives nor false positives.

Discussion
The irruption of deep learning as a new set of AI techniques represents a major challenge in medical applications. This type of technology can be very useful for those centers, especially small community hospitals, where there are no radiologists on call to interpret these images and depend on non specialized physicians.
In many fields, radiologist expertise is the key to obtaining a correct diagnosis. In this case, the use of the latest technology for computer image classification in medical images is a major contribution to human experts.
In fact, specific diagnostic support software is already used in ischemic stroke in a wide range of hospitals. In this way, this paper presents EffClass, a deep learning model  that combines two of the most successful models for image classification and we show that our model is competitive with some of the current most widely used commercial, CE certified and FDA approved methods such as Aidoc [29] or e-ASPECTS [34]. The efficiency of manual SAH diagnosis can be 98.9% [35], but at the expense of time. Radiologist usually needs from 15 to 30 minutes to diagnose a patient and inform about it in writing. Diagnosis of ICH has been a subject of several papers and is a field of intensive research using traditional techniques [8] and deep learning [22][23][24][25]. The solution [36] has AUC 0.846 (CI 0.837-0.856). Another is BRAINSCAN.AI [37] which has AUC/ROC of 0.976 for hemorrhages. It is a CE-certified solution and is now installed in more than 40 hospitals in Poland [38].
However, the use of neural network methods in medicine as a support in decision processes has an important drawback. They are usually considered as black boxes, where the high accuracy is not considered due to the lack of explanations. Human experts want to know not only the decision of the AI system but also the motivation of such a decision. In such a way, our model not only provides high accuracy on the decision, but it also provides visual explanation with an intuitive color map on the original image. In such a way, the Grad-CAM method has been used to highlight the area of the input image that has been relevant in the decision. This mixture of high accuracy together with the visual explanation makes the proposed model one of the most competitive among the current ones.
The limitation, as of today, is that the method is for research use only and the command line interface with PyTorch is comfortable only to IT specialists. It requires the extraction of the DICOM hospital image from the hospital system to a desktop computer. Our method has some higher metrics (ACC, TPR and PPV) than Aidoc FDA submission, bearing in mind that the amplitude of the confidence intervals of Aidoc FDA submission is wide. Future research lines can also be considered. From the applications point of view, the deep learning technologies proposed in this paper can be also applied to obtain classification and visual explanation of other medical decisions based on medical images. From a technical point of view, deep learning is continuously evolving, and other recent technologies on classification and visual explanation deserve to be explored, including 3D approaches.

Conclusions
We have developed a model that can detect bleeding in CT scans of a patient in DICOM 2D image files. This model has a higher AUC than the Aidoc one available commercially in a version submitted for approval by the FDA.
The model has several adjustable parameters as threshold to classify as a slice with hemorrhage and percentage of the slices of the image of the patient to classify him/her as diagnosed with hemorrhage.The tool is Research Use Only for the moment.
Hemorrhages were divided into five classes in Kaggle competition. In clinical practice the division is sometimes only to intracranial and subarachnoid. Subarachnoid ones are sometimes difficult to diagnose per slice, what can be seen in Figure 3 (d, e) for SAH. The concept of "time is brain", used for ischemic stroke, could be also applied in this devastating disease, so automatic diagnosis which takes seconds right after CT scan image reconstruction can prioritize the images for evaluation by a radiologist or help in complicated or debatable cases. Figure A1. Residual learning: a building block. Source: [6].
result of the similar part, both networks should return the same result. However, as the authors stated in the paper, it is not easy to learn the identity function stacking nonlinear layers. With the skip connection, learning the identity function turns out to be easier, allowing the networks to learn more than the plain networks.
In Figure A2, VGG-19 [39], a plain 34-layers network and a ResNet34 neural network are shown. From a technical point of view, our implementation modifies the original architecture of EfficientDet, by means of a substitution of the final layers (the so-called classification and detection heads) by a ResNet18 with two units as outputs, representing the probability of ICH.
However, we cannot simply remove the heads and put a fully connected layer instead, since the BiFPN layer provides five sets of feature maps (the five units with colors shown in Figure A3 just before the two heads), each one at different scale: 64 × 64, 32 × 32, 16 × 16, 8 × 8 and 4 × 4, consisting of each set of 64 filters. It is possible to flatten the features up and put a fully connected layer with two neurons as output (i.e., create a list with the outputs of the Bi-FPN layer and connect them to two units, but this would probably not be enough processing for all these feature maps. Our approach is to upsample every feature map to 224 × 224, since the ResNet family receives an image with a resolution of 224 × 224 as input) and passes it to a ResNet18, but we would have two problems: • Upsampling approaches such as upsampling with nearest interpolation lead to the same problem since we are not adding any information.

•
Even without adding any information, ResNet would learn from these features but, as ResNet receives 3 × 224 × 224 (channels, height, width) as input, we would need an intermediate step: reduce 320 (5 sets of 64 filters each one) channels to only 3.
We solve these problems with two intermediate groups of layers between the Bi-FPN and the ResNet: transposed convolution layers as upsampling method and convolution layers to reduce the number of channels. We call this neural network EfficientClassification. The implementation consists of the following steps: 1.
Load EfficientDet. We get a pretrained EfficientDet-D0. In this step, we use the implementation from the timm library [40]. The timm library can be installed using Pypi.

2.
Cut the regression and classification heads. We remove the regression and classification heads and only use the EfficientNet backbone and the BiFPN layers. Figure A4 shows this step and the previous one.

3.
Deconvolution. Different deconvolution operations (transposed convolutions) are applied to the feature maps to get feature maps of 244 × 224 as dimension. 42 feature maps are passed to the convolution step. The number of channels returned by every deconvolution differs depending on the resolution of the feature maps, as can be seen in Figure A5.

4.
Convolution. In this step, the number of channels is reduced to 3. Two convolution layers are applied as shown in Figure A6.

5.
ResNet. We call a pretrained ResNet18 from torchvision.models and cut the last layer as we did in the ResNet implementation section. The convolution layers output is passed to the ResNet as input.
The previous steps result in the neural net architecture shown in the Figure A7.  Grad-CAM is applied to a trained neural network. Then, the neural network is fed with an image to calculate the Grad-CAM heatmap for that image for a chosen class of interest. Grad-CAM has three steps: compute gradients, calculate alpha values by averaging gradients, and calculate the final Grad-CAM heatmap.
Step 1. Compute the gradient of y c with respect to the feature map activations A k of a convolutional layer, i.e., ∂y c ∂A k , where y c is the raw output of the neural network for class c, before the softmax is applied to transform the raw score into a probability.
Step 2. Apply global average pooling to the gradients over the width dimension (indexed by i) and the height dimension (indexed by j) to obtain the neuron importance weights a c k , producing k weights: Step 3. Perform a weighted combination of the feature map activations A k with the weights, a c k , calculated in the previous step: Typically, a resize operation is performed afterwards, since the Grad-CAM heatmap size is typically smaller than the original image size.

Appendix A.3
Since we needed to conduct many experiments due to both the problem complexity and the network architecture we used, we created a framework that controls the experiment pipeline.
This framework is built on top of the pytorch library [41] and consists of the following: • Configuration directory. This directory contains the YAML files that govern the experiments. There are three types of configuration files corresponding to three different possible phases: training, validation, and testing. • Experiments directory. If this directory does not exist, it is created automatically. It contains directories with the experiment code and, inside of them, there are loss and accuracy graphs, the configuration file used in the experiment, and the metrics (ROC-AUC, confusion matrix, and so on). • Scripts directory. This directory contains the different scripts that were used in the different phases. The most important one is the config.py file, which leads the execution and loads the configuration from the YAML config file.
Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 31 January 2022 doi:10.20944/preprints202112.0150.v2 • Main file. The main.py file just receives YAML config files and calls the config.py and the corresponding task script, allowing us to execute multiple experiments. Figure A8 shows the framework repository tree in the experiment called R34_001. Figure A8. Framework repository tree.