{'fraction_non_alphanumeric': 0.029994585814834867, 'fraction_numerical': 0.01245262587980509, 'mean_word_length': 4.801507537688442, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards the Localisation of Lesions in Diabetic Retinopathy

Convolutional Neural Networks (CNN) has successfully been used to classify diabetic retinopathy (DR) fundus images in recent times. However, deeper representations in CNN only capture higher-level semantics at the expense of losing spatial information. To make predictions very usable for ophthalmologists, we use a post-attention technique called Gradient-weighted Class Activation Mapping (Grad-CAM) on the penultimate layer of deep learning models to produce coarse localisation maps on DR fundus images. This is to help identify discriminative regions in the images, consequently providing enough evidence for ophthalmologists to make a diagnosis and saving lives by early diagnosis. Specifically, this study uses pre-trained weights from four (4) state-of-the-art deep learning models to produce and compare the localisation maps of DR fundus images. The models used include VGG16, ResNet50, InceptionV3, and InceptionResNetV2. We find that InceptionV3 achieves the best performance with a test classification accuracy of 96.07% and localise lesions better and faster than the other models.


Introduction
There has been a tremendous success in the field of deep learning, especially in the computer vision domain. This success may be attributed to the invention of Convolutional Neural Networks (CNN). It has been extended even to specialised fields such as medicine [8]. With the help of transfer learning [14], weights or pretrained model of several state-of-the-art models can be used in medical images. In this study, for example, we use deep learning backed with transferred weights of pre-trained models to classify Diabetic Retinopathy (DR) fundus images [5,8]. Even though, these models are able to achieve high performance, it is difficult to understand the reasoning behind its discrimination process. This is due to the fact that, deep learning models have a nonlinear multilayer structure [4].
Of recent, DR classification has received a lot of attention as it has been useful in the ophthalmology domain [5,8]. However, ophthalmologists may not be able to evaluate the true performance of the model [4,2]. We, therefore, attempt to mitigate this issue by providing additional evidence of model performance [4] with the help of a post-attention technique called Gradient-weighted arXiv:2012.11432v1 [cs.CV] 21 Dec 2020 Class Activation Mapping (Grad-CAM) [10]. This technique can be seen as a Computer-Aided Diagnosis (CAD) tool integrated to increase the speed of diabetic retinopathy diagnosis.
The main contribution of this paper is to use Grad-CAM to generate coarse localisation maps from higher-level semantics of the model, consequently aiding and speeding up the diagnosis process of diabetic retinopathy.

Data and Methodology
In this study, we used a publicly labelled DR fundus data -Asia Pacific Tele-Ophthalmology Society (APTOS) 3 -to train, validate and test the model. In detail, it has 3662 DR fundus images. These images are categorised into a five-class scale of increasing severity namely normal, moderate, mild, severe and proliferative [5,1]. Figure 1 shows the distribution of classes for the data. The plot  Figure 1 shows that the data is imbalanced as the normal class has the highest number of instances. We resolve the class imbalance problem by incorporating weights of the classes into the cost function. In this case, the majority class was given a small weight while the minority classes were given bigger weights. Intuitively, the model places more emphasis on the minority classes and give higher penalties to misclassifications made by the minority classes and vice versa [3,7].
For image preprocessing, we used both vertical and horizontal flips as well as standard normalisation of the data. As we randomly zoomed in some of the data, we also resized all of the images to either size 224 × 224 or 299 × 299 depending on the pre-trained model we want to use. A very important preprocessing technique used in this study was Contrast Limited Adaptive Histogram Equalization (CLAHE) [16] and this is because DR fundus images suffer from contrast issues [9]. Figure 2 shows a comparison of the distribution of DR fundus image before and after CLAHE. Fig. 2: Depicting DR fundus image and it's corresponding distribution before and after CLAHE.
It can be seen that the DR fundus image before CLAHE has more Gaussian (white) noise [9] making it fairly difficult to study. On the other hand, the Gaussian noise is reduced after adjusting the DR fundus image with CLAHE and its corresponding distribution is more normalised. This step is crucial because it affects the sensitivity and specificity of the model [9].
After preprocessing the images, we pass the preprocessed images to the model which in this case has a CNN backbone. We considered and compare four (4) CNN backbones in this study namely VGG16 [11], ResNet50 [6], InceptionV3 [12] and InceptionResNetV2 [13]. The idea is to pass preprocessed images through a pre-trained model and extract its feature maps at the last layer for classification. It should be noted that as images pass through a deep convolutional model, there exist trade-offs of losing spatial information and learning higher-level semantics, especially in the last convolutional layers. Neurons in this layer look for semantic class-specific information in the image and this is usually used for discriminative purposes. Grad-CAM thus helps to interpret and explain individual components of a model [15,10].
Grad-CAM is a technique used to produce visual explanations from decisions made by CNN. Specifically, it uses gradients of a class concept in the penultimate layer to produce a coarse localisation map which helps to identify discriminative regions in an image. For every class c, there are k feature maps. To generate localisation maps, we first compute neuron importance weight α c k by globalaverage-pooling gradients of the target y c with respect to the feature maps A k . It is important to note that A k is a spatial feature map hence it has width and height dimensions indexed by i and j [10]. Neuron importance weight α c k is given by; Finally, we pass a weighted-linear combination of neuron importance weight α c k and feature map A k to ReLU to generate the coarse localisation map. The generated map can then be overlaid on top of the input image to identify the region(s) of interest. Grad-CAM is given by; Aside from using the feature maps to generate localisation maps, it is also used for discriminative purposes. In this study, we replace the final layer with three (3) successive layers -starting with Global Average Pooling (GAP) layer, followed by a Dropout (with 50% probability) layer and finally a Dense layer. In summary, we feed a CNN model with a preprocessed and augmented DR fundus image. We then generate localisation maps using feature maps extracted from the CNN model. In addition, the feature maps are used to classify the DR fundus images in an image-level manner. Figure 3 shows a summary of the study. In this study, we train the models using Keras with TensorFlow backend and on NVIDIA's Tesla V100 GPU for 15 epochs. Besides, we use the sigmoid activation function for the last layer because we use a multi-label classification. For this reason, we used binary-cross entropy as the cost function for the model.

Results
In this section, we evaluate the models classification performance and their ability of generating localization maps for the DR fundus images. In this context, we use accuracy and Area Under Curve (AUC) as performance metric. For every image, we first predict it's class and then generate the localisation map over the image. Table 1 shows the performance of the classification task. We observed that, InceptionV3 model performed better than the other models as it had the highest accuracy of 96.05%. This is followed by VGG16, InceptionResNetV2 and ResNet50 in that order, however their performance are close to each other. Finally, we randomly select some input images (see Figure 4a) for demonstrative purposes. We pass the selected images through the Inception model (since it had the highest accuracy) to generate localisation maps and overlay the generated maps on the input images (see Figure 4). We observe that the model generates a blurry image in Figure 4b. The blurry effect can be attributed to the loss of spatial information as images passes through deep convolutional models in general. The blurry image in this case is the Grad-CAM. It may not be as useful until we map it on the input images. We see that the Grad-CAM has highlighted regions of interest on the input images after mapping (see Figure  4c).

Conclusions
In this work, we presented a technique which identifies regions of interest in DR fundus images and produces visual explanations for models. This aid ophthalmologist understand the reasoning behind the discriminative process and speed up diagnosis. Also, we observe high performance for classification as well as have precise localisation maps. We observe that the models performance are very close to each other. Hence, future work includes finding which model works best for DR fundus images.