{'fraction_non_alphanumeric': 0.024416363246947955, 'fraction_numerical': 0.007853216249018347, 'mean_word_length': 4.8437630371297455, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Introducing and assessing the explainable AI (XAI) method: SIDU

Explainable Artificial Intelligence (XAI) has in recent years become a well-suited framework to generate human understandable explanations of black box models. In this paper, we present a novel XAI visual explanation algorithm denoted SIDU that can effectively localize entire object regions responsible for prediction in a full extend. We analyze its robustness and effectiveness through various computational and human subject experiments. In particular, we assess the SIDU algorithm using three different types of evaluations (Application, Human and Functionally-Grounded) to demonstrate its superior performance. The robustness of SIDU is further studied in presence of adversarial attack on black box models to better understand its performance.


Introduction
In recent years deep neural networks (DNN) have resulted in ground-breaking performance in solving many complex and long-running problems of artificial intelligence (AI). In particular, employing DNN architectures in tasks such as object detection [1], image classification [2] and medical imaging [3] received great attention within the AI research field. As a result, it is no longer surprising to observe that DNNs have become a favoring solution for any applications involving big data analysis. As human dependency on these solutions increases on a daily basis, it is crucial from a both research and business standpoints to understand the underlying processes of DNNs that output a certain decision [4,5]. However, with the complex inner workings of the DNN networks, it is very challenging to understand which features are the major contributors to the accuracy of the output; resulting in "black box" predictors. The interpretation ability of the black box DNN provides transparent explanation and audit model output that is crucial for sensitive domains such as medical or risk analysis [6,7,8]. Consequently, a new paradigm addressing explainability of these models has emerged in AI research namely Explainable AI (XAI) [9]. XAI attempts to provide further insight into the black box models and their internal interactions that enable humans to understand a machine-generated output.
Furthermore, for end-users in sensitive domains, XAI gives the ability to interpret model features at the group level or instance level of the input which results in gaining greater trust for validating the outcome of deployed AI models. Although, there is no standard consensus in the literature regarding how to define a human-interpretable explanation method for the black box model, a widelyadopted and popular approach is to form a visual saliency map of input data showing which parts of the input have influence on the final prediction. This is motivated by the fact that the visual explanation methods can align closely with human intuition. For instance, it is more straightforward to the end-user in the medical domain to evaluate and compare the visual saliency map on a medical image produced by a DNNs model with those generated by actual clinicians. A number of visual explanation algorithms has been proposed among which methods such as LIME [10], GRAD-CAM [11] and RISE [12] are the most used examples of this class. While each of these methods can be justifiable in one way or another, apart from challenges such as gradient computation of DNN architecture (e.g., Grad-CAM) or visualizing all the perturbations modes (e.g., RISE), the generated visual explanation suffers from a lack of localizing the entire salient regions of an object, which is often required for higher classification scores. To address this high-impact problem, we proposed a new visual explanation approach known as SIDU [13] for estimating pixel saliency We briefly showed by both quantitative and qualitative analysis how SIDU can provide greater trust for in the end-user in sensitive domains. The algorithm provides much better localization of the object class in question (see, for example, Fig. 1 (d)). This results in gaining greater trust of human expert level to rely on the deep model.This paper aims at providing a more general framework of the SIDU method by presenting the proposed method in further details and exploring its characteristic via various experimental studies. concretely, we assess SIDU's visual explanation through three main levels of evaluation proposed in [14]. Since the three different categories of evaluation methods have different pros and cons, the superior performance of the SIDU can be well investigated and provide further insight. To the best of our knowledge, our comprehensive experiment studies of these different evaluation levels are the first in the context of XAI. Moreover, the ability of the XAI method to generalize its explanations of the black box in different deployment scenarios can establish further trust.
One example where black box models are subject to less generalization is the presence of adversarial attack especially in sensitive domains [15,16]. Therefore, we investigate how XAI can handle such potential threat and guard against it respectively. our main contributions in this work can be summarized as follows:

Related Work
In this work, we follow three main research directions of XAI: a) visual explanation methods developed to explain the black box model such as deep CNN, b) validity and evaluation of the generated explanation by XAI methods and c) vulnerability of black box explanation method toward adversarial attacks.
We therefore overview the literature on each of this direction in the following subsection.

Visual Explanation
For an end-user, visual explanation methods makes it easier to understand the prediction output of the black box model. One common approach to generate such a visualization is done via saliency maps and such algorithms may be divided into the following four categories: back-propagation based methods, gradient-based methods, perturbation based methods and Approximation based methods. Back-propagation methods: back propagation methods spread a feature signal from an output neuron rearwards through the layers of model to the input in a single pass; making them efficient. Layer wise Relevance Propagation [18] and DeCovNet [19] are examples of this category. Gradient-based methods: methods employ the gradient or its modified version in the backpropagation algorithm to visualize the derivative of the CNN's output w.r.t to its input, e.g. such as Grad-CAM [5]. A better way to produce input images that greatly activate a neuron a neuron was proposed in [20]. They generated classspecific saliency maps by performing a gradient ascent in pixel space to reach a maxima. This synthesized image serves as a class specific visualization and helps understand how a given CNN modeled a class. be generated easier. A good example of this class is the LIME algorithm [10].
Decision trees can also be used for this purpose but cannot be explicitly applied to any visual input [22]. DeepLift [23] evaluates the importance of each input neuron for a particular decision by approximating the instantaneous gradients (of the output with respect to the inputs) with discrete gradients. This obviates the need to train interpretable classifiers for explaining each input-output relation (as in LIME) for every test point. Our proposed SIDU [13] method falls under perturbation-based methods but can effectively localize entire salient region of the object of interest compared to the state-of-the-art XAI methods such as Grad-CAM and RISE. Furthermore, it is less computationally complex.

Evaluation of Explanation Methods
Since it is rather challenging to establish a unique and generalized evaluation metric that can be applied to any task, authors in [14] proposed tree different types of evaluations to measure the effectiveness of explanations. These are presented in the following. 3. Functionally-Grounded evaluation: This method utilizes numeric metrics or proxies such as local fidelity to evaluate explanation in different applications. The main advantage of this evaluation is that it is free from human bias that effectively saves time and resources. Most of the state-of-art methods fall into this category [19,24,21]. For example, the authors in [12] proposed casual metrics insertion and deletion, which are independent of humans to evaluate the faithfulness of the XAI methods.

Adversarial Attacks
In the context of XAI, adversarial attack generators can be divided into white box attacks and black box attacks. The Fast Gradient Sign Method (FGSM) [25] and Projected Gradient Descent (PGD) [26]  of an image while adversarial patch [28] attempts to change the pixel values in a specific region of an image. In general, the ability of changing model's output via small input perturbations makes the XAI explanation methods challenging and less reliable. Thus, to establish greater trust, it is essential for the XAI algorithms not only be effective but also robust against adversarial attack at the same time [29]. Analyzing how the black box explanation (such as proposed SIDU) can effectively handle such potential problem and help the end user to guard against possible disastrous outcome of classifier when adversarial attack is presented.

SIDU: Proposed Method
Recent XAI methods have shown that deeper representations in CNN models illustrates higher-level visual features [30] [31]. A recent approach Grad- CAM [11] interprets the importance of each neuron responsible for a decision of interest by computing the gradient information from the last convolutional layer of the CNN. Alternatively, the authors in [17] proposed a method RISE, which finds the effect of selectively inserting or deleting parts of the input (perturbation-based ) in the CNN model's output prediction. This perturbationbased method provides more accurate visual explanation saliency maps compared to gradient based methods, but still they fail to visualize all the perturbations and determine which one characterizes the best desired explanation.
Furthermore, the visual explanations generated by both the gradient-based and perturbation explanations methods failed to localize the entire salient regions of an object class responsible for higher classification scores.
To overcome the challenges of the most recent state-of-the art methods we proposed a XAI method that consequently provides better explanation method for a any given CNN model. The proposed method takes the last convolution layer for generating the masks. From these masks Similarity Difference and Uniqueness scored are computed to get the explanation of the final CNN model decision acronymed in therefore denoted SIDU. An overview of the proposed method is presented in Figure 2. Our method is composed of three steps, First we extract the last convolution layer of the CNN to generate the mask using the last convolution layer of the given model. Second, we compute the similarity differences for each mask with respect to predicted class and finally we compute the weights of each mask and combine them into a final map which shows the explanation of the prediction. Each step is described in the following subsections.

Step1: Generating Feature Activation Masks
To provide a visual explanation of the predicted output of a CNN model, we first generate feature image masks from the last convolution layers. For any deep CNN model F, we consider the last convolution layers of size n × n × N where n is size of that convolution layer and N is the total number of features Each feature activation map f c i is then converted into a binary mask M c i by thresholding each value. The binary mask M c i is then up-sampled by applying bi-linear interpolation for a given input image I with size of W idth×Height. Next, the binary mask M c i will have values between [0, 1] and it is no longer binary. The up-sampled binary masks are also known as feature activation masks. Finally, point-wise multiplication is performed between feature activation mask (Up-sampled binary mask) M c i and input image I to get the feature image mask A i c and is represented as where F is an CNN model, A c i is the feature activation image mask of feature map f c i and i = 1, ....N . The procedure of generating feature activation masks is shown in Figure 3. The feature images masks A c of object class c are used to get prediction scores which is explained in detail in the following subsection 3.2

Step2: Computing Similarity Differences and Uniqueness
The total number of feature image mask is dependent on the number of activation's in the last convolution layer of the CNN model. Let the last convolution layer of the CNN model be of size n × n × N . The total number of feature is defined as P c i and the probability prediction score for the given input image I is defined as P c org . The prediction scores vector size will be depend on the total number of classes used of training the CNN model. For example, if the CNN model is trained on ImageNet dataset, which has a total of 1000 object classes, then the size of the prediction scores vector P c i of the each individual feature image mask A c i will be 1 × 1000, where i = 1...N . Figure 4 illustrates the procedure of computing the predictions scores vector.
Once the predictions score vectors are computed for all feature image masks and original input image, we then compute similarity differences between each input feature activation image mask prediction score P c i and prediction score P c org of the original input image I. The similarity difference between these two vectors gives the relevance of feature image mask w.r.t original input image. The intuition behind computing the relevance of a feature map is to measure how the prediction changes if the feature is not known, i.e., the similarity difference between prediction scores. The relevance value of the feature activation image mask will be high if it is similar to the predicted class and low otherwise. The similarity difference of the feature activation maps SD c i is given by where P c org , P c i are the predictions for the original input image and feature image mask generated from the last convolution layer and σ is an controlling parameter, respectively.
After computing the similarity difference measure, we also computed a uniqueness measure U c between the feature image masks prediction score vectors. It is one of the most popular assumptions that the image regions which stand out from the other regions grab our attention in certain aspects. Therefore the region should be labeled as a highly salient region. We therefore evaluate how different each respective feature mask is from all other feature masks constituting an image. The reason behind this is to suppress the false regions with low weights and highlight the actual regions which are responsible for predictions with higher weights. The uniqueness measure U c i is defined as Finally, the weight of each feature importance W c i is computed as the dot product of the similarity difference SD c i and uniqueness measure U c i where where SD c i , U c i are the similarity difference and uniqueness values for the feature activation image mask A c i of the object class c. The feature importance weight will be high for the feature which has more influence in predicting the actual class object c and low for the feature with low influence.

Step3: Explanations for the prediction
To get the explanation of the predicted output class c of a CNN model, we then perform a weighted sum between feature activation mask A c i and the corresponding feature importance weights W c i , where the weights are computed by Eq. 4. The visual explanation map is in the form of a heatmap and is  represented as S c and is shown in Figure 5. The visual explanation map S c is also known as the class discriminative localization map. Thus the visual explanation of the predicted class c is given by The weighted combinations of feature activation masks to get the final visual explanation of the prediction of the class is shown in Figure 5.
In summary, for explaining the decision of the predicted class c visually, we first generate the N feature activation masks (up-sampled binary masks) from the last convolution layer of deep CNN model F which has N number of feature activation maps of size n×n. We then perform point wise multiplication between each generated up-sampled binary mask M i and the input image I to

Evaluation
In this section we evaluate the performance of the proposed visual explanation method. We conducted a comprehensive set of experiments to study the correlation of the visual explanation with the model prediction to evaluate the faithfulness. The proposed method (SIDU) is evaluated using all the three categories of evaluations [14], functionally grounded, application grounded and human grounded evaluations. The evaluation results are compared with most recent state-of-the art methods RISE [17] and GRAD-CAM [5]. A good explanation method can not only provides the appropriate explanation for the prediction but also it should be robust against to adversarial noise. To this end the proposed method is evaluated on adversarial samples and compared with most recent state-of-the art methods RISE [17] and GRAD-CAM [5].

Functionally grounded Evaluation
To preform the functionally grounded evaluation we choose two automatic causal metrics insertion and deletion proposed by [17]. The deletion metric deletes the saliency region in the image which is responsible for higher classification scores and forces the CNN model to change its decision. This metric estimates the decrease in the probability classification scores, when more pixels are removed from the saliency region. With the deletion metric, the good explanation shows a sharp drop in the predicted score and area under the probability curve will be lower. Whereas, the insertion metric measures the probability increase of predicted score. As more pixels are inserted in the image, a higher AUC rate can be achieved (i.e., effectiveness of explanation model at a greater level). We choose these metrics since they are independent of human subjects, were collected from a large number of patients with retinal diseases [33].
We conducted two experiments for evaluating the faithfulness of the proposed explanation method. The first experiment is performed on ImageNet validation dataset. We collected randomly 2000 images from the ImageNet dataset. To do a fair evaluation, we choose two existing standard CNN models, ResNet-50 [34] and VGG-16 [35] pre-trained on imageNet dataset [32]. Table 1 summarizes the results obtained on ResNet-50 for the proposed method and compares it to most recent works [17] and GRAD-CAM [11]. We can observe that the proposed method achieved better performance for both metrics, followed by RISE [17] and GRAD-CAM [11].  Fig. 6. In our proposed method, the generated masks come from the last feature activation maps of the CNN model. Due to this the final explanation map will localize the entire region of interest (object class).
We also conducted a second experiment on Medical Image dataset which has totally different characteristics. We trained the existing ResNet-50 [34] with an additional two FC layers and softmax layer on the RFIQA dataset [33].
The CNN model achieves 94% accuaracy. The proposed explanation method uses the trained model for explaining the prediction of the RFIQA test subset with 1028 images. The evaluated results of the proposed method and RISE and GRAD-CAM are summarized in Table 5. We can observe that the GRAD-CAM achieves slightly higher AUC for insertion and lower AUC for deletion followed by SIDU. RISE [17] has shown least performance in both the metrics, This can be explained by the fact that the RISE method generates N number of random masks and the weights predicted for these masks give higher weights to false regions which makes the final map of RISE noisy. The visual explanations of the proposed method (SIDU) and the RISE, GRAD-CAM methods on the RFIQA test dataset are shown in Figure. 9 (b), (c), (d).

Human grounded Evaluation
Human-grounded evaluation is most appropriate when one aims at testing a general notions of an explanation quality. Therefore, for generic applications in AI domain such as object detection and object recognition, it might be sufficient to inspect a degree to which a non-expert human can understand the cause of a decision generated by a black box model. One excellent way to measure and compare the correlation of visual explanation between a human subject and the black box is to use an eye tracker that records he non-expert subject's fixations in interactive settings. This approach is chosen because of its similarity to XAI methods, visual explanations. Both generate heatmaps representing salient areas of an object in an image.
An eye tracker is used for gathering eye tracking data from human subjects to gain an understanding of visual perception [36,37,37]. The study using eye tracking data for understanding the human visual attention is useful but difficult and expensive to collect on a large quantity [36,37]. The authors in [38] established mouse tracking approach to collecting attention maps accurate. They In our study, we investigate how non-expert subjects generated explanations via the eye-tracker compares with those of generated by XAI visual explanation methods across natural images for recognizing object class. To this end, we follow the data collection protocol discussed in detail next section 4.2.1.

Database of eye tracking data
We sample randomly 100 images from 10 different classes of ImageNet [32] benchmark validation dataset. All the collected images are RGB and are resized to 224 × 224 pixels.

Data collection protocol
In order to collect fixation, 5 human subject participate in an interactive procedure using Tobii-X120 eye-tracker [41] in the following main steps: 1. The subject sits in front of screen where the eye-tracker is ready to record the eye fixation.
2. After careful initial calibration, each image from the dataset is shown in a random order for 3 seconds and corresponding fixations of the subject are recorded.
3. We divide all 100 images into 4 equally sized data blocks with a break between each experiment in order to reduce the burden on each subject.
We further add the Cross-fixation image between two stimulis to reset the visionary fixation on the screen while going from one image to another image.

Comparison Metrics
To ensure a robust evaluation we use the three metrics to compare the XAI and eye-tracker generated heatmaps [42]. vergence is an metric, which is used to measure dissimilarity between two probability density functions [42]. For evaluating the XAI methods, eyefixation maps and the visual explanation maps produced by the model are used for the distributions. F M represents the heatmaps probability distribution from eye-tracking data, and EM indicates the visual explanation maps probability distribution. These probability distributions are normalized and they are given by : where X is the number of pixels and is a regularization constant to avoid division by zero. The KL-DIV measure is computed between these two distributions to know whether the visual explanation map which is computed from the XAI method matches human fixations. It is a non-linear measure and generally varies in ranges zero to infinity. If the KL-DIV measure between EM and F M is lower, then the EM maps have better approximation of the human eye-fixation F M .
3. Spearmans Correlation Coefficient (SCC): Spearman's correlation is a non-parametric measure that analyses how well the relationship between two variables can be described using a monotonic function [43]. It is a statistical method used mainly for measuring the correlation or dependency between two variables. This metric varies between −1 and 1, where a score of −1, represents no correlation. The SCC between two variables will be high when observations have a similar ( correlation close to 1) rank between the two variables, and low when observations have a dissimilar (correlation close to −1) rank between the two variables [43].
It is an appropriate measure for both continuous and discrete ordinal variables [43]. F M represents the heat map from eye tracking data, whereas EM is the visual explanation map. The SCC between the two random variable maps, F M and EM are given by :  Table 4 summarizes the results obtained by SIDU and two different state-of-the art XAI methods We can observe that, SIDU outperforms GRAD-CAM and RISE in all the three metrics. Therefore we can conclude that SIDU explanations matches closer with the human explanations (heatmaps) for recognizing the object class.

Application grounded evaluation
Application grounded evaluation involves conducting experiments within a real application to assess the trust of the black box models. We choose an medical case. As a test application we use the task of retinal fundus image quality assessment [33].   To perform this experiment, we choose one the most successful white box attacks, namely, gradient based attacks. Fast Gradient Sign Method (FGSM) [25] and Projected Gradient Descent (PGD) [26] are the examples of such attacks.
PGD is an iterative application of FGSM. The process of PGD is more complex and time consuming. Therefore, we have chosen the Fast Gradient Sign Method (FGSM) because of its simplicity and effectiveness. The adversarial image is generated using FGSM by adding noise to an original image. The direction of this noise is the same as the gradient of the cost with respect to the input data. The amount of noise can be controlled by a coefficient, . By applying this coefficient properly, it will change the model predictions and it is undetectable to a human observer. Figure 10 shows the different levels of FGSM adversarial noise added to the original image. We conducted two different experiments using adversarial noise to demonstrate the effectiveness of SIDU, compared to state-of-the art methods RISE and GRAD-CAM. The experiments are described in following. These values are optimal because, it is sufficient enough to pass unnoticeable by the human eye. We extracted the visual explanations heatmaps using the proposed method (SIDU), RISE [12] and GRAD-CAM [11]. The heatmaps generated by SIDU, RISE and GRAD-CAM methods are finally compared with human generated visual explanations using eye-tracker described in section 4.2.1 using the three evaluation metrics AUC, SCC and KL-DIV.       Figure 11, we can clearly observer that the propose method(SIDU) doesn't deviate much in localizing the object class which is responsible for the prediction. Therefore from these two adversarial noise experiments we can conclude that our proposed method is more robust against adversarial noise.

Conclusion
In this work, we proposed a novel method called SIDU for explaining the CNN model, predictions visually in a form of heatmap through feature activation maps of the last convolution layers in the model. The proposed method is independent of gradients and can effectively localize entire object classes in an image which is responsible for CNN prediction. The new explanation approach helps in gaining more trust in prediction results of CNN model by providing further insights to the end-user in sensitive-domain. We validated the effectiveness of our method by conducting three different XAI evaluations methods, applica-tion grounded (invoking human experts trust in medical domain), functionally grounded (using an automated causal metrics independent of humans) and human grounded evaluation. For the human grounded evaluation, we proposed a framework for evaluating explainable AI (XAI) methods using an eye-tracker.
It is designed specifically for evaluating XAI methods using non experts to understand the human visual perception for recognizing the given object class and