{'fraction_non_alphanumeric': 0.04613362844100152, 'fraction_numerical': 0.010063632590953105, 'mean_word_length': 4.665556426332288, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
GAM: Explainable Visual Similarity and Classification via Gradient Activation Maps

We present Gradient Activation Maps (GAM) - a machinery for explaining predictions made by visual similarity and classification models. By gleaning localized gradient and activation information from multiple network layers, GAM offers improved visual explanations, when compared to existing alternatives. The algorithmic advantages of GAM are explained in detail, and validated empirically, where it is shown that GAM outperforms its alternatives across various tasks and datasets.


INTRODUCTION
As the AI revolution disrupts industries and penetrates all walks of life, a growing need arises to intuitively explain machine-based decisions [6,25]. As a result, an emerging research area revolves around the need to make machine learning models more explainable. This work joins this common effort and presents Gradient Activation Maps (GAM) -a novel method for explaining visual similarity and classification networks.
A saliency map is an image depicting the relative contribution of each pixel in the input image w.r.t. the model's prediction. For example, Fig. 1 presents saliency maps produced by GAM for a classification task (a-c) and a similarity task (d-e). According to [19], a 'good' visual explanation technique should be (1) class discriminative i.e., localize the object in the image, and (2) high-resolution i.e., capture fine-grained details. However, comparing different visual explanation approaches is hard: A real methodological challenge stems from the lack of a ground-truth or a principled evaluation procedure. Hence, different works employed different evaluation procedures, often resorting to subjective visual assessments [20,22].
An actionable testing procedures for assessing the validity of saliency maps were recently proposed by Adebayo [1]. Their work revealed that despite producing quality looking visualizations, most * Authors contributed equally to this work. state-of-the-art methods produce saliency maps that are independent of either the model or the input-label relation, rendering them inadequate for producing explanations. An exception was the Grad-CAM (GC) method from [19] that stood out among all others in its ability to produce fine-grained saliency maps, while successfully passing all sanity tests [1]. Following the success of GC, an improved extension called Grad-CAM++ (GC++) was introduced and shown to outperform its predecessor on various visual explanation tasks [4].
GAM poses significant improvements upon GC and GC++ via several algoritmic features: Gradient localization, multi-layer analysis, and negative gradients suppression. These unique features lead to better saliency maps in terms of resolution, class discrimination, and object localization. In Sec. 3.3, we elaborate on the relation of GAM to GC and GC++ and explain the algorithmic advantages behind GAM's superior results.
Our contributions are as follows: • We introduce GAM -a state-of-the-art method for extracting accurate saliency maps in terms of resolution and class discrimination. GAM is shown to outperform its alternatives on various objective and subjective evaluations, across all metrics, and especially in the case of small objects. • We present a unified formulation for visual similarity and classification that enables the utilization of GAM, GC and GC++ for explaining visual similarity models (a task that was overlooked in [4,19]). • We identify and demonstrate the limitations of GC and GC++, and explain how GAM averts these problems.

RELATED WORK 2.1 Explaining Visual Classification Models
The early methods proposed by [2,13,20,21,[29][30][31][32] are seminal works in visualization and understanding deep NNs. Guided Backpropagation (GBP) [21] visualizes the output prediction by propagating the gradients through the model and suppressing all negative gradients along the backward pass. However, GBP was shown to produce saliency maps that are not class discriminative [19]. Another approach [20], uses the gradients of predicted class scores w.r.t. to the input image to generate saliency maps.
Recently, Grad-CAM (GC) [19] created saliency maps based on the activations and gradients from the last convolution layer. In GC, the gradients of each channel are pooled to scalars. Then, these scalars weigh their corresponding activation maps that are summed together to produce the final saliency map. More recently, Grad-CAM++ (GC++) [4] was introduced as an improved version of GC. GC++ uses a weighted average of the pixel-wise gradients in order to create the weights for the activation maps.
Both GC and GC++ operate on the last convolutional layer and employ gradient pooling that leads to the loss of gradient localization. In contrast, GAM utilizes the raw gradients from multiple layers in the network, enabling gradient localization with improved resolution and class discrimination.

Explaining Visual Similarity Models
Previous works attempted to visually explain the decision made by similarity networks [9,14,18,23,26,28]. These networks are optimized to cluster images that are considered similar, in a learned vector space. Other methods [15,24] determined areas that contributed to image similarity by comparing filter responses of images patches. In [5], the authors utilized GC for explaining embedding networks that were trained on similarity tasks. However, their method is independent of the similarity score itself, hence it cannot be considered a "true" explanation to similarity.
Recently, VDSN [22] was introduced as a method for visual explanation for similarity networks. VDSN produces saliency maps for image-pairs by combining the activations of the last convolution layer before and after average / max pooling. However, unlike GAM that utilizes the gradients of the similarity w.r.t. the activations from multiple layers, VDSN does not use the gradients, and hence is indepedent of the similarity score. Moreover, VDSN is limited to use the last convolutional layer in architectures that employ average / max pooling, and is applicable to similarity networks only (thus unable to visually explain classification models).

Evaluating Saliency Maps
Evaluating saliency maps is challenging, as no real "ground truth" exists, and the quality of an explanation is often subjective. In [19,20], evaluations conducted using a weakly supervised object localization task, where the output saliency map is being used to specify the region in the image in which the classified object appears. We further extend this approach to the image similarity task, by using the saliency maps to specify the regions in which similar objects appear in both images.
In [4] the authors suggested the Average Drop Percentage (ADP) and the Percentage of Increase in Confidence (PIC) metrics, to measure the change in the model confidence when using explanation maps (Hadamard product of the saliency map with the original image) instead of original image. We follow these tests and further extend them to the image similarity task.
In [1], the authors suggest sanity tests for saliency maps methods: The parameter randomization and data randomization procedures test whether the produced saliency map is sensitive to the randomization of the model's parameter and data labels, respectively. Otherwise, the method fails to faithfully explain the model's prediction. Despite producing quality looking visualizations, the tests from [1] reveal that many of the popular saliency methods do not pass the tests, and therefore are not adequate for providing satisfactory model explanations. In Appendix 4.2 we show that GAM passes these tests.

GRADIENT ACTIVATION MAPS (GAM) 3.1 A Unified Formulation for Visual Similarity and Classification
We begin by defining notations for the network's input and (internal) building blocks. The network's input is an image, denoted by ∈ R 0 × 0 × 0 . The 3D activation produced by the -th convolutional layer (for the image ) is denoted by ℎ ∈ R × × , where 1 ≤ ≤ . Note that ℎ is not necessarily produced by a plain convolutional layer, but can be the output of a more complex function such as a residual [8] or DenseNet [10] block. We further denote ℎ ≜ ℎ [ ] (∈ R × ) as the -th activation map in ℎ . Let : R × × → R be a function that maps 3D tensors to a -dimensional vector representation. We denote the mapping of the last activation maps ℎ by ≜ (ℎ ) ∈ R . Note that may vary between different network architectures. Usually, it consists of a (channel-wise) global average pooling layer that is optionally followed by subsequent fully connected (FC) layers.
Finally, let : R × R → R be a scoring function that receives two vectors and outputs a score. The use of varies between tasks: classification and similarity. In classification tasks, represents the last hidden layer of the network. The logit score for the class is computed by ( , ), where ∈ R is the weights vector associated with the class . In multiclass (multilabel) classification, is usually set to the dot-product, optionally with bias correction, or the cosine similarity. Then, either a softmax (sigmoid) function, with some temperature, transfers values to the range [0, 1].
For similarity tasks, we consider two images , ∈ R 0 × 0 × 0 , and a similarity score: ( , ). A common practice is to set to the dot-product or cosine similarity. Further note that in the specific case of similarity, the representation produced by is not necessarily taken from the last hidden layer of the network. Therefore, can be set to the output from any FC layer. For the sake of brevity, from here onward, we abbreviate both ( , ) and ( , ) with . Disambiguation will be clear from the context.

The GAM Method
Given an image , we denote the -th saliency map which is a function of the activation maps ℎ and their gradients: ≜ ℎ . We denote ≜ [ ] (similarly to the notation ℎ ). Then, we implement (ℎ , ) as: where is the ReLU activation function, and • is the Hadamard product. RSZ denotes the operation of resizing to a matrix of size 0 × 0 (the height and width of the original image ). NRM denotes the min-max normalization.
The motivation behind Eq. 1 is as follows: each filter in the -th convolutional layer captures a specific pattern. Therefore, we expect ℎ to have high (low) values in regions that do (not) correlate with the -th filter. In addition, regions in that receive positive (negative) values indicate that increasing the value of the same regions in ℎ will increase (decrease) value.
GAM highlights pixels that are both positively activated and associated with positive gradients. To this end, we first truncate all negative gradients (using ReLU). Then, we truncate negative values in the activation map ℎ , and multiply it (element-wise) by the truncated gradient map. This ensures that only pixels associated with both positive activation and gradients are preserved. Then, we sum the saliency maps across the channel (filter) axis to aggregate per pixel from all channels in the -th layer. The -th saliency map is obtained by resizing (via bi-cubic interpolation) to the original image spatial dimensions followed by min-max normalization. This process produces a set of saliency The final saliency map ≜ ( , ) is computed based on a function that aggregate the information from the saliency maps produced by last layers. In this work, we implement as follows Note that in our experiments, we found out that different implementations of , such as max-pooling, Hadamard product, or various weighted combinations of , performs worse than Eq. 2. Yet, in Sec. 4, we do investigate the effect of different values on the final saliency map .

GAM's Unique Features
GAM presents several advantages over GC and GC++: Gradient Localization: GC computes the saliency map based on a linear combination of the activation maps in the last convolutional layer as follows: When compared to GAM, the computation in Eq. 3 has two major drawbacks: First, the coefficients are the pooled gradients. Hence, in GC (and GC++), the gradient spatial information is lost. This is in contrast to our GAM approach (Eq. 1) that preserves (positive) gradient localization via the elementwise multiplication by ( ). The significance of this property is well expressed in the Positive gradients row in Fig. 2.
Multi-layer Analysis: GC produces saliency maps based on the last convolutional layer only. GAM, on the other hand, gleans information extracted from multiple layers (or blocks) that vary by their resolution and sensitivity (Eq. 2). Earlier blocks in the network are characterized with higher resolution. For example, in DenseNet, the last convolutional layer produces low-resolution activation maps of size 7 × 7 whereas the preceding convolutional layer produces activation maps of 14 × 14.
Our findings show that extracting information from earlier blocks is critical in certain architectures. In Sec. 4, we show that incorporating information from earlier blocks (i.e, setting > 1) enables GAM to produce fine-grained saliency maps that are more focused on the relevant objects. However, the application of the same feature to GC / GC++ hurts performance ( Fig. 8 and Tabs. 1, 2).
Negative Gradients Suppression: A subtle, yet highly important drawback of Eq. 3 stems from the way in which the (ReLU) operation is applied. In GC, the weighted combination of the activations ℎ is summed, where each activation is weighted by its pooled gradients . In architectures like ResNet or DenseNet, ℎ are always non-negative (due to the ReLU activation at the end of each block). However, the pooled gradients can still result in a negative value. As a result, GC might become insensitive to important regions (pixels) that should be intensified. The justification for this claim is as follows: Consider a pixel ( , ) in a region that contributes to the final score . Ideally, we wish this pixel to be intensified in the final saliency map. By its nature, such a pixel in an "important" region is expected to have positive (pooled) gradient values and positive activation values across several filters. However, it is also possible that some other filters that respond with a small, yet positive activation, will be associated with negative (pooled) gradients values. Mathematically, this is expressed by the following decomposition: If | | ≥ , then the pixel ( , ) will have an intensity ≤ 0. In this case, the pixel ( , ) as well as other pixels in the region, are zeroed and masked due to the subsequent application of (ReLU) in Eq. 3. This might further lead to a relative intensification of other, less "informative" pixels ( ′ , ′ ) (associated with much smaller contributions than of ( , )), but for which ′ ′ > 0.
GAM on the other hand, applies to the gradients before the multiplication by the activations ℎ (Eq. 1). This ensures negative gradients are zeroed and hence do not (negatively) affect the region's intensity on other channels or layers. Thus, regions with positive gradients are never masked by and "correctly" intensified according to the magnitudes of the positive gradients and activations only.
In GC, the negative gradients problem becomes noticeable when using the cosine similarity. Fig. 2 exemplifies this effect, presenting a comparison between GC and GAM (using DenseNet201). We used the 'last layer' version of GAM (Eq. 2, = 1), ensuring the improvement by GAM is indeed due to the way it computes the saliency maps, neutralizing the contribution from earlier layers. Each pair of columns in Fig. 2 presents saliency maps computed w.r.t. the cosine similarity. We see that GC (third row, marked red) produces saliency maps that intensify wrong regions (left image of each pair in rows 2). Empirically, this is explained by the accumulated activation maps and the positive gradient maps (shown in rows 4-5 after ReLU), and the negative gradient maps (shown in row 6 after negation and ReLU). In both examples (dog and chair), we observe the high magnitude of the negative gradients and their adverse effect: the final intensity in regions of interests is significantly attenuated compared to the background, resulting in poor quality saliency maps. However, as explained above, by suppressing negative gradients in advance, GAM averts this problem and successfully produces adequate saliency maps.
The poor performance of GC, when using the cosine similarity (instead of the dot-product), can be further explained mathematically: In the case of the dot-product similarity, ( , ) = , and the gradients , are guaranteed to be non-negative. This stems from the fact that in DenseNet (and many other architectures), the global average pooling operation is applied after the application of ReLU, hence both Eq. 5 shows that (and hence ) is the difference between two positive vectors, and hence may contain negative entries. Therefore, in the case of cosine similarity, negative gradients are possible, and might mask "important" regions in the image that should be intensified in the saliency map.
Finally, when using the dot product similarity, it is GC++ that completely fails. Fig. 3 compares between GC++ and the 'last layer' GAM ( = 1). GC++ weighs the pixel-wise gradients (before pooling) with the coefficients: Note that during the computation of , GC++ passes through the exponential function. However, when is the dot-product, this  may lead to an "explosion" of the saliency map values, as observed in Fig. 3. GAM, however, produces adequate saliency maps.

Subjective Evaluation
First, we demonstrate GAM's ability to explain visual similarity models. To this end, we set to the embedding produced by the (channel-wise) global average pooling layer in an ImageNet pretrained DenseNet201 model (discarding the classifier head). To determine the similarity of two images, and , the images are passed through the model to generate the embeddings and . Then, the similarity score is computed by ( , ), where is either the dot-product or cosine similarity. Figure 6: Saliency maps produced by GAM, GC and GC++ w.r.t. the classes (top to bottom) "sunglasses", "oboe", "soccer ball", "coffeepot", "matchstick" and "anemone fish".
In Fig. 4, row-pairs present saliency maps for pairs of image representations w.r.t. the cosine similarity. The saliency maps by GAM, were produced using two layers (setting = 2 in Eq. 2). We see that GAM produces quality saliency maps, while GC (column 3) consistently fails. When compared to GC++, GAM exhibits saliency maps that are more focused on the source for the similarity. Results w.r.t. the dot-product appear in Fig. 5. In this case, we see that GC++ completely fails.
Next, we turn to demonstrate GAM's ability to visually explain classification models. In this case, the saliency maps are computed w.r.t. the logits scores produced by DenseNet-201. Specifically, we compute ( , ), where is the dot-product, is the image representation, and is the weights vector associated with the class . Fig. 6 presents examples of saliency maps produced by GAM ( = 2), GC and GC++. It is visible that GAM produces saliency maps that are more class discriminative than the ones produced by GC and GC++. These results further support the analysis from Sec. 3.3, demonstrating the advantages of GAM (over GC and GC++), and show that GAM generates adequate saliency maps in all settings.

Sanity Checks for Saliency Maps
As explained in Sec. 2.3, visually appealing saliency maps can be misleading. To assess the validity of GAM for explanations, we conduct the parameter randomization and the data randomization sanity tests from [1]. GAM passed both tests. Figure 7 presents examples from the sanity checks. The first row shows two saliency maps produced by GAM w.r.t. the "tabby cat" class. We see that when GAM utilizes an ImageNet pretrained ResNet50 model, it produces a focused saliency (around the cat), but when applying GAM to the same network with randomly initialized weights, it fails to detect the cat in the image. Thus, we conclude that GAM is sensitive to model parameters and passes the parameter randomization test. The second row shows that GAM produces an adequate saliency map when the model (LeNet-5 [11]) is trained with the true MNIST labels, but fails when the model is trained with random labels. Thus, we conclude that GAM is sensitive to data labels and passes the data randomization test.

Layer Ablation Study
In this section, we test whether GAM, GC, and GC++ benefit from the use of multiple layers. On one hand, earlier layers are associated with smaller receptive fields, giving better localization. On the other hand, these layers usually account for less semantic features. Fig. 8 presents a comparison of GAM, GC and GC++ when using multiple layers ( ∈ {1, 2, 3}). We see that GAM benefits from the use of multiple layers, while GC and GC++ do not. Figures 9 and 10 demonstrate the advantage of using multi-layer GAM compared to a single layer GAM. Three images are presented, each with a small object (dog, airplane, and cat). We see that GAM based on earlier layers ( = − 1, − 2) produces more focused saliency maps due to higher resolution analysis. This leads to a better localization in the final saliency map as seen in 'GAM(sum)' ( = 3).  Figure 11 presents another layer-wise analysis, where it is observed that the last two layers (second row, last two columns), corresponding to = 2, best balances localization with the extraction of semantic features, yielding optimal results. In addition, gradient localization is observed in the 'Gradients' columns, which is a unique property of GAM (in contrast to GC that performs gradient pooling). For further explanations, see Sec. 3.3 (Gradient Localization).
Indeed, in our experiments, we noticed that GAM with = 2 best balances localization with the extraction of semantic features. Yet, when setting > 1 for GC and GC++, performance degrades. As we shall see, these trends repeat in the quantitative evaluation in Secs. 4.4 and 4.5 as well.

Objective Evaluation
Next, we present objective evaluation, following the measures suggested by in [4] (we refer to [4] for the full details): Average Drop Percentage (ADP): ADP is computed as: where is total number of images in evaluated dataset, is the model's output score (confidence) for the correct class w.r.t. the original image . is the same model's score, this time w.r.t. the 'explanation map' -a masked version of the original image (produced by the Hadamard product of the original image with the saliency map). The lower the ADP the better the result.
Percentage of Increase in Confidence (PIC): PIC is computed as: PIC reports the percentage of the cases in which the model's output scores increase as a result of the replacement of the original image with the explanation map. The higher the PIC the better the result.
We further extended the evaluation from [4] to similarity tasks, by reporting ADP and PIC w.r.t. image-pairs similarity scores (instead of class specific scores). To this end, we created a similarity subset (will be made public) by randomly sample image-pairs from the ILSVRC-15-val dataset [17] (which does not overlap with the training set used the trained the models), but with the restriction that each pair contains images that are labeled with the same ground truth class. The similarity subset contains 3000 pairs in total, 3 for each class.
In addition, we tested the ability of GAM to benefit from using several layers, when it is applied on images with small objects. We compare the localization capability on small objects by narrowing the ILSVRC-15-val dataset to a subset that contains images for which the ground truth box area is below the 25% / 10% percentile area. For the similarity experiment, we randomly sampled another 3000 pairs, from the 25% / 10% narrower sets.
The results are reported in Tab. 1 (ResNet101). For each method, we report the results both for = 1 and = 2 (note that = 3 performs on par with = 2, hence omitted). Recall that for ADP (PIC), lower (higher) values indicate better performance, and Impr. reports the relative improvement obtained by using = 2 (over = 1). We see that GAM outperforms GC and GC++ at the majority of the scenarios. Moreover, GC (GC++) completely fail when using the cosine (dot-product) similarity. This is another empirical evidence for GC and GC++ limitations (Sec. 3.3), and the fact that GAM benefits from multiple layers, whereas GC and GC++ do not (and even degrade). Finally, the results for ResNet101 exhibit the same trends, but are excluded due to space limitation.

Object Localization and Segmentation
In this section, we compare the localization capability of GAM, GC and GC++ via an extensive set of experiments across various tasks, datasets, models, and settings. We measure the quality of the produced saliency maps by Intersection over Union (IoU%) w.r.t. the ground truth bounding boxes (BBox) or segmented areas. To this end, each saliency map is binarized with a fixed threshold before drawing the predicted BBox or segmented area. The fixed threshold was chosen for each test and method separately by a hold-out set (will be made public). Table 2 presents the obtained localization accuracy (IoU%) for each combination of task, dataset, model, and method, both for = 1 and = 2, including the obtained improvement when using = 2. Again, we observe that GAM outperforms the other methods. Moreover, it is evident that GAM significantly benefit from using multiple layers (especially in the case of small objects), whereas GC and GC++ suffer from a significant degradation in accuracy when utilizing more than a single layer. In what follows we discuss the results per task.
Localization by Classification: We followed the test protocol from GC [19], where the saliency maps of a classification model are used to draw a BBox around classified objects. We apply the two-layer GAM (Eq. 2, = 2) GC and GC++ on top of pretrained DenseNet201 and ResNet101. Figures 12, 13 and Fig. 14 (Rows 1-2) present examples for the generated saliency maps and BBoxes (marked orange). Tab. 2 (row 1) presents the localization accuracy (IoU%) between the predicted and ground truth (ILSVRC- 15-val) boxes. In all cases, GAM outperforms both GC and GC++.     Localization by Similarity: We adjusted the protocol from the localization by classification experiment to support localization by similarity. To this end, we replace the classification score with the similarity score computed for image-pairs. We used the same imagepairs from the similarity subset (Sec. 4.4). Then, we drew a BBox for each image in the pair, and computed IoU% w.r.t. to the ground truth. Results w.r.t. the different similarity scores are reported in Tab. 2 (rows 5,8) and demonstrated in Fig. 15. Again, we observe that GC (GC++) fails when using the cosine (dot-product) similarity, and significantly degrades when utilizing multiple layer, while GAM performs the best and clearly benefits from multiresolution analysis.
Localization of Small Objects: We used the 25% and 10% partitions from Sec. 4.4 for testing the localization capability GAM on small objects. In addition, we conducted a localization experiment on medical imaging dataset ChestX-ray8 [27], where the classification decisions are usually made due to small details in the images. In this experiment, we used the CheXNet model from [16] that was trained on the ChestX-ray8 dataset to classify common thorax diseases. The results for the ILSVRC-15-val 25% / 10% subsets and the ChestX-ray8 appear in the classification and similarity sections in Tab. 2, and demonstrated in Fig. 14 (rows 3-4). We see that GAM significantly outperforms both GC and GC++. These findings support the observation from Sec. 4.3 that multi-layer GAM ( > 1) produces better saliency maps for small objects.
Object Segmentation: Finally, we tested the utilization of GAM, GC and GC++ for object segmentation. To this end, we applied the methods on top of two pretrained multi-label classification TResNet [3] models, trained on MS-COCO [12] and Pascal VOC [7] datasets. For each image, we computed the saliency maps w.r.t. each of the ground truth labels. Then, we computed the IoU% of the binarized saliency map w.r.t. ground truth segmentation (in pixels), for each ground truth label. The results appears in Tab. 2 (Segmentation), and exemplified in Fig. 14 (rows 5-6), and in Figs. 16 and 17. Overall, we see that GAM produces the most accurate segmentation.

CONCLUSION
This work joins a growing effort to make machine learning models more transparent and explainable. To this end, we present GAM, a state-of-the-art method for explaining visual similarity and classification models in a unified manner. Extensive subjective and objective evaluations show that GAM outperforms its alternatives across various tasks and datasets, and especially on small objects.