{'fraction_non_alphanumeric': 0.029677729677729678, 'fraction_numerical': 0.00808080808080808, 'mean_word_length': 4.866534988713318, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A HVS-inspired Attention Map to Improve CNN-based Perceptual Losses for Image Restoration

Deep Convolutional Neural Network (CNN) features have been demonstrated to be effective perceptual quality features. The perceptual loss, based on feature maps of pre-trained CNN's has proven to be remarkably effective for CNN based perceptual image restoration problems. In this work, taking inspiration from the the Human Visual System (HVS) and our visual perception, we propose a spatial attention mechanism based on the dependency human contrast sensitivity on spatial frequency. We identify regions in input images, based on underlying spatial frequency where the visual system might be most sensitive to distortions. Based on this prior, we design an attention map that is applied to feature maps in the perceptual loss, helping it to identify regions that are of more perceptual importance. The results will demonstrate that the proposed technique helps improving the correlation of the perceptual loss with human subjective assessment of perceptual quality and also results in a loss which delivers a better perception-distortion trade-off compared to the widely used perceptual loss in CNN based image restoration problems.


Introduction
In this paper, we focus on Full-Reference Image quality metrics [21] for image restoration. The PSNR and SSIM [25], have been traditionally used as FR quality metrics between two images, but they have been shown to quantify distortion, not perceptual quality. The difference and tradeoff between distortion and quality is well explained in [3]. The human visual perception process is fairly complex and still not understood completely. Recently, we have observed an increased use of deep CNN features as perceptual quality features for Image restoration and quality assessment.
The perceptual loss proposed by Johnson et al. [11] was one of the first to demonstrate how effective the deep CNN features can be as perceptual quality features, especially when used in loss functions for image restoration. The per-ceptual loss is now a benchmark and standard loss function popularly adopted in many image to image translation problems such as super-resolution, style transfer, denoising etc. [15], [24], [5]. The recent PIRM Super-Resolution Challenge Report [2] clearly iterates that the perceptual loss is the standard benchmark loss function for perceptual image restoration and is also used in combination with other losses such as the GAN loss. Zhang et al. [27] and Blau et al. [3] further demonstrate how effective the deep features really are as perceptual quality features. Considering the difficulty in quantifying perceptual quality and the importance of effective differentiable metrics, it is surprising that a significant amount of research as been subjected to novel CNN architectures for image restoration but the loss functions, which are of as much if not more importance, remain relatively un-addressed.
In this work, we exploit a known and well established property of the HVS called contrast sensitivity. Psychovisual experiments have revealed that the visual system's ability to respond to contrast, varies significantly with spatial frequency of the stimulus [17]. Neuro-scientists have been able to quantify this variation using a function called the Contrast Sensitivity Function (CSF) [20]. Considering that it is an important representative factor of our visual perception, the CSF has been used extensively by the community to improve image processing algorithms and techniques [20].
We use a variation of the CSF filtering technique of images to extract maps which represent regions which the human visual system is most sensitive to in terms of contrast. Theoretically, due to our enhanced contrast sensitivity while perceiving these regions, distortions in these regions should be more perceptible and prominent to the human observer. The map is then applied to feature maps of pre-trained networks used in the perceptual loss to give more emphasis to important areas during restoration. The results will demonstrate that the technique improves the perceptual quality of restored images and delivers a better perception-distortion trade-off compared to the standard and popularly used perceptual loss [11]. The distance between feature representations of reference and restored images is generally used as a perceptual metric in the perceptual loss and its variants. We propose a spatial attention map, applied to feature representations to enable them to focus on more important regions during the learning process.

Contrast Sensitivity Function (CSF)
The spatial theory frequency aims to correlate the response of the visual cortex with spatial frequency of input stimulus [17]. The theory is contrary to earlier theories of Hubel and Wisel which used straight edges and lines to characterize perception in the visual cortex [7]. The theory is supported by psycho-visual experiments which employ sinusoidal gratings. According to the Fourier theory, a light distribution of an image on the retina can be expressed as a linear combination of basis harmonic components. Grating stimuli have therefore been used in psycho-visual experiments to represent basis harmonic components.
Experiments such as the ones described in [17], [9] and [14] show that the neurons in various regions of the visual cortex are tuned to specific spatial frequencies. This behavior gives an interpretation of the function of different regions of the cortex, supporting the claim that a major function of the primary visual cortex is to split images into harmonic components for further processing.
The net response/sensitivity of the HVS to different spatial frequencies is summarized in the CSF. Fig. 2(a) shows the human CSF derived through psychovisual experiments. The units of the CSF are in cycles/degree. This unit translates to the number of cycles per visual degree of the ob-server.

CSF based Map Generation
Let I GT be the Y channel of the ground truth image with size MxN. The 2D DFT of the image will be (1) Where 'u' and 'v' are the horizontal and vertical frequency indicators respectively. As iterated earlier, the CSF is defined on the spatial frequency scale of cycles per visual degree. The current frequency units of the DFT need to be appropriately converted into the required scale for effective application of the CSF. Let, where ∆ = 0.25mm is the dot pitch. The cycles/degree equivalent of the DFT frequency units with a viewing distance of 'dis' millimeters will be s(u, v ) [1] where, (3) Now we have effectively expressed the DFT on the spatial frequency units of cycles/degree. To generate a map that represents the most important spatial information, we will approximate the input image using a narrow band of spatial frequencies around the peak of the CSF. Let, Now we take the inverse 2D DFT of the approximated representation to generate a map as shown in Eq. (5).
As a final step, we normalize the matix 'T' so that its largest element is unity. T M is the max norm of T in Eq.(6) (largest element of the matrix).
The viewing distance for HDTV displays should be selected for a 30 degree viewing angle. Considering parameters for viewing distance and for effective perception of common distortions (checkerboard artifacts etc) introduced by the use of the perceptual loss, we selected the viewing distance to be 55cm. We empirically selected the range of the filter to be the frequencies corresponding to 45% of the peak of the CSF i.e s l =2 cycles per degree and s u =23 cycles per degree.

Perceptual Loss
Deep CNN features are the result of non-linear transformations on images into a high dimensional manifold. The perceptual loss is based on the fact that distance between features on the transformed manifold might result in a better perceptual quality measure compared to distance between image pixels themselves. The high dimensional manifold in this case is the manifold of CNN features. The standard form of the perceptual loss, as per [11] is given in Eq.(7) Where 'Φ k m ' is the 'm th ' feature map in the 'k th ' layer with 'M' number of feature maps with dimensions 'H×W' . This approach and its variants have proven to be remarkably effective as perceptual features in FR-IQA methods [4], image restoration [24] and style transfer [5] problems. The perceptual loss is a benchmark loss function in perceptual image restoration as demonstrated in [2].

Proposed Spatially Attentive Perceptual Loss
The CSF based attention map is a representation of regions which produce higher contrast sensitivity in perception. These regions will be the regions on which distortions will be most perceptible to human observers. The perceptual loss in the form of Eq. (7) does not spatially discriminate between information in feature maps on the basis of their perceptual importance. We propose extending the perceptual loss to a form in Eq. (8).
is the CSF based attention map. The superscript 'k' represents that is has been appropriately re-sized to match the feature map dimensions in the k-th layer. The formulation in Eq. (8) will help the perceptual loss to give more importance in restoration of regions which are perceptually more important and have a visual susceptibility to perception of distortions which result in a loss of contrast, commonly associated with Super-Resolution and Blurring.
The results will demonstrate that our proposed extension provides a superior metric that is more correlated with human subjective assessment of perceptual quality and delivers a better perception-distortion trade-off when used for CNN based image restoration.

Overview
To demonstrate the superiority of our proposed extension over the classical and widely used perceptual loss, we will make use of two distinct experimental techniques. The first being objective quality assessment (OQA) tests and the second being an image restoration experiment.
OQA tests are a standard method to evaluate the effectiveness of perceptual quality metrics. The best judges of perceptual quality are human observers. OQA tests try to correlate the performance of objective metrics with human judgment of perceptual quality. The more correlated a metric is with human subjective assessment of quality, the better it can be inferred as a perceptual quality metric. For human subjective assessment of perceptual quality, we will use the LIVE subjective image quality database [22]. The data-set consists of human subjective scores in the form of Differential Mean Opinion Scores (DMOS) for a large number of images with a wide variety of distortions. The correlation of objective metric scores with human subjective scores is quantified using three metrics namely the Root Mean Square Error (RMSE), Linear Correlation Coefficient (LCC) and the Spearman Rank Order Correlation Coefficient (SROCC) after curve fitting of the objective and subjective scores using a non-linear polynomial.
For comparing the objective metrics in Eq. - (7) and -(8), we will use feature maps from different layers of a wide variety of pre-trained networks such as the VGG-16 [23], ResNet-18 [6], SqueezeNet [8] and AlexNet [13] to reinforce the validity of our approach. We will also repeat our OQA test for a wide variety of distortions such as Gaussian Blur, JPEG and JPEG2000 Compression Noise and Multiple Distortions [10] (camera image acquisition process where images are rst blurred due to narrow depth of field or other de-focus and then corrupted by white Gaussian noise to simulate sensor noise). A diverse experimental setup with multiple networks, their layers and different types of distortions will reinforce the validity and effectiveness of our approach.
Our second experiment will be an x4 Image Super-Resolution experiment using the well known VDSR network trained on the standard DIV2K data-set using the using the loss functions in Eq. -(7) and -(8) used in combination with the l 2 loss. The two loss function that will be used to train the network will hence be; and To evaluate and compare the performance of both loss functions, we will compare the perception-distortion trade-off they both deliver. The Structural Similarity Index (SSIM) and the Peak Signal to Noise Ratio (PSNR) have traditionally, some what incorrectly been used as metrics to evaluate image restoration techniques. These metrics measure distortion between two images and are not sufficient to quantify the perceptual quality. Recent works such as [3] and [27] have pointed out that to quantify the efficacy of image restoration techniques, we need to analyze something known as the perception-distortion trade-off and perceptual quality. In recent works [2], The perceptual quality has been quantified using a metric known as perceptual index (PI) which is a combination of two No-Reference quality metrics called the Naturalness Image Quality Evaluater (NIQE) [19] and the NRQM [16]. The PI of an image I in is thus defined in Eq.(11) P I(I in ) = 1 2 ((10 − N RQM (I in )) + N IQE(I in )) (11) The lesser the PI, the better the perceptual quality of the image. The distortion (measured by the SSIM and PSNR) and the perceptual quality (PI) are in a trade-off relation, as explained in [3]. The effectiveness of one technique over another can be quantified by its ability to deliver a better trade-off i.e lesser PI at the same SSIM. We will sufficiently demonstrate that our proposed loss does exactly that in comparison to the perceptual loss.
(a) lp (b) l att p Figure 3: It can be seen that our proposed extension (l att p ) is more correlated with human subjective DMOS compared to the perceptual loss (l p ). This result is for loss with feature maps 'Fire3 ReLU expand1x1' layer of the SqueezeNet on Gaussian Blur distorted images. Table. 1 demonstrates the correlation of the perceptual loss (Eq. 7) and our proposed extension (Eq. 8) with human subjective assessment of perceptual quality. The experiment is repeated over multiple pre-trained networks, their layers and for various types of distortions. It can be observed that our proposed extension is much more correlated with human subjective assessment of perceptual quality and thus a superior metric. The superiority is quantified in higher LCC and SROCC scores. A graphical representation is shown in Fig. 3 where it can be seen that the objective metric in Fig.  3-(b) is more correlated with human subjective DMOS compared to the one in Fig.3-(a). The next results will demonstrate that this effectiveness enables our proposed extension to improve perceptual image restoration compared to the traditional and widely used perceptual loss.

x4 Super Resolution Experiments
For our x4 SR experiment, we trained a VDSR on the DIV2K data-set on both the loss functions in Eq.-(9) and -(10) under the exact same training conditions, using the 'ReLU3 2' layer of the VGG-16 for the loss functions. We performed three training and testing trials for each loss function by varying the control parameter α. The higher the α in Eq.- (9) and -(10), the more weight is given to the pixel-wise l 2 loss compared to the l p or l att p loss causing an increase in the SSIM and a decrease in the perceptual quality (higher PI). This behavior is in accordance with the perception-distortion trade-off [3]. Fig. 4 shows that even Table 1: The attention module improves the correlation with human subjective scores of the modified perceptual loss (l att p ) in Eq.(8) compared to the widely used standard perceptual loss (l p ) in Eq.(7) which leads to the conclusion that l att p is a better objective perceptual quality metric. The results are repeated across multiple pre-trained networks, their layers and different types of common distortions as well as their combinations (Multiple Distortions). if the α parameter is varied, our proposed attentive perceptual loss always delivers a significantly better perceptiondistortion trade-off compared to the widely used perceptual loss [11]. At roughly the same average SSIM on the test set,  training with L att p gives delivers images with much better perceptual quality. Furthermore, even at much lower distortions, the perceptual loss fails to achieve as good perceptual quality as our proposed extension delivers at much higher  distortions, indicating a very significant improvement. This result clearly demonstrates the superiority of our proposed extension over the perceptual loss. As the perceptual loss is the benchmark in perceptual image restoration as iterated in [2], the proposed attentive perceptual loss has high practical impact in CNN based image restoration. Fig. 4 shows some crops from the DIV2K test set images restored using the VDSR trained on Eq.- (9) and - (10). It can be seen that at the same level of distortions (SSIM), the attentive perceptual loss (L att p ) improves the perceptual quality (lower PI) compared to the perceptual loss (L p ). Close examination of the crops also reveals that the L att p significantly helps in suppressing checkerboard artifacts, which are a common nuisance while using the perceptual loss. For example, while comparing Fig.5-(c) and -(d), the artifacts are clearly visible in Fig.5-(d) whereas Fig.5-(c) has a much smoother all round texture. Similar observations can be made in Fig.5-(h) and -(i) if the blue ocean background is carefully observed.
In summary, taking inspiration from the HVS, we have proposed an extension to the popular and widely used perceptual loss for image restoration. The results demonstrate that our proposed extension is inherently a better quality metric and it significantly improves the performance compared to the perceptual loss [11]. A significant amount of research has been invested into novel CNN architectures for image restoration but the issue of loss functions is still relatively un-addressed. A few variants of the perceptual loss such as the contextual loss [18] have also been proposed which instead of an l 2 distance between feature maps, calculates an approximated version of the KL-Divergence. Our proposed attention map is by definition applicable to any extension of the perceptual loss that uses deep CNN features as perceptual quality features.

Gaussian De-Blurring Experiment
We perform another image restoration experiment to demonstrate the effectiveness of our proposed spatially attentive extension to the perceptual loss over the classical and widely used perceptual loss [11]. The experiment targets restoration of heavily Gaussian blurred images using the well known DnCNN [26] architecture. The DIV2K training images were corrupted by large standard deviation gaussian blur (σ = 7) and the DnCNN architecture was trained for restoration. The network was separately trained using the perceptual loss (L p ) and the proposed spatially attentive perceptual loss (L att p ). Fig.6 shows that the L att p significantly improves the perception distortion trade-off in comparison to the perceptual loss L p . At the same level of distortion, training with the L att p results in much better perceptual quality. Fig. 7 shows images to demonstrate the improvement in perceptual quality and the perception-distortion trade-off. Considering that the perceptual loss is a standard benchmark and widely used loss function for CNN based perceptual image restoration [2], the improvement is significant and has a wide variety of practical applications.

Conclusions
Taking inspiration from the Human Visual System (HVS), more specifically the spatial frequency dependence of contrast sensitivity in our visual perception, we propose an attention map to improve the widely used perceptual loss for image restoration. Through objective quality assessment experiments, we verify that our proposed attentive perceptual loss is a better perceptual quality metric compared to the perceptual loss, owing to a better correlation with human subjective assessment of quality. Through an x4 Super-Resolution and Gaussian De-blurring experiment, we also 6: The proposed attentive perceptual loss (L att p ) significantly improves the perception-distortion trade-off compared to the widely used perceptual loss (L p ) in a (σ = 7) gaussian deblurring experiment with the DnCNN on the DIV2K data-set. The two different points in each curve are obtained by varying the values of α in Eq.- (9) and -(10) in the main paper and averaging the results over the DIV2K test set. demonstrate that our proposed extension has the ability to deliver a significantly better perception-distortion trade-off compared to the perceptual loss. Considering that the perceptual loss is the benchmark loss function in perceptual image restoration, our proposed extension can effectively be used to train novel image restoration CNNs for perceptually enhanced performance.