{'fraction_non_alphanumeric': 0.02982107355864811, 'fraction_numerical': 0.016778450178052563, 'mean_word_length': 4.717461903572321, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Influence of Image Classification Accuracy on Saliency Map Estimation

Saliency map estimation in computer vision aims to estimate the locations where people gaze in images. Since people tend to look at objects in images, the parameters of the model pretrained on ImageNet for image classification are useful for the saliency map estimation. However, there is no research on the relationship between the image classification accuracy and the performance of the saliency map estimation. In this paper, it is shown that there is a strong correlation between image classification accuracy and saliency map estimation accuracy. We also investigated the effective architecture based on multi scale images and the upsampling layers to refine the saliency-map resolution. Our model achieved the state-of-the-art accuracy on the PASCAL-S, OSIE, and MIT1003 datasets. In the MIT Saliency Benchmark, our model achieved the best performance in some metrics and competitive results in the other metrics.


Introduction
Saliency map estimation in computer vision aims to estimate the locations where people gaze in images. The input of the saliencymap estimation task is an image, whereas the output is the saliency map, which is defined as the probability density function of human gaze on the input image, as shown in Figure 1. The model to estimate the saliency map is constructed based on the fixations measured with observers for the target images using an eye tracking system. The fixations of several observers are merged to build the set of the ground truth fixation points, and to create the optimal saliency map by blurring. The estimated saliency maps are evaluated with several metrics based on the ground truth fixation points or the optimal saliency maps. The saliency maps are expected to be used for many applications, including image compression [1], video compression [2], image retargeting [3], image cropping [4], and virtual/augmented reality [5]. In addition, modeling the process of human fixations would be useful for understanding the human visual attention [6].
It is known that the parameters of the models pretrained on Ima-geNet are useful for saliency map estimation [25,28]. This would be because a human tends to look at the centers of objects [36], which are learned to be recognized in the pretrained model for the ImageNet classification task. However, there is no research on the relationship between the image classification accuracy and performance of saliency map estimation. In this paper, the influence of image classification accuracy on saliency map estimation was investigated. Moreover, the effective architectures to estimate the saliency maps were examined based on multi scale images and upsampling layers to refine the saliency-map resolution. The contributions of this paper are as follows: • The influence of image classification accuracy on saliency map estimation was studied. It was found that there is a strong correlation between image classification accuracy and saliency map estimation accuracy.
• The effectiveness of upsampling layers and the multipath architecture was investigated. It is shown that there is no need for upsampling layers, while the multipath architecture is useful for saliency map estimation. • Our model achieved the state-of-the-art accuracy for saliency map estimation in the PASCAL-S, OSIE, and MIT1003 datasets. In addition, our model achieved the best performance in some metrics for the MIT Saliency Benchmark, and the competitive results in the other metrics.
It is noted that this paper is the extended version of our previous paper in ACPR 2017 [28]. Compared with the previous paper, the results on the influence of image classification accuracy on the saliency map estimation were added, in addition to the evaluation results on the MIT Saliency Benchmark. Although the model based on DenseNet [11,13] has achieved the state-of-the-art performance in the ACPR 2017 paper, this additional study led to even better performance using the model based on Dual Path Networks (DPN) [14].
This paper is organized as follows. In Section 2, reviews of the saliency map estimation and the image classification are provided. The architecture of the proposed models, the experimental setup, and the experimental results are explained in Section 3, Section 4, and Section 5, respectively. The conclusions are described in Section 6.

Conventional methods for saliency map estimation
In the last 20 years, many saliency models have been proposed to estimate the locations in images which attract attentions of humans. Most of the conventional models have used low-level features extracted by edge-detectors and color filters or local image statistics. For example, the model proposed by Itti et al. [29] extracts early visual features such as intensity channels, color channels, and orientation channels. These visual features are normalized and integrated. AIM [30] uses coefficients of the basis calculated by ICA in local image patches. The distribution of the coefficients is estimated by the kernel density estimation, which is used for estimating saliency maps based on self-information of the local patches. GBVS [31] exploits channel-wise feature maps computed by linear filtering followed by a nonlinear transformation. To estimate saliency maps, the feature maps are transformed into activation maps and normalized by using the fully-connected directed graph of the feature maps. SUN [32] applies a bayesian framework using local feature maps to estimate saliency maps. The probability distribution over features is learned not from each individual test image but from statistics calculated over the training set of natural images. In DVA [33], the incremental coding length (ICL) using features of local image patches is proposed to maximize the entropy of the sampled visual features. In ICL, unexpected features elicit entropy gain in the perception state and are therefore assigned high energy. The probability function of feature activities of this model is updated dynamically. SIG [34] introduced a simple image descriptor referred to as the image signature. This descriptor can be used to approximate the spatial location of a sparse foreground hidden in a spectrally sparse background. AWS [35] is based on the whitening of lowlevel features and has shown good performance for saliency map estimation.
Although many saliency map models have been proposed using low-level, hand-crafted features as reviewed above, it is difficult to estimate saliency maps of images containing complex scenes [38] and to select the effective low-level features of images for saliency map estimation.

ImageNet classification
In recent years, many models based on convolutional neural networks (CNN) have been proposed and used for various computer vision tasks such as image classification [7][8][9][10][11][12][13][14], object detection [15][16][17], and semantic segmentation [18][19][20]. AlexNet [7] achieved the winning top-5 test error rate of 15.3% in ILSVRC 2012 competition [37], compared to 26.2% achieved by the second-best entry. This model has 8 learned layers -five convolution layers and three fully connected layers. Each of them except for the last fully connected layer is followed by the activation function of ReLU to prevent the activations from saturating. To reduce overfitting in the fully-connected layers, AlexNet uses the regularization method called dropout and local response normalization. SqueezeNet [10] has been proposed as smaller CNN architecture. SqueezeNet has several Fire modules: a module is comprised of a squeeze convolution layer which has only 1×1 filters, and an expand layer which has a mix of 1×1 and 3×3 convolution filters. SqueezeNet achieved AlexNet-level accuracy on ImageNet with 50 times fewer parameters.
VGG [8] obtained the first and the second places in the localization and classification tasks in ILSVRC 2014, respectively. The model has small (3×3) convolution filters which enable the model to increase depth (up to 19 weight layers) effectively. ResNet [9] won the 1st place on the ILSVRC 2015 classification task. In general, although CNN models with deeper layers have higher performance for various tasks, the problem of vanishing/exploding gradients is likely to occur. In order to avoid it, in ResNet, the residual learning framework using identity mapping by shortcuts is proposed. Their deep residual nets are easy to be optimized, and produced results substantially better than the previous networks by greatly increasing the depth. On the ImageNet dataset, the residual nets were evaluated with the depth of up to 152 layers -8 times deeper than VGG but still having lower complexity.
DenseNet [11,13] has shown the competitive results on the four object recognition benchmarks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). In contrast to ResNet, DenseNet combines features by concatenating them along channels. This architecture encourages feature reuse and substantially reduces the number of parameters. Dual Path Networks (DPN) [14] achieved the state-of-the-art accuracy on the ImageNet classification task when this model was proposed. ResNet enables feature re-usage while DenseNet enables new features exploration, which are both important for learning good representations. To take the benefits from both path topologies, DPN shares common features while maintaining the flexibility to explore new features through the dual path architectures. Since DPN achieved the high accuracy with the low computational cost and the low GPU memory consumption, this model is useful not only for research but also for real-world applications.
Fully convolutional neural networks (FCN) [18] have been proposed for semantic segmentation and have achieved the state-of-theart accuracy. Different from the architecture for image classification, FCN replaces the fully connected layers to the convolution layers. In recent years, FCN becomes the standard method for pixel level prediction such as semantic segmentation, instance segmentation, and saliency map estimation.

Saliency map estimation with CNN
CNN models can extract effective features of images for the saliency map estimation automatically. First, eDN [21] has applied CNN to the task of the saliency map estimation. In 2015, DeepGaze I [22] has achieved higher performance than eDN by using the weights of AlexNet trained for the ImageNet object recognition. This model has shown that the parameters of the model pretrained on ImageNet are useful for saliency map estimation. This would be because a human tends to look at the centers of objects [36], which are learned to be recognized in the pretrained model for the ImageNet classification task.
Many models have also applied the transfer learning approaches using deep features since DeepGaze I was proposed. In contrast to DeepGaze I, SaliconNet [23] and DeepFix [24] use FCN based on the VGG architecture which has shown better performance than AlexNet on the image classification task. SaliconNet estimates saliency maps with two image scales to allow the model to have robustness to the size of objects in images. DeepFix has dilated convolution filters [19] to enlarge the receptive field and inception modules to capture multi-scale information. DeepGaze II [25] is also based on the VGG architecture, which uses a center-bias layer to incorporate the prior distribution, and log-likelihood as the loss function. Note that the VGG features of DeepGaze II are not fine-tuned with fixation datasets. These models have improved the performance over DeepGaze I in the MIT Saliency Benchmark [39].
DSCLRCN [26] and SAM-ResNet [27] use ResNet and achieved high accuracy in the saliency map estimation task. Both models use long short-term memory (LSTM) [40] for saliency map estimation in the different ways. In DSCLRCN, LSTM sweeps both horizontally and vertically across the image. This mimics the cortical lateral inhibition mechanisms in the human visual system to incorporate global contexts to assess the saliency of each image location. DSCLRCN uses the features extracted by the model pretrained for scene recognition. In contrast to DSCLRCN, SAM-ResNet uses LSTM to compute an attention map. The feature map extracted by ResNet is input to Attentive Convolutional LSTM that focuses on the most salient regions of the input image to iteratively refine the predicted saliency map. Dilated convolution [20] has been applied to ResNet in both models. DenseNet is used in the model, DenseSal, proposed in our ACPR 2017 paper [28] to extract effective feature maps for saliency map estimation.
In these papers, the performance of the saliency map estimation was evaluated for various architectures based on the CNN

Fig. 2:
Architecture of fully convolutional neural networks for saliency map estimation models of AlexNet, VGG, ResNet, and DenseNet, though the comparison between the CNN models is scarce. Moreover, it is difficult to compare the CNN models from the published papers because the implementation details in addition to the experimental setups are diversified such as architectures, datasets, objective functions, and optimization methods. Therefore, in this paper, the influence of image classification accuracy on the saliency map estimation is investigated with 15 CNN models including AlexNet, VGG, SqueezeNet, ResNet, DenseNet, and DPN under the same architecture and experimental setups.

Architecure
In this paper, we used the architecture based on SaliconNet [23], which has shown high accuracy for estimating saliency maps. The architecture of our model for the saliency map estimation is shown in Figure 2. The model consists of two parts, Main Net and Readout Net. In the model, an image is processed at two scales: the original image and the half size of the image obtained by downsampling of bilinear interpolation. These images at two scales are separately processed by fully convolutional neural networks which share weights (Main Nets). This multipath architecture enables the model to be robust to the size of objects in images. The outputs of the two paths are concatenated along channels after resizing by bilinear interpolation. These concatenated feature maps are fed into Readout Net to output the saliency map for the input image.
In the original SaliconNet model [23], the fully convolutional neural networks based on VGG-16 [8] is used as Main Net, while a 1 × 1 convolution layer as Readout Net. In order to examine the influence of image classification accuracy on saliency map estimation, 15 models were tested as Main Net. The models are based on AlexNet, VGG, ResNet, SqueezeNet, DenseNet, and DPN. The feature maps before the last global pooling layer or the fully connected layers of each CNN model are extracted to be input to Readout Net.
Several types of upsampling neural networks were examined as Readout Net, because the output size of the fully convolutional neural network is smaller than the original input image. The size of extracted feature maps of the CNN models except for SqueezeNet is about 1/32 while that of SqueezeNet is about 1/16, although the size was increased by adjusting the stride of a pooling layer or a convolution layer in some proposed models. Readout Net examined is a multi-layer neural network composed of bilinear interpolation layers (BI), deconvolution layers (DC) [41], or sub-pixel convolution layers (SPC) [42], followed by a 1 × 1 convolution layer and Leaky ReLU [43]. Each upsampling layer resizes feature maps twice, resulting in saliency maps in the higher resolution. The 1 × 1 convolution layer predicts the saliency map from all the concatenated feature maps from the two Main Nets (×1.0, ×0.5). When the number of upsampling layers is N , the size of the predicted saliency map by Readout Net is 2 N times of the feature maps from Main Net. Note that N = 0 means Readout Net corresponds to a 1 × 1 convolution layer, same as Readout Net in the original SaliconNet.

Components of Main Net
The models pretrained on ImageNet for the image classification task were used as Main Net for investigating influence of image classification accuracy on saliency map estimation. The models used in this paper include AlexNet, VGG, ResNet, SqueezeNet, DenseNet, and DPN. The architecture of these models is shown in Figure 3. Each model has several pooling or donwsampling layers between blocks, as shown in Figure 3 (a). Each block consists of several Convolutions as shown in Figure 3 (a), each of which is composed of convolution layers and activation functions such as ReLU with/without batch normalization layers [44]. The architectures within the block are different among the models, as shown in Figure 3 (b)-(e).
• AlexNet [7] has 8 learned layers. These layers are 5 convolution layers and 3 fully connected layers. The output of the fifth layer (the last convolution layer) is extracted to be input to Readout Net. As shown in Figure 3 (b), AlexNet is based on the standard architecture without skip connections such as ResNet.
• VGG [8] has achieved the deep layered neural networks up to 19 layers, by using small (3 × 3) convolution filters instead of large size convolution filters. By stacking several 3 × 3 convolution layers, the receptive field of the convolution can be enlarged with fewer parameters: for example, the stack of three 3 × 3 convolution layers has the effective receptive field of 7 × 7. The output of the last convolution layer is extracted to be input to Readout Net. This model is also based on the standard architecture without skip connections, as shown in Figure 3 (b).
• ResNet [9] has up to 152 layers by using the residual learning framework, where Convolutions in Figure 3  It is noted that the required parameters in ResNet are much smaller than VGG models in spite of the deeper layers such as 152 layers because of the small number of channels. The output of the last convolution layer is extracted to be input to Readout Net. Let x and y be input and output feature maps of a convolution layer in ResNet, respectively. A convolution layer of ResNet can be expressed as where F (x) is a function to represent the residual mapping shown in Convolutions in Figure 3 (c), which is implemented with the following structure: BN , C n×n , and σ denote the batch normalization, the n × n convolution layer, and the activation function of ReLU, respectively. Note that only ResNet-18 has different F as follows: • SqueezeNet [10] with 18 layers has been proposed as a smaller CNN architecture compared with AlexNet, which is based on the standard architecture shown in Figure 3 (b). Each 'Convolutions' in Figure 3 (b), called 'Fire module', consists of a squeeze convolution layer with 1×1 convolution filters, and an expand layer with both 1×1 and 3×3 filters. The output of the last Fire module is extracted to be input to Readout Net.
• DenseNet [11,13] has up to 264 layers, by using skip connections similar to ResNet. In contrast to ResNet where the feature maps from Convolutions are added to the directly connected feature maps from the preceding layer (Figure 3c), DenseNet concatenates the feature maps from Convolutions with those from the preceding layer along channels (Figure 3d). This results in each layer of DenseNet receiving all the outputs of preceding layers because the input of each layer is directly transmitted to the next layer by the skip connection. DenseNet mainly consists of multiple dense blocks composed of dense layers with dense connections shown in Figure 3 (d). Let the l-th dense layer in a dense block be F l and its output be x l . Then, x l can be expressed as follows: where [x 0 , x 1 , ..., x l−1 ] represents the concatenation of the feature maps x 0 to x l−1 . F l is represented as 'Convolutions' in Figure 3 (d), which is implemented with following structure: Since the input of each dense layer has a large number of channels, BN-ReLU-1 × 1 Conv is inserted before the BN-ReLU-3 × 3 Conv to condense the input information by reducing the channels.
The number of input channels in the l-th dense layer is H + (l − 1)K channels, where H is the number of channels in the dense block input and K is the number of channels in the 3 × 3 convolution output, called growth rate. The 1 × 1 convolution and the 3 × 3 convolution reduces the channels to 4K and K, respectively. The output of the l-th dense layer is the concatenation of the input (H + (l − 1)K channels) and the 3 × 3 convolution output (K channels), resulting in the H + l × K channels. When a dense block is composed of L dense layers, the output of the dense block has H + KL channels. Each dense block except for the last one is followed by a transition layer which reduces channels using 1×1 convolution and reduces the resolution of feature maps using average pooling.
• DPN [14] used in the experiments has 131 layers, combining ResNet and DenseNet structures. As shown in Figure 3 (e), the output of 'Convolutions' is divided into two parts: one is to be added with the directly connected feature maps like ResNet, and the other is to be concatenated with them like DenseNet.The number of channels in the input and output for a layer in DPN is same as that in DenseNet, as shown in Figure 3 (d) and (e). The 'Convolutions' in DPN is composed of a 1 × 1 convolution layer followed by a 3 × 3 convolution layer, and another 1 × 1 convolution layer. In the 3 × 3 convolution layer, the grouped convolution is used as proposed in ResNeXt [12], where the feature maps are grouped along the channels before the convolution. This grouped convolution enhances the learning capacity and efficiency of the convolution. The output of the last convolution layer is extracted to be input to Readout Net.

Components of Readout Net
Readout Net examined is a multi-layer neural network composed of bilinear interpolation layers (BI), deconvolution layers (DC), or sub-pixel convolution layers (SPC), followed by a 1 × 1 convolution layer and Leaky ReLU. The last 1 × 1 convolution layer outputs the estimated saliency map using the concatenated feature maps from both two scales of Main Nets (×1.0, ×0.5).
• Deconvolution (DC): Deconvolution is also called as transposed convolution. The operation attempts to directly minimize the reconstruction error of the input image under a sparsity constraint on an over-complete set of feature maps [41]. When DC is used for the upsampling layers in Readout Net, the 1st, 2nd, and 3rd DC followed by a ReLU layer in Readout Net reduce the channels to 128, to 64, and to 32, respectively. Then, the 1 × 1 convolution reduce the channel to 1 to predict the saliency map. The filter size of DC was set to 4 × 4.
• Sub-Pixel Convolution (SPC): SPC [42] is a method to recover a high resolution image from its low resolution counter part with little additional computational cost, by rearranging the data along the channel into feature maps with a convolution operation. When SPC is used for the upsampling layers, each SPC layer reduces the channels to one forth, followed by a 3 × 3 convolution and a ReLU layer. Then, the 1 × 1 convolution predicts the saliency map from the output of the last SPC layer.
• Bilinear Interpolation (BI): Since each BI layer in the upsampling network resize a feature map twice while maintaining the feature-map channels, GPU was out of memory when 3 upsampling layers were used for the all channels of outputs of Main Net. In the case of BI, the order of upsampling and projection (1 × 1 convolution) can be inverted without any influence on the output. Therefore, the concatenated feature maps from Main Nets are first processed by the 1 × 1 convolution to output the 1-channel feature map, followed by the BI upsampling layers.

Experimental setup
The datasets used in our experiments are explained. The network was fine-tuned on training data in Salicon Dataset and OSIE, except for the experiment for MIT Saliency Benchmark (Table 8), where MIT1003 was also used for training. The datasets of PASCAL-S and MIT300 (MIT Saliency Benchmark) were used for the evaluation. For most of the experiments, Pascal-S was used for the evaluation because the reference paper [23] of the SaliconNet model also used this dataset as one of the evaluation datasets. The evaluation results for different databases in addition to Pascal-S are described in 5.5 for the comparison with the state-of-the-art models.
• Salicon Dataset [45] consists of 10,000 images for training and 5,000 images for validation. Each image was viewed by 60 observers. Different from other fixation datasets, this dataset is largescale mouse-tracking data through Amazon Mechanical Turk. Although this dataset is not the fixation dataset, it is well known that the distribution of mouse tracking points is similar to the distribution of fixations [45], and that the parameters of the model trained on Salicon Dataset is useful for saliency map estimation [25,28]. The training data in the Salicon Dataset was used for training the network, whereas the validation data was used for evaluating models in 5.1 (Figure 4).
• OSIE [46] consists of 700 natural indoor and outdoor scenes, aesthetic photographs from Flickr and Google. The 500 images were used for training and the 200 images for validation. The fixations were measured while 15 observers looked at image for 3 seconds. In order to obtain the optimal saliency maps from the fixations, the fixations for all observers were collected and blurred using the gaussian filter with the standard deviation equivalent to 1 degree in the visual angle. The training data in the OSIE dataset was used for training the network after the training with Salicon Dataset, whereas the validation data was used for evaluating the estimated saliency maps in Table 9.
• PASCAL-S [47] is built on the subset of the validation data in the PASCAL VOC 2010 [48] segmentation challenge. PASCAL-S contains 850 natural images. The fixations were measured while 8 observers looked at an image for 2 seconds. This dataset was used for evaluating the estimated saliency maps to investigate the generalization capability of the models trained on OSIE dataset. Note that this dataset was not used for training. This procedure is same as [23].
• MIT1003 [49] includes 779 landscape images and 228 portrait images. The fixations were measured while 15 observers looked at an image for 3 seconds. This dataset was used for the evaluation in Table 10. This dataset is also used as training dataset for MIT Saliency Benchmark in Table 8, where 902 images were used for training and 101 images for validation.
• MIT300 [39] was the first data set with held-out human eye movements, and is used as benchmark test data in MIT Saliency Benchmark in Table 8. This dataset consists of 300 natural indoor and outdoor scenes. The fixations were measured while 39 observers looked at an image for 3 seconds.
Saliency map is evaluated in several metrics. Each metric is categorized as the location-based or distribution-based metric. These types depend on whether the ground truth is represented as discrete • AUC-Judd is a location-based metric. The true positive rate (TPR) is calculated as the proportion of fixations falling into the thresholded saliency map [50]. The false positive rate (FPR) is calculated as the proportion of no-fixated pixels in the thresholded saliency map. After calculating TPR and FPR at each threshold, Area Under the Curve (AUC) is calculated for the curve of TPR against FPR.
• AUC-Borji is a location-based metric. TPR is calculated in the same way as AUC-Judd [50]. FPR is obtained by calculating the proportion of negatives in the thresholded region, where the negatives are collected uniformly at random. AUC for the curve is calculated as AUC-Borji.
• shuffled-AUC (sAUC) is a location-based metric. TPR is calculated in the same way as AUC-Judd [50]. FPR is calculated based on the negatives which are determined by fixation points of all the other images in the dataset. AUC for the curve is calculated as sAUC. This metric can eliminate the center bias property of human fixations, which is the property that a human tends to look at the center of an image.
• Kullback-Leibler Divergence (KL) [51] is a distributionbased metric, which measures the difference between two probability density functions. This KL was used as the loss function to train the neural networks to compare our proposed models with SaliconNet, one of the competitive methods, which has used KL as the loss function.
• Normalized Scanpath Salience (NSS) [52] is a location-based metric. Each saliency map was normalized to have the zero mean and the unit standard deviation. The values of the normalized saliency map at fixations are averaged to evaluate the saliency map.
• Correlation Coefficient (CC) [53] is a distribution-based metric, which is also called Pearson's linear coefficient. This metric is the linear correlation coefficient between the estimated saliency map and the optimal saliency map.
• Similarity (SIM) [54] is a distribution-based metric. This metric is computed as the sum of the minimum values of the estimated saliency map and the optimal saliency map for all pixels.
• Earth Mover's Distance (EMD) [50] is a distribution-based metric. EMD measures the spatial distance between two probability density functions.
First, the influence of image classification accuracy on the saliency map estimation task was investigated in 5.1. The 15 models based on various architectures in Figure 3 were tested in this experiment. These models are listed in Table 1. Since the computational cost was too high to evaluate all the models with the multipath architecture in Figure 2, the single-path structure for full-size images (× 1.0) was used with the 1 × 1 convolution as Readout Net in this experiment. The pretrained models that are provided by torchvision [55] were used. As for DenseNet-264 and DPN-131, we used pretrained models that are provided by the authors [13,14]. Due to the high computational cost, the validation dataset of Salicon Dataset was used for the evaluation. Since Salicon Dataset was used for training the network, evaluation with the validation dataset of Salicon Dataset can suppress the evaluation time. DenseNet-161 and DPN-131 gave high accuracy in this experiment for the saliency map estimation, so that these two models were used in the following experiments.
Using the two models (DenseNet-161 and DPN-131), the effective architecture and training procedure for saliency map estimation was studied with the structure of Figure 2 in 5.2-5.6. In these experiments, the network was pretrained with the image classification task followed by fine-tuning with Salicon Dataset and OSIE. Then, Pascal-S was used for the evaluation. This procedure was same as the reference of the Salicon model [23]. In Section 5.5, the different databases were also used for the evaluation to compare the accuracy with the state-of-the-art models. The architectures using DenseNet-161 and DPN-131 with the structure of Figure 2 are called DenseSal and DPNSal, respectively. The detailed structures are shown in Table  2 and 3. The DenseNet-161 includes 4 dense blocks as shown in Table 2. Although the original DenseNet-161 has the average pooling layer with the stride 2 between the 3rd and 4th dense blocks as in the other pooling layers, the stride was set to 1 at the layer to predict the saliency map in the higher resolution. The feature maps before the final layer in DenseNet-161 (Global Pooling in the original DenseNet-161) are used as the output of Main Net in the proposed model. The output size of the Main Net is the 1/16 of the input image, whereas it is the 1/32 in the original DenseNet-161. The details of the DenseSal structure are shown in Table 2.
The DPN-131 was also modified to have higher resolution in the output by adjusting the stride in the 3 × 3 convolution layer in the 4th block, as shown in Table 3. In this model, the dilated convolution [20] was employed to enlarge the receptive filed in the 4th block. The output size of the Main Net is the 1/16 of the input image, whereas it is the 1/32 in the original DPN-131.
KL and RMSprop [56] were used as the objective function and as the solver, respectively. PyTorch [57] was used as the implementation platform. The mini-batch size and learning rate were set to 1 and 10 −5 during training, respectively. We subtract the per-channels mean value of training images from each image as pre-processing. Note that the 2015 version of Salicon Dataset was used in the experiments except for the experiments in Figure 4, where the 2017 version of Salicon Dataset was used. Since it was found that the performance of the model trained on the 2015 version of the Salicon Dataset was better than that on the 2017 version, we decided to use the 2015 version in most of the experiments.

5
Experimental results

Influence of image classification accuracy on saliency map estimation
The 15 models shown in Table 1 were examined to study the relationship of the saliency map estimation accuracy with the image classification accuracy. The single-path architecture with 1 × 1 convolution as Readout Net was used in this experiment to suppress the computational cost. Main Net of the each model was initialized with the weights trained on ImageNet, followed by fine-tuning with the training dataset of Salicon Dataset. The validation dataset of Salicon Dataset was used for the evaluation in this experiment to suppress the computational cost. The learning rate was set to 10 −5 for first 10 epochs and 10 −6 next 10 epochs in this experiment. It is shown in Figure 4 that there is a strong correlation between   achieved the best performance in all metrics. It is seen from these results that the ability of object recognition is important for saliency map estimation.    Figure 2 (SaliconNet model [23] with N=0 in Readout Net) was used as the baseline. In the default settings, the networks pretrained with ImageNet were fine-tuned with Salicon Dataset and OSIE. Pascal-S was used for the evaluation, as in [23]. First, VGG-16, DenseNet-161, and DPN-131 were compared using a 1 × 1 convolution layer as Readout Net (N=0). The sizes of outputs in these models are the 1/16 of the input images. The model using VGG-16 is the same as SaliconNet. In Table 4, these models were compared by evaluated with KL on the PASCAL-S dataset, though all the results for the other metrics are also provided in Appendix. The KL metrics was selected to be shown in the text since the metric was used as the objective function during the training.

Comparison of architectures
The KL values of DenseNet-161 and DPN-131 with the multipath structure were lower than VGG-16, meaning that the accuracy of estimating saliency maps was improved by using DenseNet-161 and DPN-131, while the accuracy of DenseNet-161 and DPN-131 were same. In addition, it was confirmed that the accuracy was improved by using two scales (×1.0 and ×0.5) in these models.
In addition to the accuracy, the computational costs were compared among these models, as shown in Table 5. As can be seen from the table, the computational costs for DenseNet and DPN were about 2.5 times and 5.2 times of VGG, respectively. The cost in the single path with the half-size image was about the half of that in single path with the full-size image, while the cost in multipath was about 1.4 times of the single path with the full-size image. Thus, there was a trade-off between the accuracy and the computational cost.

Comparison of training datasets
In CNN, the overfitting to the training data can be suppressed by using the initialization of the network parameters trained with a large amount of image data such as ImageNet. For the saliency map estimation, it is known that the parameters of the model trained on ImageNet for image classification and Salicon Dataset are useful [23,25]. Therefore, the influence of datasets used for training the network was investigated. The evaluation results with KL on Pascal-S after the training is shown in Table 6 for the model with the multipath architecture and a 1 × 1 convolution layer as Readout Net (see Appendix for the other metrics). From the results in Table 6, it is clear that the model parameters trained on ImageNet were most It is also shown that the models with DenseNet-161 and DPN-131 achieved better performance than the model with VGG-16 when the ImageNet was used for the training. On the contrary, their performances were worse than VGG-16 without the ImageNet training. The reason of these results would be overfitting to the training data in DenseNet-161 and DPN-131 with small training dataset such as OSIE. From these results, not only the architecture of the networks but also the parameters pretrained with the ImageNet classification task are important for the saliency map estimation.
In the following experiments, the models were trained by Ima-geNet, Salicon Dataset, and OSIE, in order unless otherwise stated. During the training with OSIE, the input images were resized to 640×480 due to the GPU memory limitation.

Comparison of upsampling layers in Readout Net
Readout Net aims to convert the features extracted by Main Net into a saliency map. Since the output size of Main Net is much smaller than the original input image, the 4 types of upsampling neural networks were examined to estimate the saliency map in the higher resolution. Figure 5 is the results on the PASCAL-S evaluated by KL used as the objective function. The results for the other metrics are shown in Appendix. It can be seen from Figure 5 that most of the results for DenseSal and DPNSal showed higher accuracy than the results on SalinonNet (VGG-16, N=0) shown in the doted lines on the graph. The accuracy on DenseSal and DPNSal was not improved as the upsampling layers increased. This result indicates that the upsampling layer is not necessary in the saliency map estimation task. Different from semantic segmentation or instance segmentation, the fine prediction is not important because the saliency map is blurred by convolving ground truth fixation points with a gaussian filter. In addition, the type of the upsampling networks did not affect the accuracy, as can be seen from the graph.

Comparison with state-of-the-art models
The results of DenseSal and DPNSal with the 1 × 1 convolution for Readout Net were compared with the conventional methods on various evaluation metrics on PASCAL-S, as shown in Table 7. ITTI [29], GBVS [31], SUN [32], DVA [33], SIG [34], and AWS [35] are the methods using manually designed image features. Salicon-Net and DeepGase II based on CNN showed the high accuracy in the results, where SaliconNet was evaluated for both authors' implementation through the webtool [23] and our implementation (VGG-16, N=0). As can be seen from the table that DenseSal or DPNSal achieved the state-of-the-art accuracy on all the evaluation metrics. Thus, it was confirmed that DenseNet and DPN were useful not only for image classification but also for the saliency map estimation.
In order to compare the performance of our models with the state of the art methods, the saliency maps for the MIT300 were submitted to MIT Saliency Benchmark [39]. DenseSal and DPNSal with the initialization of weights by the ImageNet classification task were fine-tuned with the datasets of Salicon Dataset, OSIE, and MIT1003, in order. The images in MIT1003 and MIT300 were resized smaller than or equal to 640 × 480 pixels or 480 × 640 pixels. It can be seen from Table 8 that DenseSal achieved the best performance in KL and EMD, whereas the competitive results in the other metrics. DPNSal also achieved the best performance in SIM, sAUC, CC, and NSS, with competitive results in the other metircs.
Furthermore, DenseSal and DPNSal were compared with Sali-conNet [23] on the validation data in OSIE and MIT1003. In these experiments, the networks were fine-tuned using Salicon Dataset and the training dataset of OSIE with the initialization of the ImageNet classification task, so that MIT1003 was untrained database. Only the four metrics (SIM, CC, NSS, KL) were used for the evaluation because it takes time to calculate AUC-based metrics and EMD for these databases with the large number of fixations. The results are shown in Table 9 and 10. It can be seen from the results that Dens-eSal and DPNSal outperformed over SaliconNet by a large margin. In these databases, DPNSal were better than DenseSal in all the metrics examined.

Examples of saliency map estimation
The examples of saliency map estimation are shown in Figure 6 In (a), the surfer in the center of the image is focused on by the observers. Although SaliconNet cannot estimate this area with high probability, DenseSal and DPNSal can capture this information. In (b), a squirrel is in the center of the image and is focused on by observers. Estimated probability well concentrated on the squirrel in the order of DPNSal, DenseSal, SaliconNet. DPNSal can capture the baby face in the image corresponding to the ground truth fixation map in (c), while SaliconNet cannot capture this. In (d), although the activations were observed in the saliency maps by SaliconNet and DenseSal at both the face and the small car in the background, DPN can focus on the face as the actual fixations. In (e), the human fixations concentrated on the end of passages. Although SaliconNet and DenseSal were unable to capture this information, DPN can do that. This result implies that DPN can learn the information even though there are few images in the training dataset. However, there are some example images of the passages which are not able to be estimated even with DPN, such as (f).
In (g), the actual fixations mainly concentrated on the texts, while the activations were observed in the saliency maps at the animal illustrations. To solve this problem, the model must consider the relationship and priority of fixations among objects in an image. In (h), the actual fixations also mainly concentrated on the texts. However, all models cannot capture that. The reason would be the number of images including handwritten characters is insufficient in the training dataset. In natural images without eye-catching objects such as (i), human fixations are often gathered in the center of an image, while the networks do not model this center-bias property.

Conclusions
In this paper, the effective architecture was investigated for saliency map estimation, in addition to the influence of the image classification accuracy on the saliency map estimation. First, it is shown that there is a strong correlation between image classification accuracy and saliency map estimation accuracy. In addition, it was found that not only the architecture but also the initialization strategy using the weights pretrained with the Ima-geNet classification task were important for estimating the saliency maps. These results indicate that the model which is pretrained with the ImageNet classification and has achieved high performance on the classification task is also useful for the saliency map estimation task. The reason for this would be that human fixations often concentrate on objects in image, while the model pretrained on Ima-geNet can react on many objects in images because ImageNet has a wide variety of object categories. If the model is initialized with Therefore, if the model trained on a large fixation dataset including a lot of objects such as ImageNet, it would be possible to achieve good performance for saliency map estimation without pretraining on ImageNet for the image classification task. Second, the effective architecture and training procedure for saliency map estimation was investigated using DenseSal and DPN-Sal. It was confirmed that the multipath architecture was useful because the network was able to be robust to the size of objects in images. Moreover, it was found that the upsampling layers to refine the resolution of saliency maps were not necessary for the saliency map estimation because the saliency maps were created by blurring the fixation points. DenseSal and DPNSal achieved the state-of-theart accuracy on PASCAL-S. Moreover, the models achieved the best performance in MIT Saliency Benchmark in most of the metrics.
In the future, the architecture that can find more complex structures should be studied for further improvement of saliency map estimation accuracy. From the results in this paper, the higher accuracy a model achieved in the image classification task, the higher saliency map estimation accuracy is expected in the saliency map estimation task using the model. However, it is thought that there is a limit to accuracy using such model. For example, the human fixations tend to concentrate on objects or regions which are looked at by a person in images [59]. Since this tendency is more complex than the tendency that the objects are often looked at by a human, the conventional methods including our model cannot estimate such information. To solve this problem, the model must estimate the gaze direction of a human in an image and capture objects or regions which are looked at by the person. Objective functions for training the models for saliency map estimation also need to be investigated. It is known that the metrics measure qualitatively different things. Therefore, if we use a metric as loss function, we will get better results in the metric but worse results in other metrics. This is why loss functions and metrics for saliency map estimation need to be studied more. Another future work may include the development of a network architecture in consideration of the center-bias property of human fixations [25].