{'fraction_non_alphanumeric': 0.03239993319601403, 'fraction_numerical': 0.012748427322830262, 'mean_word_length': 5.177441540577717, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Evaluation of Pre-Trained CNN Models for Geographic Fake Image Detection

Thanks to the remarkable advances in generative adversarial networks (GANs), it is becoming increasingly easy to generate/manipulate images. The existing works have mainly focused on deepfake in face images and videos. However, we are currently witnessing the emergence of fake satellite images, which can be misleading or even threatening to national security. Consequently, there is an urgent need to develop detection methods capable of distinguishing between real and fake satellite images. To advance the field, in this paper, we explore the suitability of several convolutional neural network (CNN) architectures for fake satellite image detection. Specifically, we benchmark four CNN models by conducting extensive experiments to evaluate their performance and robustness against various image distortions. This work allows the establishment of new baselines and may be useful for the development of CNN-based methods for fake satellite image detection.


I. INTRODUCTION
Deep learning has proven to be effective in solving various problems in the field of computer vision [1]. In particular, with the development of new and robust generative deep learning algorithms [2], a powerful new application has emerged known as deepfake. Deepfake consists in generating synthetic media, i.e., images, audios or videos, with manipulated content.
With the introduction of more advanced generative neural networks (such as Autoencoders (AEs) [3] or Generative Adversarial Networks (GANs) [2]), the availability of highperformance computational resources (e.g. GPU), and the easy access to a large number of social medias and public databases [4]- [6], deepfake generation has progressed significantly. Synthetic yet very realistic data generation has indeed improved significantly [7], making it very difficult to distinguish between real and synthesized videos, even for a human observer [8].
Despite their various positive applications [9], there is a potential harm and malicious use of deepfake, that can negatively affect privacy, democracy and national security [10], [11]. Hence, the topic of deepfake detection has gained more attention by the research community [12], [13]. For instance, the Defense Advanced Research Project Agency (DARPA) initiated a novel research direction aimed at focusing academic research efforts to accelerate the development of new and more robust deepfake detection techniques. The National Institute of Standards and Technology (NIST) and Facebook launched the Media Forensics Challenge (MFC2018) and the Deepfake Detection Challenge (DFDC) to promote the development of advanced solutions that detect AI-generated and manipulated videos. As a result, the number of proposed solutions have increased significantly to address this challenging problem of deepfake detection. [14].
Satellite images are a valuable and common source of data for cartography and geographic information science, which are important domains that give humans the ability to analyze and explore the real geographic world. Deepfake geography [15], on the other hand, has emerged as a recent concern which can pose a national security threat [16]. The aim of deepfake geography consists of producing falsified satellite images by replacing an image at one location with another image, while still maintaining a realistic appearance for the entire scene [17]. Despite the remarkable advances in deepfakes detection, research focus on fake satellite images detection is very limited.
Recently, with the widespread growth of the geographic deepfake problem [18], researchers started looking at detecting whether satellites images have been manipulated or not [19]. Among the first attempts to tackle this problem is the introduction of public fake satellite image dataset [15]. The authors proposed a GAN-based image-to-image translation approach to generate fake satellite images. By learning the characteristics of an urban area, a fake satellite image is then generated by mapping the learned characteristics onto a different base map. Moreover, the authors proposed a method based on handcrafted features to determine whether the satellite image is authentic or fake. Following this work, another interesting method called Geo-DefakeHop [19] has been proposed. This method is based on parallel subspace learning (PSL), which consists of multiple filter banks used simultaneously. The authors focused on the higher frequency components to learn the differences between real and fake satellite images.
In this paper, we evaluate the performance of several deep convolutional neural network (CNN) architectures, which have shown state-of-the-art accuracy for general image classification tasks, for fake satellite images detection. We conduct a thorough evaluation using the most promising CNNs with the aim of establishing new baselines and promoting future research in this under-explored area of geographic fake image detection.
The rest of this paper is structured as follows. The relevant related works are presented in Section II. In Section III, the methodology is described. Experimental results and discussions are given in Section IV, and the conclusions are drawn in Section V.
II. RELATED WORK Images/videos of faces are by far the most targeted in the fake multimedia field [6], [20], [21], and very few works deal with fake satellite images. Early studies identify deepfakes by detecting specific artifacts that occurred on the most common first-generation manipulated videos [22]- [24]. For instance, in [24], the authors used statistical differences in color components to distinguish synthetic deepfakes from authentic images. However, it is difficult to handcraft all the most suitable and meaningful features.
Unlike previous deepfake generation techniques, GANbased approaches are able to produce more realistic deepfake videos with high visual quality [25]- [28], making the deepfake detection task more challenging. Therefore, more robust methods have been developed that focus on detecting subtle features and inconsistencies between frames brought by the GAN model [29]. For instance, in [30], the authors proposed a detection method based on color features and a linear Support Vector Machine (SVM) for the final classification, achieving promising results on the NIST MFC2018 dataset [31]. Zhang et al. [32] used the bag-of-words method to extract a set of discriminating features that were fed into various classifiers to discriminate manipulated face images from real ones. A significant drawback of these approaches is that the models have to be re-trained again when simple unseen image perturbation attacks (such as noise, blur, cropping or compression) are presented.
In recent years, it has been shown that deep neural networks tend to perform better than traditional image forensic tools [33]- [35]. Consequently, most developed methods adopted it to automatically extract discriminating features to detect deepfakes. These CNN-based approaches mainly rely on the presence of visual artifacts or inconsistency of intrinsic features between fake and real images/videos [36]. These artifacts can be detected by well established and pre-trained CNN models such as VGG16 [37], ResNet50 [38], and so on. For instance, Rossler et al. [5] used XceptionNet model that was trained on the Faceforensics++ dataset to detect manipulated images. Nguyen et al. [39] proposed the use of capsule networks that takes features obtained from the VGG-19 network [37] to detect forged images and videos. Marra et al. performed in [40] an interesting study in order to classify correctly unseen types of GAN-based fake generated data. The authors proposed a multi-task incremental learning detection method based on the XceptionNet model and achieved promising results.
Despite the progress made on deepfake detection methods in face images and videos, research on fake satellite image detection is still limited, mainly due to the lack of labeled fake satellite image datasets. In [15], the authors draw the public's attention to the potential danger inherent in the progressive evolution of this technology (GeoAI capabilities) and its impact on our perception of the geographic world. For this purpose, they recently proposed the first fake satellite image dataset using a GAN-based image-to-image translation approach. This dataset consists of falsified satellite images from three cities, including Tacoma, Seattle and Beijing. Furthermore, the authors identified 26 salient hand-crafted features that show a significant value difference between authentic and fake satellite images. They classified these features into three classes, which are spatial, histogram and frequency features. These salient features are concatenated and fed into an SVM classifier trained to differentiate fake satellite images from the authentic ones. Chen et al. [19] introduced a learning method for the detection of fake satellite images called Geo-DefakeHop. The idea behind their approach is that GANs can generate high-quality realistic images by reproducing well the low-frequency responses of synthesized images. However, it cannot do the same for high-frequency components due to the complexity constraint. Thus, the authors proposed to use multiple filter banks, named parallel subspace learning (PSL), selected a few discriminant features from each filter bank and grouped them for binary classification.

III. METHODOLOGY
In this section, we start by briefly introducing the considered CNN architectures. Then, we discuss the details of training and tuning these models using the transfer learning technique for the fake satellite image detection task.

A. Deep Convolutional Neural Network Architectures
Taking advantage of the proliferation of large datasets in addition to increasing in computational power, deep neural networks (DNNs) have become the most efficient methods for many computer vision tasks. For instance, in some visual recognition tasks, the DNNs are able to recognize images better than humans.
Over the past few years, many DNNs models have been proposed and have shown top results on the ImageNet dataset [43], which represents the large-scale image classification dataset. Among them, we can cite VGG16 [37], InceptionV3 [41], ResNet50 [38] and Xception [42], which we considered in this study. The number of parameters and the depth of each model are summarized in Table I. In addition, top-1 and top-5 accuracy that refers to model's performance on the ImageNet validation dataset are also provided. Following the success of the AlexNet [44] model, VGG network [37] was the first architecture that relied on small size filters, which improves the CNN performance. This is due to the fact that two 3 × 3 convolutions layers have the same receptive field as a single 5 × 5 convolution, while having fewer parameters and taking less computation. VGGNet has two variants, VGG16 and VGG19, which represent the number of convolutional layers in the model. InceptionV3 [41] is based on some of the original ideas of GoogleNet [45] and VGGNet. In InceptionV3, several techniques for optimizing the network have been exploited, including factorized convolutions, regularization, dimension reduction, and parallelized computations. Deep residual networks [38], shortened ResNet, introduced a concept that allowed the development of much deeper networks. Deeper networks can learn more complex functions and representations of the inputs that usually lead to better performance. However, a deep network still faces the challenges of network degradation as well as exploding or vanishing gradients. In ResNet, residual blocks were introduced, in which the inputs are added back to their outputs with the aim of creating identity mappings. Such identity mappings are performed using the so-called skip or residual connections. Xception [42] takes the principles of Inception to the extreme. The main idea behind Xception is its depthwise separable convolution, i.e., a depthwise convolution followed by a pointwise convolution, which provides better performance than InceptionV3. Xception adjusted the original inception block by mapping the spatial correlations separately for each output channel instead of partitioning input data into multiple compressed chunks and then performing a 1×1 depth conversion to capture cross-channel correlation.

B. Transfer Learning for Geographic Fake Image Detection
Transfer Learning (TL) is a powerful machine learning technique that involves reassigning learned classifiers to new tasks. In other words, the knowledge acquired by the model after training on the data of the first task, encoded in the weights of the model, can be exploited by a new target task, as illustrated in Figure 1. Thus, in this work, we investigate the potential of directly using CNN models pre-trained on ImageNet dataset for fake satellite image detection. To reach this goal, for each considered model, the last FC layer is removed and the rest of the network is considered as feature extractor. Additionally, for classification purposes, three FC layers of size 1024, 512 and 2, respectively, are stacked. For the first three epochs, the weights of feature extraction, i.e., those of pre-trained network, are frizzed, after which they are fine-tuned by continuing end-to-end back propagation. This way, we retrain all layers of the model with fake satellite image dataset.

IV. EXPERIMENTAL RESULTS
To assess the performance of the four considered pretrained CNN models, namely VGG16 [37], InceptionV3 [41], ResNet50 [38] and Xception [42], we experimented with the UW fake satellite image dataset [15]. We compared the performance of these four models to seven methods combining spatial, histogram and frequency features, as proposed by Zhao et al. [15]. In addition, in order to assess the robustness against various image distortions, we also considered the effects of JPEG compression and Gaussian noise on the performance of each model.
The training of the models is performed with back propagation using ADAM optimization with a learning rate lr of 0.0001. A dropout is applied with a rate r equals to 0.2 and each mini-batch contains 32 images. We adopted three commonly used performance criteria to evaluate the models: F1 score, precision and recall [46].

A. Dataset
The UW dataset was introduced in 2021 by Zhao et al. [15] as the first publicly available fake satellite image dataset. The fake satellite images were generated by CycleGAN from three cities (i.e., Tacoma, Seattle and Beijing), while its authentic satellite images are collected from Google Earth's satellite images. UW contains 8064 satellite images, of which 4032 images are authentic color satellite images of spatial resolution 256 × 256, and their fake analog-generated images. Figure 2 illustrates some fake satellite images from UW dataset.
We follow the same experimental setting as described in [15]. The dataset is randomly split into training (90%) and test (10%) sets. In order to fine-tune hyperparameters, the training set is further divided into training (70%) and validation (20%) sets.

B. Results and Discussion
We compared the detection performance of the four models under three scenarios: 1) original images from UW dataset, 2) images distorted using JPEG compression, and finally, 3) images distorted using Gaussian noise. For the second and third scenarios, it is important to note that distortions were only applied to the test set.
1) Detection performance on the original images: Table II shows the detection performance on the original UW test set. From this table, we can see that the CNN-based methods achieves much better performance than the methods of Zhao et al. [15]. In particular, ResNet-50 achieves the best results  with an F1 score of 0.990 and outperforms both InceptionV3 and Xception, although they have relatively the same number of parameters (see Figure3). Similar high performance is  3) Detection performance on the Gaussian noise images: Here, we measure the impact of Gaussian noise on detection performance. For this, the images of the original test set are corrupted with three sigma Gaussian noise levels. The result is shown in Table IV. As can be seen, additive Gaussian noise has a negative impact and the detection performance of all methods drop significantly on distorted images compared to the original versions. The reason is that noise is usually dominant at high frequencies, and we know that the most salient features for detecting fake images are at high frequencies. Thus, the noise disturbs this prominent information.

V. CONCLUSION
In this work, we evaluated the efficiency of CNN-based models for the detection of fake satellite images. To this end, four CNN-based models, namely VGG16, ResNet-50, InceptionV3 and Xception, were pre-trained on ImageNet dataset and fine-tuned on UW dataset to distinguish between authentic and counterfeit satellite images.
Evaluation results in terms of F1 scores, precision, and recall revealed that the ResNet-50 outperformed all other methods, particularly InceptionV3 and Xception. Moreover, we compared the performance of the CNN-based methods against hand-crafted-based methods. Our experiments showed that CNN-based approaches significantly outperform the handcrafted-based methods. We also analyzed the effect of image compression and noise on the detection performance. Typically, these distortions cause a significant drop in performance compared to using the original images. In particular, Gaussian noise has the greatest negative impact on the detection performance compared to JPEG compression. Surprisingly, the detection performance of the hand-crafted-based methods seems to be the least affected by image compression. We plan to release all the data and codes for public use to support the principle of reproducible research and fair comparison.