{'fraction_non_alphanumeric': 0.022811725846407928, 'fraction_numerical': 0.01416701073492981, 'mean_word_length': 4.827518796992481, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SeeTheSeams: Localized Detection of Seam Carving based Image Forgery in Satellite Imagery

Seam carving is a popular technique for content aware image retargeting. It can be used to deliberately manipulate images, for example, change the GPS locations of a building or insert/remove roads in a satellite image. This paper proposes a novel approach for detecting and localizing seams in such images. While there are methods to detect seam carving based manipulations, this is the first time that robust localization and detection of seam carving forgery is made possible. We also propose a seam localization score (SLS) metric to evaluate the effectiveness of localization. The proposed method is evaluated extensively on a large collection of images from different sources, demonstrating a high level of detection and localization performance across these datasets. The datasets curated during this work will be released to the public.


Introduction
Seam carving is a popular image manipulation technique [1,3,45] that is effective for content aware image retargetting. In seam carving, the input image is resized by removing or inserting "seams" which are defined as connected pixel paths from top-to-bottom or left-to-right. These seams are chosen by their optimality according to an energy function computed for each pixel. That is, the optimal seam is the seam with the lowest energy along its path. Commonly used energy functions in seam carving are computed by measuring the contrast of a pixel with its neighbors. Removing an optimal seam has fewer artifacts in resized images than a randomly chosen seam, and protects image content that is highly textured. Seam carving can be extended to remove entire objects from images by assigning the energy of object pixels to a low value such that the seams forcibly pass through the object marked for removal. Since seam carving leaves a large percentage of pixel values untampered, it poses a challenge to image forgery detection. Seam carving is included as a feature in popular image editing software such as GIMP [31] and Photoshop [53]. The ease of access to these programs along with increasing availability of satellite image data from commercial satellites presents a growing problem for organizations that rely on accurate satellite data. Satellite images have been manipulated to influence public opinion such as in the Malaysia Airlines flight incident [26], nighttime flyovers of India during the festivals [9], and fake spliced images of bridges [13].
Furthermore, satellite imagery is particularly susceptible to seam carving based image manipulations. One reason is that in satellite imagery, objects of importance generally occupy fewer pixels than conventional images and consequently require fewer seams to remove, reducing the potential for visual artifacts in seam carved satellite images. In Figure 1, we show an example of object removal where we remove a set of buildings from the original image. Moreover, objects can be displaced to change their geographical coordinates (latitude, longitude) using seam carving much more easily in satellite imagery than typical images. Since satellite images are captured from high altitudes, they often have large, smooth regions that are ideal for seams to pass through, making them good candidates for object displacement while preserving visual image quality. In Figure 4, we show an example of object displacement accomplished by removing seams on one side of the building and inserting on the other. Displacing objects can be especially malicious in satellite imagery, where a single pixel can correspond with as much as 60m 2 of ground area. In applications such as military target acquisition, this can be the difference between a successful and unsuccessful mission.
While there are several works proposed to detect seam carving based image forgery [11,15,21,32,38,39,48,56,57,58,59], to our knowledge, none of these methods have investigated localization of removed and inserted seams at the pixel level, and their design is restricted to classification at the image level, or classification and localization at a patch level. We develop and evaluate our method on satellite imagery as a case study due to the potential ramifications of seam carving based manipulations.
Towards addressing the above challenges, we propose a two stage, deep learning based seam carving detector with two key advantages over existing methods. The first advantage is the ability to localize the seams at pixel level resolution. This is invaluable in discovering the extent of potential manipulations in satellite imagery. By considering the location of seams, one may be able to discover not only that an object may have been removed in an image, but where that removed object used to be. The second advantage is generalizability. Since image level classifiers are trained on original satellite images, they are prone to become specialized to the distribution of data they are trained on. We demonstrate that our method is generalizable to not only training dataset distribution but different seam carving techniques. The main contributions of the paper are: 1. We propose a method for robust detection of seam carving manipulations and accurate localization of seams removed or inserted by seam carving in satellite imagery.
2. We develop the seam localization score (SLS), a specialized metric to better evaluate the localization performance of seam carving detectors on specific seams.
3. We have created unique, large seam carving datasets that we plan to release to the public * .
The rest of this paper is organized as follows. In Section 2 we review existing forensic approaches related to our work. Section 3 gives an overview of seam carving and several variations on its original formulation. Section 4 details our proposed framework for localized detection of seam carving forgeries. Section 5 describes the datasets that are curated to carry out our seam carving forensic experiments. Metrics that are used to evaluate our models are described in Section 6. Experimental setups and results are detailed in Section 7. Finally, we conclude the paper in Section 8 by reviewing the pros and cons of our method and possible research areas that merit further exploration.
Most image forensics techniques that have been developed so far target consumer images [7,8], which significantly differ in nature from satellite sensors (e.g. different compression schemes, color channels, orthorectification based post processing and so on). Furthermore, it has been observed that many of these techniques do not perform well when naively applied to overhead images [2,5,16,37,50,55]. To address this issue, several forensic techniques that work for satellite/overhead imagery have recently been proposed [5, 23,24,55]. One method uses handcrafted watermarks to detect manipulations in satellite images [23]. Although this method is quite effective, it cannot be utilized if the watermark is not inserted at the time of image acquisition by a trustworthy source. Another technique leverages conditional GANs to detect and localize splicing forgeries in satellite images by estimating a forgery mask [55]. A second GAN based technique [24] encodes patches from an image into a low dimensional vector space that are used as input into a support vector machine (SVM) to detect the presence of forgeries at a patch level. Finally, Sat-SVDD [5] is a kernel-based one-class classification method that detects splicing forgery with the help of support vector data description (SVDD). In comparison to these methods, our paper explores the challenging case of detection and localization of seam carving based manipulations in satellite images.
There have been several works proposed over the past decade to reveal traces of seam carving in digital im-* Links to the dataset will be shared in near future. ages [11,15,21,32,38,39,48,56,57,58,59]. These include methods using steganalysis [48], hashing [15,32], local binary patterns (LBP) [57,58,59], and deep learning based methods [11,38,39,56]. In [21], the authors propose a method based on block artifacts to estimate the location of an object removed using seam carving. However, this method mainly focuses on finding a region in a seam carved image where an object has been removed and does not produce localization maps where seams have been removed/inserted. One deep learning method proposes a Convolutional Neural Network (CNN) based approach to perform image level classification using local binary patterns [11]. Another CNN based approach uses a customized network that learns and uses more effective features via joint optimization of feature extraction and pattern classification [56]. These two proposed methods perform image level classification to detect the presence of seam carving manipulation. One method that performs localization at a patch level employs a CNN called LFNet that is specifically designed to learn low-level features, capturing local artifacts from seam carving based image resizing [38]. The authors in [38] improved their network design with a new architecture, ILFNet [39]. The authors in [40] proposes a two stage model where stage-1 performs patch level localization and stage-2 performs image level classification. The method proposed in this paper localizes seams at a pixel level as opposed to the patch level strategies in prior work and focuses on satellite imagery, which is more resistant to seam carving artifacts than consumer images. In contrast to these proposed methods, our approach both detects images that are seam carved and also generates a seam localization mask which highlights pixels on the manipulated image where seams have been carved or inserted.

Overview of Seam Carving
Seam carving is a content-aware image retargetting algorithm proposed in 2007 [3]. A seam is defined as an optimal 8-connected path of pixels from top to bottom or left to right. The optimality of a seam is determined by assigning an energy value to each pixel in a given image. One possible energy function proposed in [3] is the gradient magnitude, also termed as backward energy.
where I(x, y) is an image indexable by row x and column y, and E(I) is the energy map of I. Then, the optimal vertical seam for an m by n image can be defined as The optimal seam is then the seam that minimizes the energy function over all possible seams, This optimal vertical seam can be found through dynamic programming, computing a minimum cumulative energy matrix M (x, y) for all possible seams through (x, y) by traversing from the second row to the last row: Seam carving can be applied to remove objects by setting energy map values from a user-provided mask to a "low" value, and successively removing seams until the object is removed. The user can insert seams to restore the original image size, as well as preserve objects through a mask that tags regions to have a "high" energy. Figure 2 is one such example of this object removal strategy and perceptually it is very difficult for a human to deduce that objects have been removed in this image. Seam carving can also be utilized for object displacement in a satellite image by marking the region on one side of an object for seam removal and then inserting seams on the other side of the object after the seams have been removed (Figure 4). In this way, objects can be "shifted" to a different GPS location with very little perceptual difference.

Other Seam Carving Methods
There have been several modifications to the original seam carving algorithm,. One such modification was proposed in 2008 [45] that considers the energy introduced into the resized image by bringing previously non-adjacent pixels together by removing seams. This formulation, termed as forward energy, computes the energy map by looking at the differences in pixel values depending on the direction of the potential seam, defining three possible cases: up, up and to the left, and up and to the right.
Then, the cumulative energy matrix can be updated as: where e(x, y) is an optional additional energy measure.
Another seam carving variation proposes using an importance map based on salient region detection [1]. In this particular formulation, salient regions are uniformly identified considering global contrast as opposed to local edges in the two original methods outlined above. This prevents the necessity to recompute the energy map after each optimal seam is found. The saliency map used in place of the energy map also incorporates color information, which the two methods above omit. To calculate the saliency map, the image is converted into Lab color space. Then, the final map is the Euclidean distance between the average pixel vector and a Gaussian blurred version, approximated by a 5 x 5 binomial filter (both in Lab space): The final variation of seam carving that we explore is called seam merging [35]. This method merges a two-pixelwidth seam element into one new pixel during image reduction and inserts a new pixel between the two pixels during image enlargement. This algorithm utilizes importance and structure energies to define seam optimality, as well as an additional energy term that suppresses artifacts generated by excessive reduction or enlargement from repeated merging or inserting.

Proposed Approach
Here, we describe our approach for localized seam detection. At its core, the problem can be formulated as binary classification, where we would ultimately like to predict whether the given image is seam carved or not. In order to achieve this, we propose a two stage framework. In Section 4.1, we develop localization models that flag regions of the image that are seam carved. In Section 4.2, we perform a final classification whether the image has been manipulated by seam carving by fusing the outputs of stage 1.

Stage 1 -Localization of Seams
In order to obtain localized detection of seams in an image we implement a fully convolutional network to learn a mapping from satellite image to seam mask, where the seam mask contains the locations of removed or inserted seams as described in Section 5.1. U-Net has been extensively used in image segmentation [44], and its architecture is composed of a contracting path (encoder) followed by an expanding path (decoder). Encoder maps the input image to a feature vector, which will then be used by the decoder to flag seam carved regions of the input image. The basic encoder-decoder model's localization power is enhanced in U-Net by applying skip connections between encoder feature maps and decoder outputs. We chose Efficient-NetB7 [49] as our baseline network since it performed the best when compared to other standard networks (see Section 7.1 for comparative experiments) EfficientNet is a baseline network composed of sequential mobile inverted bottleneck convolutions (MBConv) blocks that can be scaled up to improve accuracy at the cost of increased computation. This generates a family of models from Efficient-NetB0 to EfficientNetB7 with the EfficientNetB7 model being the most accurate, but also computationally the most expensive. EfficientNetB7 also outperforms other encoder architectures when used for pixelwise classification, such as in [4]. For the decoder architecture, we use four sets of transposed convolutional, batch norm, dropout, and ReLU layers to upsample encoded features back to the size of input image. In Figure 3, we show an overview of our proposed method, where we see that the localization stage is comprised of two pixelwise classifiers -one model trained for seam removal detection and another model trained for seam insertion detection, both using U-Net and Efficient-NetB7. On a test image, these two models output predicted seam localization masks flagging the regions where seams have been removed and inserted.

Stage 2 -Classification
The pixelwise classifiers described above output two seam prediction masks with the same dimensions as the input image, one localizing removed seams and the other inserted seams. Both these masks are concatenated and fed as input to a standard CNN (ResNet50 [22]) to perform vanilla binary classification to identify if an image has been seam carved or not.

Datasets
In this section, we give a brief overview on the characteristics of source datasets used, and how we generated our forgery datasets. Three common satellite imagery datasets (xView [27], xBD [20], and Orbview-3 [17]) have been used to evaluate our method. The xView dataset contains 1, 127 images at varying high resolutions. Using these high resolution images, we generated a dataset of 53, 943 images by randomly cropping 512 x 512 regions from the original dataset. When dividing the images into a train-test-val split, all 512 x 512 images from a given xView sample are assigned to the same split. For xView, we allocated 70% of the images into training set, 15% into validation set, and remaining 15% into test set. The xBD dataset contains a total of 22, 098 pre-disaster and post-disaster RGB images of size 1024 x 1024, where we've preserved the original dataset's train-test-val split of 80:10:10 and randomly cropped 512 x 512 regions as with the xView dataset. The xBD dataset also contains ground truth building masks that can be used in seam carving object removal. These building masks are quite small and make up an average of 0.37% of all pixels in a particular training set image. On average, a building mask occupies 916 pixels 2 . Since the xBD dataset contains satellite images at 0.3m per pixel, these building objects correspond with around 82m 2 of ground area despite taking up a small number of pixels. In the most extreme cases, a building mask can take as little as 0.9m 2 up to 15, 943m 2 of ground area. Finally, we curated a third dataset consisting of 48, 000 Orbview-3 images by randomly selecting non overlapping 512 x 512 regions. The train-test-val split is again maintained to be 80:10:10. While the xView and xBD datasets contain 8 bit RGB images (image intensities are in range 0 to 255), the Orbview-3 dataset has single channel 16 bit images. These three datasets provide variation in the geographical location of the images as well as different levels of color depth.

Ground Truth Seam Masks
Ground truth seam masks for training pixelwise classifiers are generated while seam carving original dataset images. As we remove seams from the original image, all the preserved pixel locations that were adjacent to removed seams are marked, indicating manipulation. As we insert seams into the image, all the pixel locations of the inserted seam are flagged. A visual example of a ground truth seam mask is shown overlaid on the resulting seam carved image in Figure 1b.

Pixelwise Classification Datasets
Datasets for pixelwise classification are generated by seam carving pristine samples to remove the top 10% of optimal seams and inserting seams to restore the original image dimension. All of our models are trained on 512 x 512 images, cropped from top left of the seam carved images. Ground truth seam masks are generated as described in Section 5.1 and similarly cropped. For the remainder of the paper if unspecified, forward energy is used to define the optimality of seams and examine the generalization capabilities of our models across several seam carving variants in Section 7.2.

Stage 2 Classification Datasets
Since we train models from each stage independently, stage 2 datasets are easily generated once pixelwise classifier training is complete. We obtain the prediction masks from trained model inference on cropped seam carved and original images to form a dataset of manipulated and pristine samples required for the image level classification.

Evaluation Metrics
In this section, we briefly describe the evaluation metrics used to select and assess our models. Since we're operating in a binary classification setting at a pixel level in the localization stage and at image level in the classification stage, we use metrics based on confusion matrices. For a dataset of N samples, the i-th sample will have a confusion matrix: where TP is the number of true positives, FP is the number of false positives, FN is the number of false negatives, and TN is the number of true negatives. Then, the pixelwise accuracy for the i-th image is The confusion matrix and the accuracy over the entire dataset is computed from cumulative confusion matrices for every sample. However, since seam carving based manipulations tend to remove less than 10 percent of the pixels from source datasets, pixelwise accuracy is an inadequate representation of the performance of our models as a naive method predicting all negatives will achieve above 90% accuracy. To address the inherent imbalance in our generated datasets, we prioritize several more relevant confusion matrix derived metrics. For the rest of this section, we report these metric formulae at the dataset level, knowing that we can compute image level statistics as needed. A few metrics that attempt to address class imbalance are precision, recall and F1 score.
F1 Score = 2 · Precision · Recall Precision + Recall While useful, these three metrics are inherently biased towards the positive class, and independent of the number of true negatives. For seam carving localization, we would like to not only incorporate how close the predicted seams are to the true seams, but measure the efficacy of our model on correctly identifying untampered regions. One metric that satisfies this specification is the Matthews Correlation Coefficient (MCC), a balanced measure ranging between ±1 that can be used regardless of the degree of class imbalance in a dataset due to its invariance to the choice of which class is positive or negative [10]. It should be noted that in all cases where cropping resulted in all ground truth negatives for a particular sample, all metrics aside from pixelwise accuracy are set to 0.

Customizing Confusion Matrix Metrics
While these confusion matrix metrics are widely used to evaluate binary classification performance, we make a slight adjustment to the way we calculate confusion matrices (and their derived metrics) for our specific use-case. Since seams are only one-pixel wide and our confusion matrices are calculated on a pixel-wise basis, they are extremely sensitive to the spatial distribution of the prediction mask. For example, take the seam insertion mask from Figure 4b. If we shift all the seams one pixel to the right and compare the shifted mask to the original, our confusion matrix metrics break down and report poor results. Specifically, the recall between the original and shifted masks becomes 0.194 when the shifted version is in fact localizing the seams quite well.
To properly represent the performance of our model's predictions, we modify the confusion matrix calculation such that we assign a true positive if our prediction is within a buffer of p pixels of a ground truth positive. This relaxation is similar to a metric used in evaluating road detectors, especially in the context of aerial imagery [36,52]. It's worth noting that if we assign a true positive in this way, we do not double count the ground truth positive used to evaluate predicted negative pixels. If we predict negatively and the ground truth at that location is positive but has been used, we assign it as a true negative. We employ the strategy described here with p = 1, and for the particular example in this section the recall between the original and shifted masks becomes 1.0, due the 1 pixel buffer described above.
To make this clear, we refer to these customized metrics as MCC-1, F1 Score-1, Precision-1, and Recall-1 throughout the remainder of this paper.

Seam Localization Score (SLS)
To fully capture the localization performance of our models, we develop a metric based on the seams that are inserted or removed. By keeping track of the seams associated with manipulated pixels, we can evaluate how well our model localizes each specific seam. For a particular vertical seam s taken from an image of height h and width w, the corresponding ground truth seam mask will associate h pixels with seam s. Then, we sum over each seam pixel from 1 : h, the absolute distance to the nearest predicted positive in each row. If we do not predict a positive pixel in a particular row, the absolute distance is set to w. Finally, we normalize by the number of rows h. We refer to this metric the seam localization score (SLS). For an image with N seams we compute an image level SLS by summing the score for each seam, and dividing by N . The SLS for any particular seam ranges from 0, perfect overlap, to w. When the SLS for a particular seam is less than one, we can interpret that on average, the seam was less than one pixel away from its ground truth location.

Experimental Results
Here, we describe the experiments that are carried out towards localized seam carving detection. First, we cover our multistage approach towards localized detection of seam carving, using pixelwise classifiers, and then present the performance of localization models using the metrics detailed in Section 6. Then, we demonstrate the generalizability of our model across different training distributions and seam carving algorithms.

Pixelwise Classification
To achieve more fine-grained localization of seams and and incorporate the spatial relationships of input images, we   train a step-down, step-up fully convolutional neural network based on U-Net as described in Section 4.1 to detect removed and inserted seams at pixel-level resolution (Figure 1). We report stage 1 evaluation metrics from Section 6 using a variety of encoder architectures in Table 1 on our xView test dataset, as generated in Section 5.2. The best performing model used an EfficientNetB7 encoder architecture, achieving a MCC-1 score of 0.903. We note that the performance increase gained by using EfficientNet cannot be solely due to the increase in model capacity. The second best performing network was MobileNetV2, which has the smallest number of trainable parameters among all networks tested. Our seam insertion detector of the same Effi-cientNetB7 architecture, achieves a MCC-1 score of 0.956 on the xView test set. We also observe that although trained on xView images, the seam insertion detector generalizes well to xBD and Orbview-3 datasets with an MCC-1 of 0.938 and 0.942 respectively.

Generalizability of Pixelwise Classifiers
Here, we summarize several experiments demonstrating the generalizability of pixelwise classifiers across datasets and various seam carving techniques. Pixelwise classifiers trained on our xView dataset work well on xBD and Orbview-3 datasets and vice versa. Moreover, even though pixelwise classifiers are trained with datasets forged using forward energy, they are generalizable to different seam carving techniques. Table 2 shows that pixelwise classifiers for seam insertion detection are generalizable across different datasets. A seam insertion detector trained on xView has only minor drops in performance on xBD and Orbview-3 test sets.

Generalizability Across Datasets
We also observe that while seam removal detectors are not as generalizable as seam insertion detectors, they still perform well on different datasets as shown in Table 3. Although the decrease in MCC-1 scores for seam removal detectors are larger when stress testing across datasets, we note that the scores themselves are still quite good, and are indicative of adequate performance for stage 2 classification.

Generalizability Across Seam Carving Methods
We also tested the generalizability of our models across various seam carving methods. So far, all of our results have been reported on datasets that have been generated using forward energy seam carving. We report test evaluation metrics on our xView dataset in Table 4, where we have generated test sets using backward energy, frequency tuned saliency maps, and seam merging variations of seam carving as developed in Section 3. In this table, the model has been trained on only forward energy seam carving data, but we see good performance across different seam carving techniques, with the lowest performance on the dataset generated using saliency map based seam carving. This is most likely due to how different the seams from saliency map seam carving look compared to the other techniques.

Stage 2 Classification
In stage 2, we generate a dataset of stage 1 predictions on manipulated and pristine images to train a final binary classifier to check if the input image has been seam carved. We use a stage 1 model trained on the xBD dataset as described in 5.2. This stage 1 model is used to obtain predictions on a combination of xBD datasets. The first xBD dataset that we predict on is described in 5.2. We remove the 10% most optimal seams, which are often distributed throughout the image, and reinsert seams to restore the original image dimensions. The second xBD dataset we predict on incorporates the original dataset's ground truth building masks and uses seam carving to remove a building and reinsert seams to restore the original image dimensions. We call this dataset "xBD OR". Thus, our stage 2 model is trained on a combination of stage 1 predictions on best seam removed and object removed images. Finally, we generate best seam removed test datasets of xView, xBD and Orbview-3 using individually trained pixelwise classifiers. For example, the Orbview-3 test dataset for stage 2 is generated using an Orbview-3 trained pixelwise classifier. Although the xView and Orbview-3 test datasets are generated using different pixelwise classifiers than the training dataset, our stage 2 model performs very well on both, achieving almost 99% accuracy. This shows that the stage 2 final classification model is robust to the pixelwise classification model used to generate the input prediction masks.

Experimental Setup
In this subsection, we cover the experimental setup for training our models. Seam removal and seam insertion pixelwise classifiers for the xView dataset are trained using the Adam optimizer with a learning rate of 0.0003, momentum coefficients β 1 = 0.9 and β 2 = 0.999 and a numerical stability constant of = 10 −6 . The loss function used is a pixelwise mean squared error. We find that using binary cross entropy as the loss function produces comparable results, but the mean squared error leads to smoother convergence. We use a batch size of 8 where each batch is sampled randomly from the training set without replacement for each epoch. We train for 30 epochs and select the model with the highest validation set accuracy, reducing the learning rate by a factor of 0.2 if there is no validation accuracy improvement for 5 epochs. For training on xBD and Orbview-3 datasets, we keep all hyperparameters the same except for a learning rate adjustment to 0.0001.
Stage 2 binary classifiers are trained using identical hyperparameters as above, except that we use a cross entropy loss, a learning rate of 0.0001, and train for 35 epochs. These hyperparameters are the same for the stage 2 model used in Table 6 and Table 5.

Seam Carving Retargetting Ratios
In this section, we present results of two experiments where we vary the number of seams removed and inserted from the original xView dataset.
In Table 6, we provide MCC-1 and SLS scores for both seam removal and insertion detectors as well as stage 2 binary classification accuracies on test datasets of varying seam carving retargetting ratios using a model that is trained on 10% seam carved data. Test datasets are generated by seam carving 512x512 patches from xview test split by different percentages. We can see from this table that our seam carving detector is remarkably generalizable to other seam carving retargetting ratios despite being trained on only 10% seam carved data. In terms of an overall image classification, our framework achieves over 99% accuracy at detecting seam carved images on all retargetting ratios except 2%, where it achieves the lowest accuracy at 98.56%. In general, the SLS score of the seam removal detector decreases the further away we move from 10% seam carving, while the MCC-1 only decreases as we increase the retargetting ratio. This showcases the usefulness of the SLS metric as a seam carving localization metric. In the case of 2% seam carved data our seam removal detector achieves its best MCC-1 score of 0.918 due to the large amount of non manipulated pixels in the ground truth and predicted masks. However, the SLS score of 2.05 shows that our model is not  as good at predicting seam locations as the MCC-1 score might lead us to expect. In the seam insertion case, we see good performance across the range of retargetting ratios tested. The lowest MCC-1 score for seam insertion is 0.920 on 50% seam carved data, which is higher than the best MCC-1 score for seam removal at 0.918 on 2% seam carved data. The seam insertion SLS scores show that our model is able to localize inserted seams within 1 pixel precision across all retargetting ratios. Although we show results on a model that was trained using a 10% seam carving ratio in this paper, the results were similar for models trained on other percentages of seam carving too. In Table 7, we show results of training on datasets of different seam carving retargetting ratios. We include the results on the 10% seam carved test set, and provide evaluation metrics on 20%, 30%, 40%, and 50% seam carved test datasets. This table shows that our method is applicable to other seam carving retargetting ratios and achieves similar results. The SLS score is around 1 for all values tested, and the F1 Score-1 and MCC-1 scores are high. Notably, the 20% seam carving dataset has the highest performance while the 50% seam carved dataset has the worst.

Seam Carving detection on post processed images
We explored the performance of our models on the dataset of images that are JPEG compressed after seam carving. Retraining the models on post processed datasets resulted in the similar image level detection, while the models that are not trained on post processed images has shown drop in performance, as shown in Table 8. Same trend has been observed when we replaced JPEG compression with rotation as post processing step, as shown in Table 9.

Visual Examples
Besides content aware image resizing, seam carving can be used to remove or displace objects in a given image. In Figure 5 and Figure 6, we present the sample manipulations that one can do using seam carving and visualize the localization results of stage 1 pixelwise classifiers on manipulated images. Figure 5 has five examples illustrating the application of seam carving to remove objects in satellite imagery while making sure that the seam carved satellite image looks authentic to the human eye and retains its original dimensions. In Figure 5a, truck under the removal mask (red) is taken out and seams are inserted to restore the original image, all while leaving pixels in the protective mask (green) are left undisturbed. This is achieved by setting the energy map values at the removal mask locations to a low energy value, forcing seam carving algorithm to pass through. When inserting seams to restore the original dimensions, pixels at the locations of the protective mask are set to a high energy value, ensuring that the seam carving algorithm ignores them. Figure 5b has an example in which an excavator under the red mask is removed from the image. Similar examples are shown in Figures 5c, 5d, and 5e, where pixels under the red colored mask are removed while pixels under the green colored mask (if present) are left undisturbed. Figure 6 has five examples illustrating seam carving based manipulations to displace objects in a given image while retaining the visual authenticity and size of the original image. In Figure 6a, the white colored building at the center of the image is displaced by 50 pixels to left. This is achieved by forcing the seam carving algorithm to remove 50 seams from left side of the object and insert back same number of seams to right using removal and protective masks. In Figure 6b, the road is displaced by 50 pixels to left, whereas in Figure 6c, road is displaced by 50 pixels to right. An excavator is moved to right by 50 pixels in Figure  6d, and aeroplanes are moved to left by 40 pixels in Figure  6e. Figure 7 visualizes the distribution of removed seams in a given image that is seam carved using different seam carving algorithms described in Section 3. It can be observed that even though the seams are removed using different seam carving algorithms, our seam removal detector in stage 1 is able to predict the locations of the removed seams.

Comparison against Existing Approaches
We compare our method with two seam carving detection methods, that uses LBP [57] and ILFNet [39]. Although these methods are not specifically designed for localized detection in satellite imagery, we compare our image level stage 2 results with them as they are perform-     Table 9: Stage-2 Test accuracy(%): Improvement in detection accuracy when the model is trained on post processed images.
ing image level classification to detect seam carved images.
Comparative results are shown in Table 10, where all of the forensic techniques have been trained on the xView training set generated as in Section 5.2. We observe that although ILFNet achieves a comparable accuracy with the proposed method on the xView test set, it's generalizability performance drops around 5% and 2% when testing on xBD and Orbview-3 while our method drops less than 1%. Similarly, a method that utilizes local binary pattern based feature extraction combined with an SVM classifier performs reasonably well on the xView test set, but fails to generalize to other datasets.

Conclusion
In this paper, we proposed a method to detect and localize seam carving based manipulations in satellite images. We use a two stage approach that first localizes removed/inserted seams via pixelwise classification and then performs a final classification if an image has been seam carved. We enable localization of seams as well as a generalizable framework across different datasets and seam carving techniques. Finally, we detailed the curation of three unique, large seam carving satellite image datasets that will be released to the public. Although the proposed method is not specifically restricted to satellite imagery, we present our findings on satellite images as a case study and leave further evaluation on more conventional images with varying compression schemes and preprocessing to be explored in future work.