{'fraction_non_alphanumeric': 0.03565965258184043, 'fraction_numerical': 0.015110310364755073, 'mean_word_length': 4.999898021619416, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning JPEG Compression Artifacts for Image Manipulation Detection and Localization

Detecting and localizing image manipulation are necessary to counter malicious use of image editing techniques. Accordingly, it is essential to distinguish between authentic and tampered regions by analyzing intrinsic statistics in an image. We focus on JPEG compression artifacts left during image acquisition and editing. We propose a convolutional neural network (CNN) that uses discrete cosine transform (DCT) coefficients, where compression artifacts remain, to localize image manipulation. Standard CNNs cannot learn the distribution of DCT coefficients because the convolution throws away the spatial coordinates, which are essential for DCT coefficients. We illustrate how to design and train a neural network that can learn the distribution of DCT coefficients. Furthermore, we introduce Compression Artifact Tracing Network (CAT-Net) that jointly uses image acquisition artifacts and compression artifacts. It significantly outperforms traditional and deep neural network-based methods in detecting and localizing tampered regions.


Introduction
With the advance of mobile devices and image editing software, image editing has become easy and popular. Together with social networking services, edited images can be spread quickly. These changes enable people to create more beautiful selfies, reduce camera shake, place an unaccompanied friend in a group photo, remove undesired objects, and share these edited images with others. However, these advances cause social problems when edited images are used as false evidence or fake news. An object-removed surveillance camera image might falsely confirm that a criminal was not at a crime scene or vice versa. A fabricated photo suggesting a celebrity scandal might damage the celebrity's reputation. Therefore, to prevent malicious image manipulation, it is critical to detect them and localize forged regions.
Among many image manipulation types, copy-andpasting some regions onto an image either from the same image (copy-move) or another image (splicing) is one of the most popular and straightforward image editing techniques. Because these manipulations are applied to local regions, analyzing them is more challenging than kernel-based or pixel-level manipulation (e.g.,  Fig. 1 Challenge of localizing manipulated regions from a JPEG image. Although many neural networks can trace noise precisely to detect manipulation, they are not ideal for capturing compression artifacts. The proposed approach considers RGB and DCT domains jointly to track visual clues and compression traces accurately. Given a possibly manipulated image ( Fig. 1(b)), this study predicts the manipulated region ( Fig. 1(g)). This study significantly outperforms state-of-the-art methods (Figs. 1(d), 1(e), and 1(f)) in detecting and localizing forged regions.
hue modification, blurring, contrast enhancement, or brightness adjustment) applied to the global region. Furthermore, splicing and copy-move may not leave visual clues visible to the human eyes that consider the harmony between the pristine image and the objects to be pasted (Figs. 1(a) and 1(b)). Consequently, in the last decade, many forensic approaches have been proposed to detect and localize image manipulation (Verdoliva, 2020;Korus, 2017).
A fundamental assumption underlying manipulated region detection and localization is that the image acquisition artifacts (Lukas et al., 2006) or JPEG compression artifacts (Wang and Zhang, 2016) of manipulated regions have different statistical properties from those of the pristine regions. An image acquired from a digital camera undergoes inherent internal processes. Thus, intrinsic statistical characteristics are left in digital images for each device and shooting setting. Moreover, most camera-equipped devices apply lossy compression (conventionally, JPEG) to the digital image for storage efficiency, leaving compression artifacts in the image. Characteristics of image acquisition and compression artifacts are consistently maintained within the media data if no manipulation occurs. Furthermore, the statistical characteristics of these artifacts can be changed when manipulation is applied. Image forensics aims to classify manipulated regions with different statistical fingerprints from pristine regions, so it is essential to understand detailed processes of image acquisition and JPEG compression.
First, image acquisition artifacts refer to traces from the processes applied when creating a digital image from shooting a scene. The types of representative image acquisition artifacts are as follows: lens aberration (Yerushalmy and Hel-Or, 2011), sensor pattern noise (Lukas et al., 2006), interpolation traces from the color filter array (CFA) (Bammey et al., 2020), and postprocessing artifacts caused by color correction, white balance adjustment, and gamma correction (Swaminathan et al., 2008). These artifacts are device-and setting-dependent fingerprints that accompany the im- Statistical differences between tampered and authentic regions. DCT histograms are obtained from Y-channel DCT coefficients at the frequency (1,1) for tampered and authentic regions separately. The x-axis is the DCT coefficient value and the y-axis is the relative frequency. The manipulated region follows a Laplacian distribution, whereas this distribution is interrupted for the authentic region. More in Sect. 2.
age acquisition process and are difficult to distinguish with the human eye. In the field of multimedia forensics, rule-based, handcrafted feature-based, and datadriven approaches to capture changes in the statistical properties of each acquisition artifact have been studied (Verdoliva, 2020). These approaches are designed to detect and expose manipulated regions by revealing inconsistencies in fine acquisition artifacts.
Second, JPEG is the most actively used compression standard to reduce storage space, leaving subtle but distinct artifacts due to quantization-based compression applied to the discrete cosine transform (DCT) domain (Barni et al., 2017). In image forensic research, double JPEG detection, i.e., determining if a JPEG image has been compressed once or twice, is being actively stud-ied (Wang and Zhang, 2016;Park et al., 2018;Verma et al., 2020). This task helps localize the manipulation regions. A region pasted onto another image likely has a statistically different distribution of Y-channel DCT coefficients compared to the authentic region (Figs. 2(b) and 2(c)). The authentic region is doubly compressed, first in a camera and again as part of the forgery, leaving periodic patterns in the histogram. The manipulated region follows a singly compressed distribution, based on the secondary quantization table (Sect. 2.2 and Popescu and Farid (2004)). Therefore, the ability to explore these compression artifacts helps in inferring and localizing the manipulated region. However, it is difficult to know in advance which region has been tampered with, i.e., what we observe is the sum of two histograms ( Fig. 2(d)).
Based on these observations of the two types of artifacts left in the manipulated image, we use both RGB and DCT domain information to detect and localize image manipulation. We propose an end-to-end trainable neural network-based image manipulation detector named Compression Artifact Tracing Network (CAT-Net). It traces image acquisition artifacts and JPEG compression artifacts accurately. The RGB domain enables the network to explore and learn fine-grained visual artifacts such as sensor pattern noise, block artifacts, and other acquisition artifacts. The DCT domain is used to explore compression artifacts.
However, supplying DCT coefficients directly to a convolutional neural network (CNN) is inadequate because the convolution throws away the spatial coordinates, which are crucial for DCT coefficients. Recently, Yousfi and Fridrich (2020) try to solve this problem using DCT volume representation in a steganalysis classification task. We adopt this representation in our network to learn the distribution of DCT coefficients. We demonstrate that this representation is also adequate for forgery localization tasks. Furthermore, the designed network includes only specially chosen network components to learn the image compression artifacts. Moreover, we propose a new pretraining method that uses double JPEG detection. This paper extends our previous study (Kwon et al., 2021), which introduced a JPEG compression artifacttracing method for image splicing detection. Whereas previous research only targeted splicing forgery, this paper also deals with copy-move forgery. New custom datasets are added to improve the performance further. More extensive experiments are performed with ten comparative methods, whereas only two methods were used in previous research. The results are reported with various metrics and newly added heatmaps. Fi-nally, we released our code and trained weights publicly at https://github.com/mjkwon2021/CAT-Net.
Our main contributions are summarized as follows: • We propose CAT-Net that learns compression artifacts based on DCT volume representation. This approach outperforms previous state-of-the-art networks using histogram representation in detecting double JPEG compression. Furthermore, we successfully transferred these weights to image manipulation detection and localization.
• CAT-Net learns the distribution of DCT coefficients without losing spatial information to finely localize tampered regions. In contrast, previous histogram approaches lose spatial information and function only for classification. CAT-Net is the first neural network that accepts DCT coefficients directly into a segmentation network.
• For the first time, CAT-Net localizes manipulated regions considering RGB and DCT domains jointly. The network captures image acquisition artifacts in the RGB domain and compression artifacts in the DCT domain. Extensive experiments with diverse benchmark datasets demonstrate that CAT-Net significantly outperforms state-of-the-art manipulation detectors.
The remainder of this paper is organized as follows. Section 2 explains forensic clues and reviews relevant previous studies. Section 3 proposes our forensic approach. Section 4 explains a double JPEG pretraining scheme and evaluates CAT-Net in terms of learning compression artifacts. Section 5 describes the main experiments, image manipulation detection and localization, and demonstrates the performance of CAT-Net. Section 6 concludes the paper.

Related Work
In this section, we review forensic clues including image acquisition artifacts and JPEG compression artifacts. We then introduce previous forensic approaches related to this study. Two types of inevitable artifacts remain in the digital image without manipulation: image acquisition artifacts and compression artifacts. These artifacts are essential forensic clues because their intrinsic properties differ before and after the manipulation process.

Image Acquisition Artifacts
Image acquisition artifacts denote fine artifacts generated when camera-equipped devices obtain a digital im- age. We can detect manipulated regions by distinguishing between authentic and tampered regions. Accordingly, we should partition the image into areas from the same image. Therefore, understanding camera-specific and capture-setting artifacts caused by the image acquisition process is advantageous. Figure 3 illustrates the detailed processes of image acquisition from a digital camera. The terms in the arrow brackets refer to acquisition artifacts generated from a specific acquisition process. Before the light from the photographed scene reaches the sensor on the digital camera, it passes through the lenses, an anti-aliasing filter (i.e., optical low-pass filter), and a CFA. Because of the minor defects in the manufacturing process of a lens, the lens produces several types of image aberrations: spherical aberration, field curvature, lens radial distortion, and chromatic distortion. For source camera identification (Choi et al., 2006) and forgery detection (Yerushalmy and Hel-Or, 2011), these lens aberrations can be used as forensic fingerprints. The light passed through the lens passes through an anti-aliasing filter, which reduces aliasing and moire patterns. Then, the light passes through a CFA before reaching the sensor. The CFA is a mosaic of color filters that block out a particular portion of the spectrum, inducing each pixel to detect only one specific color among red, green, and blue (Piva, 2013).
The sensor, composed of minimal addressable elements that collect photons and convert them into electrical signals, is a critical component of a digital camera. With an analog-to-digital converter on the imaging sensor, the voltages can be sampled to digital signals (Lukas et al., 2006). The two sensor types include the charge-coupled device (CCD) and the complementary metal-oxide-semiconductor (CMOS), which both leave fine traces of sensor pattern noise (Piva, 2013). The sensor pattern noise is caused primarily by imperfections during imaging sensor manufacturing. The main types of sensor pattern noise are fixed pattern and photo-response nonuniformity (PRNU). Because the pattern noise is an inherent property dependent on a specific camera model, it is actively used for image forensics as a distinct feature (Lukas et al., 2006;Chierchia et al., 2014;Korus and Huang, 2016).
For a CFA-based sensor (e.g., CCD or CMOS), the digitized sensor output is interpolated, exploiting the color interpolation process (i.e., demosaicing) to obtain the missing pixel values for the three-color layers (Piva, 2013). In this process, CFA interpolation artifacts are applied to the image, and these traces can be used to detect image forgery (Bammey et al., 2020;Choi et al., 2013). The output signal is then further processed based on post-processing, such as color correction, white balance adjustment, and gamma correction. These postprocessing processes are fine corrections for perceptual quality, during which acquisition artifacts are added to the digital image.
Finally, the digital image is written to the camera memory device in a user-selected image format. JPEG is a representative lossy compression technique for digital images that mitigates or removes the highfrequency components. This paper uses compression to denote lossy compression (instead of loss-less compression, which is irrelevant for compression artifacts). The details of JPEG compression are introduced in the following subsection.

JPEG Compression Artifacts
In this subsection, we review the JPEG compression process and observe the double quantization artifacts left in the DCT domain. An input image is divided into non-overlapping 8 × 8 blocks, each block individually transformed using the DCT. In this paper, we consider only Y-channel DCT coefficients because chroma channels (i.e. C b and C r ) are less useful for forensics. The DCT coefficients are then quantized using a single 8 × 8 quantization matrix. Quantization is an element-wise operation described as: where q 1 is the quantization step, u is a value in the DCT domain, and [·] is a rounding operator. The quantized coefficients and the quantization table -not the spatial domain pixels -are saved in a JPEG file. The coefficients are dequantized when the image file is opened (i.e., during the JPEG decoding process): where v is the quantized DCT coefficient. Q q1 is not mathematically invertible, so the loss of information occurs here. Double quantization can then be described as: where q 1 is the primary quantization step and q 2 the secondary quantization step. We then investigate the relationship between an initial DCT coefficients histogram and a double compressed histogram. Assume a DCT coefficient in the u 1 -th bin in the former is relocated in a u 2 -th bin in the latter, i.e., Q q1,q2 (u 1 ) = u 2 . Then, the number of original histogram bins n(u 2 ) contributing to bin u 2 in the double quantized histogram can be expressed as follows (Lin et al., 2009): where · is the flooring operator and · is the ceiling operator. Based on Eq. (4), n(u) is periodic with period q 1 /gcd(q 1 , q 2 ) where gcd is the greatest common divisor. Therefore, the double compressed region has periodic patterns in the histograms of quantized DCT coefficients. For example, Fig. 2 illustrates a double compressed image with quality factor 70 followed by 90 and the effects of double quantization at frequency (1, 1). Then, q 1 = T 70 (1, 1) = 7 and q 2 = T 90 (1, 1) = 2 where T x (i, j) is the value of the (i, j) component of the quantization table with the quality factor x, where i, j = 0, ..., 7. Thus, based on the Eq. (4), n(7k) = 7, n(7k+1) = 0, n(7k+2) = 0, n(7k+3) = 7, n(7k+4) = 7, n(7k + 5) = 0, and n(7k + 6) = 0 where k is an integer. This values coincide with the observation that specific bins are empty in Fig. 2(b). This periodic pattern is an example of many double compression effects (More in Sect. 2.4).
The above reasoning assumed that quantization uses rounding to the nearest integer with tie-breaking toward positive infinity. However, different operations such as rounding toward zero can be used depending on camera manufacturers or image editing software (Agarwal and Farid, 2018;Butora and Fridrich, 2020). Furthermore, information loss occurs during decoding by rounding after inverse DCT is applied and truncating to the proper image pixel range [0,255]. The precision of the DCT transform also impacts the distribution of coefficients (Lukáš and Fridrich, 2003). Accordingly, quantization artifacts in real-world implementations are diverse and should be handled with care. The manipulated and authentic portions exhibit different statistical distributions in the DCT histogram. The authentic regions are compressed twice. The tampered region is treated as single compression because the 8 × 8 grid used in the second compression is likely misaligned with the primary compression grid (with probability 63 64 ). Even when the two grids align, blocks containing the boundary of the pasted object have both authentic and tampered pixels, so these blocks do not follow the double compression rules (Wang and Zhang, 2016).

Image Forensics Using Image Acquisition Artifacts
Image forensics aims to verify the authenticity of media content by detecting and exploring manipulation artifacts. It is challenging to localize and detect manipulation applied to local regions (e.g., splicing or copy-move) and related studies are steadily progressing. Table 1 summarizes historic image forensic approaches. The forensic clues used by each approach are categorized primarily into image acquisition and JPEG compression artifacts. This subsection reviews previous studies that explore traces of image acquisition. These studies either use acquisition artifacts directly as forensic features or explore low-level features to detect statistical changes on acquisition artifacts caused by image manipulation. Mahdian and Saic (2009) propose a forgery localization method using local noise standard deviation estimated based on tiling the high-pass wavelet coefficient. Amerini et al. (2011) use a scale-invariant feature trans-form (SIFT) to detect copy-move forgery; the pairs of SIFT descriptors between the pristine and manipulated regions are selected using a clustering algorithm. Ferrara et al. (2012) perform block-based forgery detection exploring demosaicing artifacts, a subtle deformation applied to the original CFA pattern during the forgery process. Lyu et al. (2014) formulate blind noise estimation as an optimization problem and detect local noise inconsistency to localize region forgery.
Inspired by computer vision tasks that have achieved significant progress after adopting CNNs, CNNs are actively exploited in image forensics to localize forged areas and detect fine-grained manipulation clues (Verdoliva, 2020;Nam et al., 2020;Yu et al., 2020). Bayar and Stamm (2018) propose a constrained layer-based network that jointly suppresses the content of a given image and adaptively learns features from noise-like signals generated by image manipulation. Zhou et al. (2018) place SRM kernel (Fridrich and Kodovsky, 2012) as a pre-processing layer and uses the Faster R-CNN (Ren et al., 2015) architecture to detect the manipulated area in units of objectlevel. Boroumand et al. (2018) place unpooled layers at the early part of the network, which extracts rich features of low-level signals and illustrates excellent performance for steganalysis. Huh et al. (2018) propose a self-supervised approach to train a model and explores the inconsistency of EXIF metadata. Their research exhibits outstanding performance in localizing manipulation but requires significant computation to compute the consistency for every patch pair. Bi et al. (2019) frame manipulation localization as a segmentation problem. They design a neural network based on U-Net (Ronneberger et al., 2015) and analyze the conventional semantic property by providing an RGB pixel image as input to the network. Wu et al. (2019) design a ManTra-Net that extracts features using the SRM kernel and constrained layer for preprocessing and performs pixel-wise anomaly detection. Their research classifies various types of manipulation successfully. However, the performance of the forgery localization is not robust to JPEG compression because it uses compression as one of the manipulation types. Kniaz et al. (2019) propose a generative adversarial network-based framework for training a discriminative segmentation model to localize manipulated regions. Cozzolino and Verdoliva (2019) propose an approach to extract intrinsic noise of a camera model (i.e., Noiseprint), where the content of a given image is suppressed and acquisition artifacts are enhanced. Their study explores anomalies with respect to the dominant pristine model for localizing manipulated parts. Bammey et al. (2020) exploit an unsupervised CNN that learns to explore the underlying pattern of CFA interpolation artifacts and detects suspicious regions by identifying local mosaic inconsistencies. Marra et al. (2020) present a framework comprising of three phases -patch-wise feature extraction, image-wise feature aggregation, and global decision -that enables the use of rich features gathered at full resolution from the whole image. Hu et al. (2020) present a local self-attention block-based CNN that models and establishes the spatial relationship between patches at multiple scales to capture forensic fingerprints in forgery localization. Liu and Pun (2020) propose a fusion network that concentrates on learning low-level features and explores forensic hypotheses such as noise and JPEG discrepancies.

Image Forensics Using JPEG Compression Artifacts
In this subsection, we review previous forensic methods exploiting forensic clues caused by JPEG compression. Lam and Goodman (2000) mathematically illustrate that histograms of JPEG DCT coefficients follow the Laplacian distribution. Image editing conventionally involves additional compression and breaks the distribution, leaving compression traces in the image. Therefore, JPEG compression artifacts have been used as important fingerprints in image forensics. Lukáš and Fridrich (2003) observe a fundamental characteristic left in the DCT domain when an image is forged. A pasted portion of a forged image likely exhibits traces of single compression, while the rest of the authentic region exhibits signs of double compression. The researchers present properties of missing values and double peaks in a histogram of DCT coefficients to detect double compression and estimate the primary quantization table. Fu et al. (2007) observe that the distribution of the first digits of the DCT coefficients follows Benford's law, which is violated if the image is double compressed. This law is used for Q-factor estimation and double JPEG detection. Ye et al. (2007) use inconsistency of JPEG blocking artifacts to detect forgeries. A blocking artifact measure is calculated based on the estimated quantization table using the power spectrum of the DCT coefficient histogram. Lin et al. (2009) use JPEG double quantization effects such as periodic peaks and valleys in the DCT histogram to detect image forgeries automatically at the scale of 8 × 8 blocks. A block posterior probability map computed from histograms is thresholded to differentiate the tampered and authentic regions. Bianchi and Piva (2012) design a unified statistical model characterizing the DCT coefficients for both aligned and nonaligned double JPEG compression. The model is used to compute a likelihood map indicating the probability of each DCT block being doubly compressed. Iakovidou et al. (2018) use JPEG grid alignment abnormalities for forgery detection. The method evaluates multiple grid positions using a fitting function, where lower contribution areas are identified as grid discontinuities. Nikoukhah et al. (2019) perform global grid detection via determining the likeliest JPEG blocks containing the largest number of zero coefficients to detect grid non-alignment caused by splicing.
In the deep learning era, there have been subsequent studies on double JPEG detection using DCT histograms to generate neural network features. Wang and Zhang (2016) are the first to use histogram features as input to a CNN for double JPEG detection. They also achieve forgery localization by integrating imagelevel classification results using an overlapping stride of 8 pixels. Barni et al. (2017) integrate histogram computation as part of a CNN, allowing GPU to construct histogram features in parallel. They also improve the structure of the CNN using two-dimensional (2D) convolutions instead of one-dimensional (1D) convolutions. Park et al. (2018) improve classification performance by appending a reshaped quantization table in fully connected layers.
CNN-based approaches with a DCT histogram are confined to image-level classification, primarily because using a DCT histogram requires a fixed size input and removes spatial information for localization. Previously, using a DCT histogram was mandatory because CNNs could not learn from naive DCT coefficients due to their predominantly decorrelated and locally heterogeneous nature. This study is the first to use a segmentation model based on DCT coefficients, which is possible due to the DCT volume replacing the DCT histogram and carefully designed network components. Moreover, we pretrain our network in a double JPEG detection task to produce rich initialization for learning compression artifacts left in the DCT coefficients.

Proposed Method
We describe how to extract features over DCT coefficients to learn their distributions using standard CNN components. Accordingly, we propose a JPEG artifact learning module (Fig. 4) that can be placed at the starting point of a CNN. Furthermore, we propose CAT-Net, a complete end-to-end image manipulation detection network. CAT-Net comprises an RGB stream, DCT stream, and fusion stage (Figs. 5 and 6). It accepts RGB pixels, DCT coefficients, and a quantization table as network inputs and outputs a probability map of each pixel being tampered with. We first describe four key points that enable a CNN to learn the distribution of DCT coefficients: DCT volume representation, frequency-wise operations, grid-aligned cropping, and transfer learning from double JPEG detection. We then describe the detailed network architecture. Finally, we describe how CAT-Net processes non-JPEG images.

DCT Volume Representation
As explained in Sect. 1, CNNs cannot automatically learn the compression artifacts from raw DCT coefficients because the convolution assumes a translationinvariant property and handles every coefficient the same. However, the spatial coordinates are critical for DCT coefficients. Thus, we convert the input array of DCT coefficients, M, to a binary volume (Yousfi and Fridrich, 2020) using a transformation f : where clip(·) clips the array element-wise into the interval [−T, T ] and abs(·) takes element-wise absolute values. The DCT coefficients are recorded in channel indices with 0 or 1. The clip(·) function is due to memory constraints. A larger T enables capturing a broader range of histogram bins (Fig. 2) but requires more GPU memory. We chose T to be 20, experimentally. The abs(·) function is due to the symmetry of the DCT histogram as depicted in Eq. 4. With the identity −x = − x , we obtain n(−u) = n(u). Thus, information loss caused by taking absolute values is negligible, but the feature map size becomes almost half.
For manipulation localization in JPEG images, the DCT volume representation is more accurate than the DCT histogram, which detects double JPEG compression (Wang and Zhang, 2016;Barni et al., 2017;Park et al., 2018). Whereas the DCT histogram merges information patch-wise and loses its visual representation, the DCT volume maintains image resolution suitable for prediction at the pixel level. Nevertheless, the ability to extract statistical information is an improvement over DCT histograms (Sect. 4).
The DCT histogram is the result of applying global average pooling to the DCT volume. Thus, the DCT volume is a feature before losing location information. Furthermore, the convolution on this representation produces much richer statistical features such as co-occurrence. For example, consider a 3 × 3 kernel K ∈ R (T +1)×3×3 , where all the elements in K are zero except K[m, 1, 1] = 1, K[n, 1, 2] = 1, and  Fig. 4 Proposed JPEG artifact learning module architecture. The DCT volume conversion f (·) and the frequency component separation g(·) are depicted in Eqs. 5 and 6, respectively. m, n ∈ {0, 1, ..., T }. If a convolution using the kernel K is applied to the DCT volume and global average pooling is followed, the horizontal co-occurrence for coefficient pairs (m, n) is computed. In contrast to JPEG image steganalysis, which evaluates the probability of whole image manipulation, our goal is localizing manipulation. Therefore, we aim to extract features among different DCT blocks and use convolutions with a dilation of 8, which enables frequency-wise operations.

Frequency-wise Operations
In contrast to RGB pixels, DCT coefficients represent different frequencies depending on where they are located. The DCT coefficient at (x, y) represents frequency (x mod 8, y mod 8) of the x 8 , y 8 image subblock. Conventional convolutions (with stride one) mix these frequency components. All operations should be performed on a frequency-wise basis to avoid this. Namely, these include an 8 × 8 convolution with a dilation of 8, a 1 × 1 convolution, quantization table multiplication, and frequency component separation (Fig. 4). The 8 × 8 convolution with a dilation of 8 operates on the same frequencies because DCT coefficients consist of 8×8 blocks. The 1×1 convolution is also valid because it does not mix frequency components. A quantization table is used to help the network learn compression history. Park et al. (2018) are the first to use a quantization table in fully connected layers. However, because our network is fully convolutional, we cannot follow their approach. We solve this problem by mimicking the JPEG decoding process. The quantization table is multiplied element-wise with a feature map. This approach uses the role of quantization tables for dequantizing quantized coefficients (Eq. 2). Quantized and dequantized feature maps are both used in our module.

Frequency component separation
is an index changing mapping that can be implemented using only reshaping and permutation: where c, i, j starts from 0. After frequency component separation, the feature maps can be used without special care, i.e., conventional 3 × 3 convolutions may follow.

Grid-aligned Cropping
Deep neural networks take input images of the same fixed size when training to construct a batch. Conventional computer vision networks use resizing or random cropping to satisfy this constraint. With DCT coefficients, these two methods cannot be used because they destroy position information. We propose a new cropping method that can be used for DCT coefficients. Images should be cropped in a grid-aligned scheme to enable these components to function correctly. Given an input image M and crop size h × w, the conventional cropped image h(M) can be represented as: where NumPy index slicing is used. Grid-aligned cropping requires h, w, i, j to be the multiples of eight. This simple remedy enables the neural network components described in Sect. 3.2 to function. With grid-aligned cropping, each feature map channel after frequency component separation represents one frequency. For example, the first channel corresponds to frequency (0, 0), the second channel corresponds to frequency (0, 1), and so on. If conventional random cropping were used instead, frequency components would be distributed over   all channels and the subsequent convolution could not distinguish frequencies.

Pretraining on Double JPEG Detection
It is common practice in the deep learning literature to start training using pretrained weights from a similar task. People often use pretrained weights from image classification for semantic segmentation, especially from ImageNet (Krizhevsky et al., 2012). We also initialize the RGB stream pretrained on ImageNet to extract visual clues more efficiently. However, being the first study to use DCT coefficients as input to a segmentation network, there is no "common practice" to pretrain the DCT stream. We introduce a new pretraining scheme on a double JPEG detection task, classifying single and double compressed JPEG images. The DCT stream is pretrained with various quantization tables on a double JPEG detection task to learn to handle real-world compression artifacts. The pretrained weights are transferred to the image manipulation detection task. Obtaining datasets with various compression parameters for double JPEG detection is much easier than obtaining forgery images with ground truth masks. Ablation studies demonstrate that this pretraining scheme helps the network train faster and achieve higher detection performance (Sect. 5.5).

Network Architecture
This subsection describes how we design the CAT-Net structure. CAT-Net consists of the RGB stream, DCT stream, and fusion stage. The RGB stream takes an RGB image as input and learns image acquisition artifacts, such as sensor pattern noise, EXIF metadata, blocking artifacts, or visual content itself. The DCT stream takes raw DCT coefficients and a quantization table obtained from the JPEG header as inputs and learns compression artifacts. The network is built on top of HRNet (Wang et al., 2020). The structure of the RGB stream is HRNet itself. The DCT stream is a three-resolution variant of HRNet, replacing the first stage with our JPEG artifact learning module.
We adopt HRNet in a forensic task for the first time because it maintains high-resolution representations through the entire process, enabling us to capture the overall picture without losing fine artifacts essential for forensic investigations. With HRNet as the backbone, CAT-Net can acquire fine-grained forensic clues and learn the correlation among different regions. In addition, the HRNet feature map sizes are well suited to tracing JPEG artifacts. Because DCT is applied in 8 × 8 blocks, the minimum resolution the DCT stream can predict is 8 × 8, which is 1 8 th that of the input size (depicted in the yellow feature map in Fig. 5). This resolution can be easily joined by concatenation with the second resolution in the RGB stream. Furthermore, HRNet uses stride-2 convolution to downsample feature maps and does not use pooling layers. Recent studies have demonstrated that pooling is undesirable for tasks that require subtle signals because pooling reinforces content and suppresses noise-like signals (Boroumand et al., 2018). Although this behavior is desirable for computer vision tasks, it is inappropriate for forensic tasks because the noise-like low-level feature is an important clue.

Processing Non-JPEG Images
Although JPEG is one of the most widely used formats for storing image data, many other formats are commonly used. Non-JPEG images may not contain DCT coefficients or quantization tables required for the DCT stream. CAT-Net permits these images (assuming they are uncompressed, e.g., PNG) to be its inputs. In this case, CAT-Net computes DCT coefficients by applying DCT to RGB pixel values and assumes the quantization table is filled with ones. This straightforward approach can be implemented with little difference by initially compressing the image with JPEG quality 100 without chroma subsampling. Therefore, CAT-Net can also process non-JPEG images.
A forged image with uncompressed image format does not imply that the original image is uncompressed. It only implies that the final forged image is saved without compression. Thus, compression artifacts during the image acquisition may exist, so analyzing DCT coefficients may be advantageous. However, due to the use of DCT coefficients and quantization tables, the DCT stream is not suitable for analyzing forensic clues Each is attached at the end of the DCT stream to classify double JPEG images for pretraining (Sect. 4) and localize the forgery using the DCT stream for the ablation study (Sect. 5.5), respectively. RGB stream heads used for ablation study can be similarly constructed using four resolutions.
in images with other compression formats (e.g., HEIC). In this case, CAT-Net must rely on the RGB stream.

Double JPEG Detection
Double JPEG detection is a binary classification task to determine whether a given JPEG image is JPEG compressed once or twice. This task requires the ability to analyze the compression artifacts in an image. Therefore, the DCT stream, a sub-network of CAT-Net, is pretrained on this task to capture rich compression artifacts. The primary purpose of this task is to initialize the image manipulation detection network more efficiently. A classification head is attached at the end to convert the segmentation network to a classification network (Fig. 7).

Datasets
We used 1.054 million singly and doubly-compressed JPEG images provided by Park et al. (2018). They compressed raw images (Gloe and Böhme, 2010;Bas et al., 2011;Dang-Nguyen et al., 2015) with 1,120 distinct quantization tables including 51 standard tables (Q50-Q100) and additional custom tables obtained from requested images to their public forensic web service. Consequently, their dataset closely represents realworld compression parameters. We used 21 thousand images for testing and the rest for training.

Evaluation Metrics
Because this is a binary classification task, accuracy (Acc), true positive rate (TPR), and true negative rate (TNR) are measured. We treat doubly compressed images as positives.  Table 2 illustrates double JPEG detection results. The first mega row presents four methods taking RGB pixels as inputs. Two general computer vision networks, ResNet (He et al., 2016) and HRNet (Wang et al., 2020), cannot learn compression artifacts at all. Neither can the ManTra-Net (Wu et al., 2019) feature extractor part (Image Manipulation Trace Feature Extractor) learn, as reported by the researchers (Fig. 3 of their paper). SRNet (Boroumand et al., 2018) is a steganalysis network designed to trace minute signals, but it cannot learn either. The RGB domain is not suitable for detecting JPEG double compression. The next mega row reveals that two general-purpose networks cannot learn the compression artifacts, supporting our previous claim that CNNs cannot learn the compression artifacts when raw DCT coefficients are supplied directly to them. The third and last mega rows are the methods using the DCT histogram and DCT volume, respectively. The results reveal that the DCT histogram was indeed a suitable feature representing the distribution of DCT coefficients. The DCT volume is also a highly effective representation of compression artifacts. The DCT stream without the quantization table (DCT Stream w/o QT ) differs from the normal DCT stream in which

Implementation Details
The third mega row of Table 2 is from Park et al. (2018), the dataset provider. We performed all other experiments. We used a stochastic gradient descent optimizer with Nesterov momentum (0.9) and weight decay (10 −4 ). The learning rate started from 0.05 and decreased by a factor of 0.1 every 10 epochs. We trained until 30 epochs and report the test result for the highest-performing epoch.

Image Manipulation Detection
This section illustrates the experiments of image manipulation detection and localization. After CAT-Net is initialized using ImageNet classification and double JPEG detection, it starts end-to-end training using authentic and tampered images in a supervised manner. First, we describe datasets used for training and testing. Then, the detailed training methods, baseline methods, and localization evaluation metrics are described.
We then present the quantitative and qualitative localization results. Finally, the ablation studies, effect of compression quality, and robustness tests on additional compression are illustrated. Table 3 summarizes the datasets used in the experiments. We collected nine publicly available datasets. CASIA v2 (Dong et al., 2013) is a popular dataset for copy-move and splicing forgery, images collected from several sources. We used masks provided by a third-party user (Pham et al., 2019) because ground truth masks are not provided officially. Fantastic Reality (Kniaz et al., 2019) includes many spliced images for various scenes along with ground truth masks. Although authentic images have diverse (153) quantization tables, the tampered images have only one quantization table. IMD2020 (Novozamsky et al., 2020) includes real-life manipulated images and manually created ground truth masks. This dataset contains the most diverse quantization tables because images were collected from the Internet and hence reflect real-world compression schemes. NC16 SP (Guan et al., 2019) is a subset of NC16 provided by the National Institutes of Standards and Technology (NIST). NC16 contains high resolution and challenging manipulated images. Although there are several forgery types, we only use splicing forgery. Carvalho (De Carvalho et al., 2013) DSO-1 contains images of people. Forgeries were created by adding one or more individuals from one image to another with postprocessing to increase photorealism. Blocking artifacts are evident when zoomed in, indicating that although the images are not finally saved in JPEG format, the source images were JPEG compressed, leaving compression artifacts. Columbia (Ng et al., 2004) is a historic dataset for manipulation detection. Ground truth masks are obtained by calculating the difference between authentic and forged images followed by post-processing. The images in this dataset were not compressed in a camera, so they left no compression artifacts. GRIP (Cozzolino et al., 2015) contains realistic copy-move forgery images. In that dataset, the ground truth mask contains not only the tampered object region but also the source object region. Thus, we manually remove the source object region so the masks are consistent with the masks in other datasets. CoMoFoD (Tralic et al., 2013) contains copy-move forgeries carefully designed to make forgery detection challenging. The original images were obtained in an uncompressed format. The forged images were heavily postprocessed with JPEG compression, noise adding, image blurring, brightness change, color reduction, and contrast adjustments to hide tampering traces. Manual mask removal of the source object region was also performed on this dataset. COV-ERAGE (Wen et al., 2016) contains copy-move images designed to counter a similarity-based copy-move forgery detector. Similar but genuine objects are included deliberately, inducing many false positives for those detectors.

Datasets
We distinguished training and test datasets to measure the ability to generalize over real-world data, i.e., not splitting the same dataset into training and test components. We used the six smallest datasets for testing and the remaining three for training. Those three datasets contain a limited number of images and limited kinds of quantization tables, insufficient to represent real-world image distribution and compression artifacts. Thus, we created five custom datasets and used them for training (Fig. 8). SP COCO was constructed using the COCO 2017 dataset (Lin et al., 2014). Similar to Wu et al. (2018) and Zhou et al. (2018), spliced images were automatically created by selecting one or more arbitrary objects in one image and pasting them onto another image at random posi-tions, with random rotation and resizing. These images were then compressed. In this paragraph, compression refers to JPEG compression at random quality factor ranges from 60-100. We did not apply additional postprocessing, such as blurring the spliced boundary, because that might mislead the network to act like a blur detector. CM COCO was constructed similarly, but the copied objects came from the same image. CM RAISE was constructed using RAISE (Dang-Nguyen et al., 2015) as an image source but COCO as an object mask. First, the RAISE image was compressed. Then, an arbitrary region was selected using unrelated random polygon annotation from COCO. That region was then pasted within the same image, and finally, the whole image was compressed. This process often creates removal-like forgeries when the background region is selected and copy-pasted. CM-JPEG RAISE was constructed by simply applying additional compression to CM RAISE. This approach mimics the scenario where a forged image is sent through SNS, inducing one more compression. JPEG RAISE is an authentic dataset, created by simply compressing RAISE.

Implementation Details
We initialized CAT-Net weights by pretraining on Im-ageNet (Krizhevsky et al., 2012) classification for the RGB stream and double JPEG classification for the DCT stream (Sect. 4). The network was trained endto-end with authentic and tampered image data. We sampled the balanced number of images in each dataset to construct one epoch and efficiently manage the wide variety of dataset sizes. Accordingly, each epoch did not include all training images but only a subset of them. Training images were cropped to 512 × 512 patches aligned with an 8×8 grid (Sect. 3.3). Full-resolution images were used for testing, which was possible because the proposed network was fully convolutional. The network was implemented in PyTorch (Paszke et al., 2019) using a stochastic gradient descent optimizer with a momentum of 0.9. The batch size was 22. We trained for 200 epochs. The learning rate started from 0.005 and decayed exponentially to 0 at the end. The objective was to minimize the pixel-wise binary cross-entropy loss with fivefold more weight on the tampered class. The experiments were performed using two NVIDIA TITAN RTX graphic cards. We compared our model performance with ten other methods. The code for seven traditional methods was obtained from MKLab (Zampoglou et al., 2017): DBA (Ye et al., 2007), NOI1 (Mahdian and Saic, 2009), ADQ (Lin et al., 2009), NADQ , CFA (Ferrara et al., 2012), NOI2 (Lyu et al., 2014), and CAGI (Iakovidou et al., 2018). We converted output maps in the range [0, 255] to probability maps in the range [0, 1]. The code for three deep neural networks and the trained weights were obtained from official public repositories: EXIF-SC (Huh et al., 2018), ManTra-Net (Wu et al., 2019), and Noiseprint (Cozzolino and Verdoliva, 2019). For EXIF-SC, mean-shift was used for output aggregation. ManTra-Net could not infer some extra-large NC16 SP images with full resolution due to GPU memory constraints (24GB). We cropped these 268 images and their corresponding ground truth images to 2560×1440 (QHD) to test ManTra-Net.

Evaluation Metrics
Our task is a binary segmentation, labeling each pixel in the input image as tampered (positive, 1) or authentic (negative, 0). Thus, each output pixel can be marked as true positive (G:1, P :1), true negative (G:0, P :0), false positive (G:0, P :1), or false negative (G:1, P :0), where G is the ground truth mask and P is the prediction output. G and P are 2D binary arrays with the same size as the input image.
We evaluate network performance using accuracy (Acc), F1 score, and average precision (AP). The accuracy is defined as: Acc(G, P ) = TP + TN TP + TN + FP + FN .
However, the problem of accuracy in forensics is that there are much more negative (authentic) pixels than positive (tampered) pixels in the ground truth image. Thus, outputting all pixels as negative produces high accuracy. The F1 score is used to emphasize the positive class -it is the harmonic mean of precision and recall: Accuracy and F1 score only measure the binary decision map with a fixed threshold. Although the fixed threshold is indeed essential, we also use average precision to measure the performance free from the threshold. The average precision is the area under the precision-recall curve, which measures an average performance among all thresholds. For forgery localization tasks, it is sometimes ambiguous which of the two segments is tampered with.
Thus, based on Huh et al. (2018), we also use the permuted metrics for evaluation, defined as: p-Acc(G, P ) = max Acc(G, P ), Acc(G, P ) p-F1(G, P ) = max F1(G, P ), F1(G, P ) p-AP(G, P ) = max AP(G, P ), AP(G, P ) where negates (flips) the prediction. Permuted metrics measure how well a model can distinguish authentic and tampered regions, not its ability to identify which is which. Some studies use permuted metrics without explicitly specifying 'p-' (Cozzolino and Verdoliva, 2019) or use a similar flipping strategy depending on ground truth (Wu et al., 2019) or prediction (Huh et al., 2018). Furthermore, some papers use varying thresholds per image and report the best value, resulting in much higher numbers (Cozzolino and Verdoliva, 2019;Huh et al., 2018). The authors claim a varying threshold is used to measure the performance without threshold selection ability. However, because that performance is measured via AP, we chose to use the fixed threshold (0.5) for accuracy and F1 score, to strictly measure the detection performance. Each metric is calculated per image and averaged over a dataset. Table 4 presents a performance comparison among eleven methods: seven traditional approaches, three state-of-the-art deep neural networks, and our CAT-Net. The results are depicted for six independent datasets: three splicing datasets and three copy-move datasets. All test datasets are completely unseen during training, i.e., not test splits, to measure the general performance for real-world forgeries. We also report accuracy for authentic images in each dataset, if provided, to observe the false positive rate for untampered images. Figure 9 illustrates some prediction outputs of the six highest-performing methods. Among eleven methods, CAT-Net achieves the highest localization performance for five out of six forgery datasets in terms of both p-F1 and p-AP. In particular, the CAT-Net results for GRIP are surprising (Table 4). CAT-Net achieves 76.45% p-F1 and 91.87% p-AP, while the second-best methods are 10.98% p-F1 (Noiseprint) and 15.12% p-AP (CAGI). The other ten methods could not localize the forgeries, unlike CAT-Net, which significantly outperformed those methods. Although GRIP creators tried not to leave any forgery traces, the acquisition-level compression artifacts remained and those are detected by our detector. However, CAT-Net could not detect forgeries well in CoMo-FoD (14.01% p-F1, 21.46% p-AP), defeated by 22.06% p-AP). The result was caused by the absence of initial compression, which indicates the DCT stream could find no traces. In contrast, although Columbia also does not contain initial compression traces, CAT-Net achieves excellent performance due to the RGB stream. CAT-Net attains 93.97% p-F1 and 95.87% p-AP, while the secondhighest performing method, EXIF-SC, achieves 78.05% p-F1 and 94.50% p-AP. Their method is suitable for Columbia because this dataset is claimed to be uncompressed, so EXIF metadata is unharmed.

Results
The permuted accuracy of authentic images suggests that DBA has the lowest false positives for untampered images. However, because its localization performance for tampered images is low in many cases, we conclude that this method predicts tampered regions relatively less often. CAT-Net achieves a high score for authentic and tampered images, implying that it could be used for image integrity verification. CAT-Net achieves state-of-the-art performance in image manipulation detection and localization. Table 5 and Fig. 10 present the ablation study results. Two substreams are separately trained using the same training settings to observe the contribution of each stream. In some cases, the DCT stream outperforms the RGB stream and, in others, the opposite -depending on the existence of compression artifacts. If the compression artifacts exist, the DCT stream outperforms the RGB stream and vice versa if the DCT stream cannot trace meaningful compression traces. For the datasets without initial compression, the joint performance sometimes decreases because the DCT stream produces unhelpful features, negatively impacting the entire network. Nevertheless, full CAT-Net exhibits the highest overall performance using both streams.

Ablation Studies
The last row of Table 5 illustrates the effect of double JPEG pretraining. CAT-Net w/o D.P indicates CAT-Net started training from random initialization for the DCT stream, not from the pretrained weights using double JPEG detection. For the datasets with compression traces, pretraining on double JPEG detection improves the localization performance significantly. For example, CAT-Net w/o D.P attains 24.60% p-F1 and 43.43% p-AP for GRIP. The performance increases to 76.45% p-F1 and 91.87% p-AP when the training starts from double JPEG initialization. Furthermore, Fig. 11 illustrates that double JPEG pretraining produces faster training. These results are likely due to the various quantization tables used in double JPEG  pretraining (1,120 types). It is challenging to acquire a forged image and a ground truth mask pairs for diverse quantization tables. In contrast, it is easy to ob-tain singly and doubly-compressed images with diverse quantization tables because we may obtain raw images and compress them once or twice. Therefore, we recommend using double JPEG pretraining for future research.
However, when tested on the datasets without useful compression traces, the overall performance decreases when pretrained. For example, CAT-Net w/o D.P exhibits 95.15% p-F1 and 98.21% p-AP for Columbia, which decrease to 93.97% p-F1 and 95.87% p-AP when started from double JPEG pretraining. When we use pretraining, the DCT stream produces more accurate predictions during training, causing the entire network to focus more on the DCT stream than the RGB stream. When evaluating images without compression traces, this stream tries to use unavailable compression traces more frequently, reducing performance. Hence, we conclude that pretraining the DCT stream with double JPEG detection enables rich initialization of the forgery localization task where compression artifacts remain.

Effect of Compression Quality
This subsection analyzes the effect of first and second compression quality. We created forged images similar to Fig. 1(b) but with different JPEG compression qualities. Recall that the forged image contains the copymoved object (the cherry blossom tree) and the spliced object (the sign). The authentic image ( Fig. 1(a)) was first compressed using JPEG quality Q 1 , and another authentic image, the sign, was compressed using JPEG quality 70. Then copy-move and splicing manipulation were applied, followed by the second compression using JPEG quality Q 2 . Figure 12 depicts CAT-Net's localization results with diverse compression qualities. The smaller Q 1 and the larger Q 2 , the higher the localization performance. The diagonal images from the top-left to the bottomright illustrate the special cases when Q 1 = Q 2 . In these cases, CAT-Net had a lower chance of detecting the copy-moved object because the same quantization produces significantly fewer compression artifacts. In contrast, the spliced object was accurately detected even when the same quantization was used in the first and second compression (Q 2 = 70). This is mainly be-cause the spliced object was from a different image, unlike the copy-moved object. The RGB stream detected that the object had different acquisition artifacts from the other area, so CAT-Net could localize the sign as tampered. Furthermore, both objects were hard to detect for images with low second compression qualities (Q 2 ≤ 60) because the strong final compression extensively destroys the forensic clues. Figure 13 illustrates the localization performance when images are JPEG compressed once more. Additional JPEG compression conventionally occurs when the manipulated images are transmitted through the Internet, like posting on social media or sending via messengers. These services often use JPEG compression to reduce storage or bandwidth. Thus, detectors should maintain their performance with additional JPEG compression.

Robustness Tests on Additional Compression
CAT-Net achieves the highest performance in 16 out of 24 settings in terms of p-F1 score and in 23 out of 24 settings in terms of p-AP. Consequently, CAT-Net is robust to additional JPEG compression for various quality factors compared to other neural network approaches and is suitable for detecting real-world forgeries.

Conclusion
We presented a new approach using image compression artifacts to detect and localize image manipulation. This study is the first to accept DCT coefficients directly into a segmentation network, which was possible due to DCT volume representation and specially chosen neural network components. We also introduced a new pretraining method that uses double JPEG detection. Our neural network approach was the first to use both RGB and DCT domain information for forgery localization. Proposed CAT-Net significantly outperformed state-of-the-art forgery detectors. This study is a starting point for using compression artifacts in deep learning-based image forensics. We hope that many future studies will build upon this idea.