{'fraction_non_alphanumeric': 0.03211165323953251, 'fraction_numerical': 0.012027686372404403, 'mean_word_length': 4.964358222422739, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exploiting Fine-grained Face Forgery Clues via Progressive Enhancement Learning

With the rapid development of facial forgery techniques, forgery detection has attracted more and more attention due to security concerns. Existing approaches attempt to use frequency information to mine subtle artifacts under high-quality forged faces. However, the exploitation of frequency information is coarse-grained, and more importantly, their vanilla learning process struggles to extract fine-grained forgery traces. To address this issue, we propose a progressive enhancement learning framework to exploit both the RGB and fine-grained frequency clues. Specifically, we perform a fine-grained decomposition of RGB images to completely decouple the real and fake traces in the frequency space. Subsequently, we propose a progressive enhancement learning framework based on a two-branch network, combined with self-enhancement and mutual-enhancement modules. The self-enhancement module captures the traces in different input spaces based on spatial noise enhancement and channel attention. The Mutual-enhancement module concurrently enhances RGB and frequency features by communicating in the shared spatial dimension. The progressive enhancement process facilitates the learning of discriminative features with fine-grained face forgery clues. Extensive experiments on several datasets show that our method outperforms the state-of-the-art face forgery detection methods.


Introduction
Over the past few years, face forgery technology has made significant progress. With public accessible tools such as deepfakes (Tora 2018) or face2face (Thies et al. 2016), people can easily manipulate the expression, attribution or identity of faces in images. Various powerful algorithms are utilized to generate realistic forged faces, which are hardly discerned by human eyes. These realistic forged faces have been used in pornography or political rumors, which are harmful to the community and evoke social panic. Under this background, the face forgery detection task was born and has attracted more and more attention recently.
Many significant works (Chollet 2017; Li et al. 2020a;Dang et al. 2020;Chen et al. 2021;Qian et al. 2020;Zhao et al. 2021;Gu et al. 2021) have made great contributions tracted by universal CNN (Afchar et al. 2018;Chollet 2017) to conduct binary classification. Recently, two typical manners for mining subtle task-specific artifacts have been explored. On the one hand, some works (Dang et al. 2020;Li et al. 2020a;Chen et al. 2021;Zhao et al. 2020;Wang et al. 2020) utilize auxiliary supervision such as blending boundary (Li et al. 2020a) or forged mask (Dang et al. 2020;Chen et al. 2021). However, the mask ground-truth they rely on is difficult to access in most cases. On the other hand, some attempt to employ the frequency-aware features to assist classification, using DCT (Qian et al. 2020), local frequency  or SRM (Luo et al. 2021). Although these works have made considerable performance, the frequency features extracted by their strategies are relatively coarse, which burdens the network to assemble discriminative feature patterns. Moreover, all these frequencyaware approaches either directly concatenate RGB and frequency features at the end of the network or fuse them only at a shallow layer. This makes it difficult for them to fully utilize the decomposed frequency information and limits the possibility of discovering fine-grained face forgery cues. To address the above issues, we propose a novel Progressive Enhancement Learning (PEL) framework aiming at exploiting face forgery clues. Specifically, we employ the patch-wise Discrete Fourier Transform  with the sliding window to decompose the RGB input into fine-grained frequency components. Frequency domain information is assisted and complementary to the RGB features. And different from conventional frequency components, fine-grained frequency components can produce more combinations of features to discover potential artifacts.
To fully exploit RGB image and fine-grained frequency components, we design a two-stream network structure with two novel enhancement modules that progressively enhance the forgery clues. One is the self-enhancement module. It captures the traces in different input spaces separately based on spatial noise enhancement and channel attention. The other is the mutual-enhancement module, which achieves concurrent enhancement of both the RGB and frequency branches through feature communication in the shared spatial dimension. We insert these two enhancement modules after each convolutional block of the network. The earlier enhancements facilitate feature extractions in the later layers, and the forgery cues hidden in the fake faces are sufficiently excavated and magnified.
Extensive experiments are conducted to demonstrate the effectiveness of our method. With the progressive representation enhancement, better results are obtained on highquality forged images. Our method exceeds the other comparison methods and achieves state-of-the-art performance on the common benchmark FaceForensics++ dataset and the newly published WildDeepfake dataset. The cross-dataset evaluations on three additional challenging datasets prove the generalization ability, while the perturbed evaluations prove the robustness of our method.
Our contributions can be summarized as follows: • We propose a progressive enhancement learning network aiming at exploiting the combined fine-grained frequency components and RGB image. • We first perform fine-grained decomposition in the frequency domain, and then propose two novel enhancement modules, i.e., self-enhancement and mutualenhancement modules, which progressively enhance the feature to capture the subtle forgery traces. • Extensive experiments and visualizations reveal the significance of feature enhancement in face forgery detection, and demonstrate the effectiveness of our proposed method against the state-of-the-art competitors.

Related work
Due to the rapid development of face forgery methods, face forgery detection has recently attracted more attention in computer vision communities. Great efforts have been made to explore the authenticity of human faces. Early works use hand-crafted features to seek for the artifacts in forged faces. Fridrich et al. (Fridrich and Kodovsky 2012) extract steganalysis features and train a linear Support Vector Machine (SVM) classifier to distinguish authenticity. After that, Cozzolino et al. (Cozzolino, Poggi, and Verdoliva 2017) cast the features mentioned above to a convolution neural network. Matern et al. (Matern, Riess, and Stamminger 2019) analyze the facial regions which are likely to generate artifacts and design specific descriptors to capture them. Yang et al. (Yang, Li, and Lyu 2019) estimate the inconsistency of head poses between adjacent faces. With the rapid progress of deep learning, some CNN-based works (Bayar and Stamm 2016;Afchar et al. 2018; Chollet 2017) emerge to extract high-level semantic information for classification. However, their vanilla structures are unsuitable for the fast-growing and realistic forgery faces.
To further mine the heuristic forgery cues, two typical manners are widely used. One is resorting to auxiliary supervisions. For example, Nguyen et al. (Nguyen et al. 2019) incorporate the classification and segmentation task into one framework simultaneously. Dang et al. (Dang et al. 2020) further highlight the manipulated area via attention mechanism. Face X-ray (Li et al. 2020a) focuses on the blending boundary induced by the image fusion process. Besides, some works Zhao et al. 2020) resort to the similarity between local regions and achieve superior performance on benchmark datasets. However, the manipulated mask above methods relied on training is not applicable to some wild datasets (e.g., Wild Deepfake dataset (Zi et al. 2020)), limiting practical application.
The others notice the artifacts hidden in the frequency domain that are difficult to observe directly. Singhal et al. (Singhal et al. 2020) extracts frequency features through the residuals of image median filtering for detecting manipulated faces. Frank et al. (Frank et al. 2020) reveal the peculiar frequency spectrum pattern of GAN-generated deepfake images. F 3 -Net (Qian et al. 2020) employ the frequency information in two ways: image components which are transformed from a certain frequency band, and local frequency statistics calculated by frequency spectrum of patches. Chen et al. ) use high-frequency components of the image to assist in calculating patch-similarities. Luo et al. (Luo et al. 2021) use SRM filter to guide RGB features. FDFL ) utilize frequency information by DCT and adaptive frequency information mining block. Although these methods have achieved remarkable performances, the exploitation of frequency information is coarsegrained. More importantly, their vanilla learning process is insufficient to make full use of the subtle artifacts in the frequency domain. Different from existing works, we propose a novel progressive enhancement learning network that exploits the fine-grained frequency components. Fig. 1 illustrates the proposed PEL framework. We transform the RGB input into the fine-grained frequency component and fed them together into a two-stream network with EfficientNet as the backbone. Each convolutional block of the backbone is followed by a self-enhancement module and a mutual-enhancement module to progressively enhance the fine-grained clues in an intra-and inter-stream manner.

Fine-grained Clues
As shown in the leftmost of Fig. 1, we apply domain transformation that decomposes the input RGB image into frequency components, revealing fine-grained forgery cues hidden in the frequency space.
Without loss of generality, let x rgb ∈ R 3×H×W denotes the RGB input, where H and W are the height and width of image. First, the RGB input x rgb is transformed to YCbCr space (denoted by x ycbcr ∈ R 3×H×W ) that coincides with the widely-used JPEG compression in the forged video. Then, we slice x ycbcr via a sliding window to obtain a set of 8 × 8 patches. p c (i,j) ∈ R 8×8 denotes the patch sliced by the (i th,j th) sliding window of a certain color channel.
, where each value corresponds to the intensity of a certain frequency band. We flatten the frequency spectrum and group all components of the same frequency into one channel to form a new input, following the patch location in the original input:

Mutual-Enhancement
Concat 1×1 Conv  Figure 1: The proposed PEL framework. The RGB input is decomposed into frequency components to mine fine-grained clues hidden in frequency space. In the self-enhancement module, the features of each stream are enhanced separately using a noise enhancement block and a channel attention block. In the mutual enhancement, both features are enhanced in the shared spatial dimension. The two enhancement modules are selectively inserted after each convolutional block of the network.

Self-Enhancement
where x f lat ∈ R 192×H2×W2 is the newly formed input, H 2 and W 2 are the vertical and horizontal step numbers of sliding window, and d (i,j) ∈ R 3×8×8 denotes the concatenation of all the d c (i,j) . In this way, the original color input is decomposed and recombined to fine-grained frequency-aware data while maintaining the spatial relationship to match the shift-invariance of CNN. Finally, a 1×1 convolution layer is applied to the frequency data to adaptively extract the most informative frequency bands, forming the fine-grained frequency input x f req ∈ R 64×H×W .

Self-Enhancement Module
The self-enhancement module consists of two mountable sub-modules: a noise enhancement block and a channel attention block, which enhance the features of each stream separately. The visual artifacts in newly forged faces have significantly been eliminated (Li et al. 2019b), pushing subtle noise to a more critical role in the face forgery detection task. To this end, we design a noise enhancement block to excavate the artifact hidden in the feature noise, especially that of the fine-grained frequency input.
Given the RGB input x rgb and the frequency input x f req , let f rgb in ∈ R c×h×w and f f req in ∈ R c×h×w denote the feature maps extracted by intermediate convolutional block of RGB stream and frequency stream, where c is the number of channels, h and w denote the height and width of the feature maps, respectively. Since the RGB and frequency streams have the same network structure except for the first convolutional layer, we use the superscript s ∈ {rgb, f req} to indicate different stream.
For noise enhancement, we first extract the noise of input feature maps f in as follows: where M is the median filter with a kernel size of 3 × 3 that can filter noise such as salt to denoise at the feature level. By subtracting the input features and denoising features spatially, f noi manifests the noise information of the input feature.
Then we apply a Sigmoid function (denoted by σ) to amplify the subtle noises and suppress the exaggerated value of noise to prevent polarization. After that, a depth-wise 1 × 1 convolution layer is applied to the feature noise to adjust the amplitude of noise enhancement. Finally, the feature noise is added to the input feature f in to obtain the noise-enhanced feature f ne , which is used as input of the subsequent block. The above process can be summarized as follows: f s ne = f s in + Conv(σ(f s noi )).
(3) Since the noise enhancement block considers the lowlevel noise of each channel separately, we further utilize a channel attention block to facilitate inter-channel interactions within each stream. Specifically, the channel attention block enhances features along the depth dimension, which combines the information from all channels and determines the significance of each channel, and enhances them accordingly. The output feature f out is obtained as: where M LP is a multi-layer perceptron, GAP denotes the global average pooling, and GM P is the global max pooling.
In each stream of the network, the noise enhancement module is inserted after the first two convolutional blocks. This is because high-level features capture semantic information within a large reception field and are not suitable to extract the noise with local concepts. And the channel attention block is empirically mounted after the second to the penult convolutional blocks. With the self-enhancement module, features are enhanced within the stream without changing their size, providing more helpful features for the following mutual-enhancement module.

Mutual-Enhancement Module
As the features of two streams are originated from different input spaces, they provide distinct cues that can complement each other. Thus, we design a mutual-enhancement module to comprehensively integrate the dual-stream knowledge along spatial dimension to enhance the features.
Concretely, the dual feature maps are concatenated and go through a point-wise convolution and a Sigmoid function to get two spatial attention maps for two streams, respectively: where A ∈ R 2×h×w is the spatial attention maps, Conv denote the point-wise convolution layer, and Cat concatenate the features along with the depth. The input feature of each stream is multiplied with its attention map (i.e. a rgb and a f req which are the two channels of A), and then go through a depth-wise 1 × 1 convolution layer to adjust the enhancing intensity. Finally, we add the enhanced feature with the input feature as follows (the feature size remains the same): The mutual-enhancement module is inserted after the second to the penult convolutional blocks and placed behind the self-enhancement module. In this way, we can highlight forged regions that benefit from the other stream while maintaining stream-specific fine-grained forgery clues, which result in more discriminative features.

Loss Function
In the preceding blocks, the two features enhanced by the mutual-enhancement module are fed into the next convolution blocks. In the last block, the outputs of two streams are concatenated for classification. We adopt the widely-used Binary Cross-Entropy loss as the objective function: L = − 1 n (y gt ×log(y pred )+(1−y gt )×(log(1−y pred )), (7) where y gt is the ground truth label, and y pred denotes the prediction of the network. The parameters of the network are updated via the back-propagation algorithm.

Experiment
In this section, we first introduce the overall experimental setup, then present extensive experimental results to demonstrate the effectiveness and robustness of our approach, and finally visualize the enhanced fine-grained face forgery clues and class activation map.

Experimental Setup
Datasets. We adopt five widely-used public datasets in our experiments, i.e.,  (Kowalski 2018) and NeuralTextures (NT) (Thies, Zollhöfer, and Nießner 2019). Different levels of compressions are used on raw videos, and two counterparts are generated, i.e. high quality (HQ) and low quality (LQ). We follow the official splits by using 720 videos for training, 140 videos for validation, and 140 videos for testing.
• WildDeepfake is a dataset with 7,314 face sequences entirely collected from the internet, in which 6,508 sequences are for training and 806 are for testing. The contents in this dataset are diverse in many aspects, e.g. activities, scenes, and manipulation methods, which makes this dataset more challenging and closer to the real-world face manipulation scenario.
• Celeb-DF is composed of 590 real videos based on 59 celebrities from youtube, and the corresponding 5639 fake videos tampered with the improved DeepFake algorithm (Li et al. 2020b).
• DeepfakeDetection (DFD) is a large dataset that contains over 3000 manipulated videos in various scenes using publicly available deepfake generation methods.
• DeepfakeDetectionChallenge (DFDC) is a prominent face swap video dataset, with over 100,000 total clips sourced from 3,426 paid actors, produced with several Deepfake, GAN-based, and non-learned methods.
Implementation detail. We implement the proposed framework via open-source PyTorch (Paszke et al. 2017). We sample 50 frames per video at equal interval, then use the stateof-the-art face extractor DSFD (Li et al. 2019a) to detect faces and resize them to 320 × 320 (training of FF++) or 224 × 224 (training on WildDeepfake). The EffcientNet-B4 (Tan and Le 2019) pre-trained on ImageNet was adopted as the backbone of our network, which is trained with Adam optimizer with the learning rate of 2 × 10 −4 , the weight decay of 1 × 10 −5 , and the batch size of 32. The stride of the sliding window is set to 2 in all experiments. The results of comparisons are obtained from their paper, and we specify our implementation by † otherwise.

Experimental Results
Intra-testing. Following (Rossler et al. 2019;Qian et al. 2020), we compare our method against several advanced techniques on the FF++ dataset with different quality settings (i.e., HQ and LQ), and further evaluate the effectiveness of our approach on the WlidDeepfake dataset that is closer to the real-world scenario. The quantitative results are shown in Tab.1. It can be observed that our proposed method outperforms all the compared methods in both Acc and AUC metrics across all dataset settings. These gains mainly benefit from the proposed PEL framework that can sufficiently excavate and magnify the fine-grained forgery cues.
Cross-testing. As new forgery methods are emerging all the time, the generalizability of detection models directly affects their application under real-world scenarios. To test the generalization ability, we perform cross-dataset evaluation, i.e., training on one dataset while testing on another different dataset. This is more challenging since the testing distribution is different from that of training. We compare our method against four competing methods through crossdataset evaluations, as shown in Tab. 2. Specifically, we first trained these models on FF++ (LQ) and then evaluated their AUC and EER metrics on WildDeepfake, Celeb-DF, DFD and DFDC, respectively. From Tab. 2 we can observe that our method outperforms all the competitors significantly on all testing datasets. We further trained our model on Wild-Deepfake and evaluated it on the other four datasets, obtaining similar results that our method substantially outperforms the compared methods in nearly all cases. Robustness. In the process of video acquisition and transmission, various noises such as blur or salt will be introduced to the digital data; hence the robustness towards perturbation of the detection model is essential to the real-world application. Thus, it's important for a model to be robust to all kinds of perturbation. A series of experiments are conducted to verify the robustness of our method. Specifically, we apply Gaussian noise, salt and pepper noise, and Gaussian blur to the test data, and measure the decay of Acc and AUC (denote as ∆Acc and ∆AUC) to assess the robustness of the detection model, as shown in Tab. 3. The results turn out that the performance of our method degrades the least in most cases, which mainly benefits from our novel noise enhancement mechanism.

Ablation study
Components. As shown in Tab.4, we develop several variants and conduct a series of experiments on the FF++ (LQ) dataset to explore the influence of different components in our proposed method. In the single-stream setting, using only the integral RGB data or the fine-grained frequency data as input leads to similar results, and our proposed two modules can enhance the performance a little (the mutualenhancement module is replaced with a spatial attention). In the two-stream setting, combining the original two stream can slightly improve the performance, which verifies that the fine-grained frequency input is distinct and complementary to the RGB data. The performance can be improved by adding the proposed self-enhancement module or mutualenhancement module, reaching the peak when using the overall PEL framework. This shows the effectiveness of each module: the self-enhancement module excavates the essential information within each stream separately, and the mutual-enhancement module enhances the above information by integrating them. Denoising methods. We seek a suitable denoising method by exploring filters with different types, kernel sizes and combinations. Tab. 5 lists the result on FF++ (LQ) dataset. For the filter type, we use the median filter, the mean filter and the non-local means (Buades, Coll, and Morel 2005), in which the median filter beats the other two. For the kernel size, we set it as 3, 5, 7, respectively, and it turns out that the smallest kernel performs best. We further combine multiple median filters to extract noise with different size regions, followed by 1 × 1 group convolutional layer to reduce the num-  ber of channels. However, the single median filter seems to be better than multiple filters. Based on the above analysis, we choose the single median filter with a kernel size of 3 as the default denoising setting of our method under all experiments.

Visualizations
Class Activation Mapping. To explore the region of interest for different face types, we visualized the attention maps of several samples by class activation mapping (CAM) (Zhou et al. 2016). As shown in Fig. 2, both RGB and frequency streams can locate the artifacts, but with different emphasis. The RGB stream process inputs in the original image space with relatively focused attention maps: for the forged images, the focus area mainly coincides with the mask of forged regions; while for real images, the attention is evenly focused. The frequency stream considers a larger area (i.e. the entire face), which is often complementary to the focus area of the RGB stream, owing to the fine-grained frequency components' ability to reveal more subtle artifacts. In this manner, two streams can contribute to each other for a better forgery clues extraction. Self-enhancement. Fig. 3 illustrates the enhanced region of feature maps (i.e., the residual between f in and f out in Sec. 3.2) via the first self-enhancement module, where mask denotes the forged region which is obtained from the pixellevel difference between a forged image and its corresponding original image. From Fig. 3, we can observe that although the overall feature distributions of real and fake are similar, the latter manifests a higher enhancement in local manipulated regions, such as eyes or mouth. Such a finegrained self-enhancement mechanism enables our method to uncover subtle forgery cues.
Mutual-enhancement. Fig.4 illustrates the representation residual before and after the last mutual-enhancement module. With the cooperation of the Complementary RGB feature and frequency feature, the manipulated regions on both streams are enhanced simultaneously, increasing the discriminability between real and fake faces. Moreover, both streams suppress the facial region in real images while intensifying that in fake images, enhancing the distinctiveness of fine-grained clues. We also observe different enhanced regions between RGB and frequency streams, i.e., the former fit manipulated mask well, while the latter cover whole face with more subtle cues. The visualization illustrates that RGB and frequency information are independent but can benefit from each other.

Conclusion
In this paper, we propose a Progressive Enhancement Learning (PEL) framework to conduct the discriminative representation enhancement based on fine-grained frequency information. Firstly, we decompose the image into finegrained frequency components and fed it into a two-stream network together with the RGB input, which aids in the subsequent two enhancement modules. The self-enhancement module captures the traces in different input spaces based on spatial noise enhancement and channel attention. The mutual enhancement module achieves concurrent enhancement of both branches through feature communication in the shared spatial dimension. The PEL framework can fully exploit the fine-grained clues in high-quality forgery faces and achieves state-of-the-art performance on both seen and unseen datasets. Visualizations of feature maps and class activation mappings reveal the inner mechanism and explain the effectiveness of our method.