{'fraction_non_alphanumeric': 0.033426573426573424, 'fraction_numerical': 0.017846153846153845, 'mean_word_length': 4.881065964796842, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Spatial-Temporal Frequency Forgery Clue for Video Forgery Detection in VIS and NIR Scenario

In recent years, with the rapid development of face editing and generation, more and more fake videos are circulating on social media, which has caused extreme public concerns. Existing face forgery detection methods based on frequency domain find that the GAN forged images have obvious grid-like visual artifacts in the frequency spectrum compared to the real images. But for synthesized videos, these methods only confine to single frame and pay little attention to the most discriminative part and temporal frequency clue among different frames. To take full advantage of the rich information in video sequences, this paper performs video forgery detection on both spatial and temporal frequency domains and proposes a Discrete Cosine Transform-based Forgery Clue Augmentation Network (FCAN-DCT) to achieve a more comprehensive spatial-temporal feature representation. FCAN-DCT consists of a backbone network and two branches: Compact Feature Extraction (CFE) module and Frequency Temporal Attention (FTA) module. We conduct thorough experimental assessments on two visible light (VIS) based datasets WildDeepfake and Celeb-DF (v2), and our self-built video forgery dataset DeepfakeNIR, which is the first video forgery dataset on near-infrared modality. The experimental results demonstrate the effectiveness of our method on detecting forgery videos in both VIS and NIR scenarios.


Introduction
In recent years, with the widespread application of Deep Neural Network (DNN) and Generative Adversarial Networks (GAN) technologies, face generation, and editing technology have become more and more realistic and controllable. However, the abuse of this technology has caused widespread public concern. The current mainstream face forgery technologies include face replacement (Li et al. 2020a;Chen et al. 2020;Li et al. 2021a;Zhu et al. 2021;Agarwal and Farid 2021), expression replay (Xue et al. 2020;Yao et al. 2020;Burkov et al. 2020;Song et al. 2021), attribute editing (Gao et al. 2021;Afifi, Brubaker, and Brown 2021;Xu et al. 2021;Fu et al. 2021), and non-target face generation (Pidhorskyi, Adjeroh, and Doretto 2020;Park et al. 2020;Karras et al. 2020). Face replacement refers to replacing the entire face of one target with the face of another. Face expression replay refers to synchronizing the facial expression of the original video into the target video to reproduce the expression. Face attribute editing usually modifies specific attribute features of face images such as skin color, hair color style and whether to wear glasses or not, etc. Non-target face generation refers to the generation of people who do not exist in the real world, which can be accomplished with the help of GAN. With the rapid development of these technologies, the generation of fake images or videos has become more and more controllable. It is extremely hard to distinguish the real and fake videos on social media only with naked eyes. Thus, the malicious spread of these fake videos on the Internet has arose great challenges for society and citizens. For example, forged videos may affect national security, especially tampering with inflammatory videos of some leaders to spread through the Internet may affect the relationship between the two countries and even lead to a war. Fake videos deliberately forged by criminals are likely to have serious public opinion impacts on citizens. On the other hand, near-infrared (NIR) technology has been widely used in heterogeneous face recognition systems because of its advantage that it is robust to illumination. (Wang et al. 2020a) pointed out that due to the difference in spectral components between NIR and VIS modalities, visible light and near-infrared (VIS-NIR) face recognition is still a challenging task and proposed a method of converting visible light into near-infrared modality to help the near-infrared face recognition system. Furthermore, the NIR-based face recognition systems are also threatened by face forgery attacks. For example, an attacker can inject a fake near-infrared face into the NIR-based access control system to escape identity authentication and thus avoid tracking; the NIR-based face unlocking technology used in the Xiaomi 8 phone was successfully cracked by simply using a black and white printer printout and adding shadows around the eyes and nose of the face to give it a more three-dimensional appearance. In addition, near-infrared is often used for night video surveillance because it is robust to illumination. However, if the NIR surveillance images collected in the night scene are forged to create rumors out of nothing and instigate the spread of public opinion, it will damage the reputation of citizens (Park et al. 2017). Therefore, research on forgery detection in NIR modality is also very important. Considering the wide application of authentication under the NIR scenario, the severe challenges of face forgery and detection in the NIR modality should be considered (Wang et al. 2022).
To address these challenges, a lot of research efforts have been dedicated to the field of face forgery detection. Approaches for face forgery detection can be broadly divided into two categories image/frame-based and videobased methods. [44] utilize the inconsistency in estimating 3D head pose from faces for deepfake detection. [23] combined Representation learning (ReL) and Knowledge Distillation (KD) to enhance the generalization ability of forgery detection models against different generative techniques. (Yu, Davis, and Fritz 2019) first proposed the GAN fingerprint phenomenon, it found that images generated by different GAN models leave diverse features which are called fingerprint features. Inspired by the principle that synthesized images in the frequency domains are often more able to reflect the visual artifacts, some attempt to obtain information from frequency domains, such as Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT). For example, (Frank et al. 2020) analyzed the real and GAN-generated images and found that the images manipulated by GANs perform obvious grid-like traces in the frequency domain. It is proved that the Discrete Cosine Transform (DCT) spectrum is more linearly separable than the RGB domain. Wang et al. (Wang et al. 2020b) proposed that the GAN artifact on the DCT spectrum is arisen by the operation of upsampling. Extensive experiments show that adding perturbation can remove the subtle artifact to a certain extent. (Dzanic, Shah, and Witherden 2020) showed that deep network-generated images share an observable, systematic shortcoming in replicating the attributes of these high-frequency modes, and modeled this clue to detect forgery images. However, (Chandrasegaran, Tran, and Cheung 2021) believes that the high-frequency Fourier spectral attenuation difference is not an inherent defect of CNN-generated models, and these shortcomings can be erased by modifying the upsampling operation. Recently, more and more researchers believe that the image/frame-based method ignores the temporal inconsistencies between frames. Therefore, they directly extract video-level features for forgery detection. (Li et al. 2020b) proposed Sharp Multiple Instance Learning (S-MIL) extract spatio-temporal features to fully capture the intra-frame and inter-frame inconsistency. (Chugh et al. 2020) proposed a deepfake video detection method by comparing the feature differences between audio and video sequences. However, existing methods did not take both the spatial frequency domain and the temporal clue into account which restricts the performance and robustness of video forgery detection.
In this paper, we aim to propose a Discrete Cosine Transform-based Forgery Clue Augmentation Network (FCAN-DCT) for video forgery detection to achieve a more comprehensive representation of fake videos. FCAN-DCT consists of a backbone network and two branches: Compact Feature Extraction (CFE) module and Frequency Temporal Attention (FTA) module. The backbone network first extracts the feature map of the video sequences and then converts the feature map of the RGB space to the frequency domain through 2D-DCT and the converted feature maps can be considered as frequency spectrums. However, due to the discreteness of the DCT transform, the spectrum information extracted by the backbone network is not compact. Considering that the frequency components in the h and w axes on the DCT spectrum gradually increase from low to high, CFE adopts a block strategy and uses the maximum value in each block to represent the weight of each of the frequency domain components. Therefore, a more compact spatial frequency domain representation is obtained. FTA uses the channel attention mechanism on the frequency spectrum to obtain the temporal attention map of the entire frame sequences and makes a feature fusion with FTA to extract the spatial-temporal features among different frames. Therefore, FCAN-DCT learns a comprehensive and compact forgery feature representation for forgery videos in the frequency domain. We have conducted sufficient experiments on WildDeepfake (Zi et al. 2020) and Celeb-DF (v2) (Li et al. 2020c) respectively, and the experimental results have outperformed the state-of-the-art methods. In addition, face forgery detection in multimodal scenarios is rarely explored except for (Wang et al. 2022) which is a NIR image forgery dataset. Therefore, we further construct a NIR video forgery dataset, DeepfakeNIR, to evaluate the robustness and generalization ability of our method and promote the research on video forgery detection in multimodal scenarios. Moreover, in order to sinumate the real scenerao and increase the challenge of our dataset, we also conduct some perturbations such as Gaussian blur, Gaussian white noise and JPEG compression on DeepfakeNIR and the perturbed DeepfakeNIR are called DeepfakeNIR++. In summary, the main contributions are as follows: • We focus on the spatial-temporal frequency forgery clue for video forgery detection by paying attention to the discriminative features among frame sequences, which fills the gap of video forgery detection in the frequency domain.
• We propose a Discrete Cosine Transform-based Forgery Clue Augmentation Network (FCAN-DCT). The CFE and FTA modules make full use of the information in different video frames to obtain compact and discriminative features respectively. These two branches are combined to achieve a more comprehensive spatial-temporal feature representation in the frequency domain.
• We firstly perform video forgery detection experiments on both VIS and NIR scenarios, including two wildly used datasets WildDeepfake, Celeb-DF (v2), and our newly constructed DeepfakeNIR dataset, to evaluate the effectiveness and robustness of the proposed method.
• We present a novel video forgery detection dataset (DeepfakeNIR) with diverse distributions in terms of posture, occlusion, and expression. There are a total of 3,816 NIR videos, where the ratio of real and fake NIR videos is 1:

Related work
In recent years, with the significant advancements made in the generation of deepfakes, tampered facial images and videos have caused a significant amount of negative impact on social media, and also motivated methods to detect these forged videos. Existing methods define video forgery detection as a binary classification problem and develop imagelevel and video-level solutions.

Facial Image Forgery Detection
Early face generation technologies often directly show obvious visual artifacts and inconsistencies in the facial area due to their uncontrollable characteristics. Many previous works employ face or head statistical inconsistencies for image forgery detection. For example, the inconsistency in estimating 3D head pose from face images is introduced in (Yang, Li, and Lyu 2019) to determine the authenticity of an image. Through the unremitting efforts of these works, facial image forgery detection has achieved remarkable success in different aspects. (Yang et al. 2021) argues that the features extracted by the classification framework couple a lot of content-related information. It proposed a GAN Fingerprint Disentangling Network (GFD-Net) to disentangle the fingerprint feature from GAN-generated images. (Kim, Tariq, and Woo 2021) perform domain adaptation tasks employing the Representation Learning (ReL) and Knowledge Distillation (KD) paradigms to counter with various deepfake generation techniques. However, with the development of DeepFake generation techniques, these imagebased detectors may fail to capture temporal inconsistencies across multiple frames. A large number of research efforts have been dedicated to exploring the video-level features.

Facial Video Forgery Detection
The early forgery generation technology is relatively limited, and it may show obvious temporal inconsistency between frames visually. Physiological signal based on blink frequency which is not well represented in synthetic fake videos is explored in (Li, Chang, and Lyu 2018). Agarwal et al. (Agarwal et al. 2019) found that the tampering with the face area in the video can cause the pattern of facial expressions and head movements inconsistent with the person's identity when speaking. In this way, soft-biometric models of individual national leaders were established, and these models were used to distinguish between real and fake videos. However, this approach is not universal. Recently, with the widespread application of Deep Neural Network (DNN) and Generative Adversarial Networks (GAN) technologies, face generation and editing technology have become more and more realistic and controllable but it also brings more security issues to digital forensics. Aiming at the partial faces manipulation problem in DeepFake video detection, (Li et al. 2020b) proposed Sharp Multiple Instance Learning (S-MIL) in which a spatial-temporal encoded instance is designed to fully capture the intra-frame and inter-frame inconsistency. (Chugh et al. 2020) proposed a deepfake video detection method based on the dissimilarity between audio and video sequences. (Hosler et al. 2021) proposed a novel method to justify the authenticity of videos by detecting the consistency of the emotions predicted by the speaker's face and voice. (Haliassos et al. 2021) detected forged videos based on high-level semantic irregularities in lip movements, which can be generalized to new forgery methods and various perturbation operations. (Zheng et al. 2021) investigated the effective temporal cues for more robust video face forgery detection and proposed a general and flexible framework which combined fully temporal convolution network (FTCN) and Temporal Transformer to detect temporal incoherence explicitly.

Frequency-based Forgery Detection
Considering that synthesized images in the frequency domain are often more able to reflect the visual artifacts, a lot of efforts have been dedicated to the field of forgery detection in the frequency domain, such as Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT). (Dzanic, Shah, and Witherden 2020) showed that the decay rate of the high-frequency component in the Fourier spectrum is clearly distinguishable between real and deep network generated images. Using this clue, it proposed a novel model to detect synthesized images. (Chandrasegaran, Tran, and Cheung 2021) argued that the high-frequency Fourier spectral attenuation difference is not an inherent feature of the CNN-based generative model, and this forgery clue can be erased by slightly modifying the upsampling operation, so the identification based on this feature is not robust. (Qian et al. 2020) conducted DCT operation on the image, and then inversely transforms back to the spatial domain based on the low-frequency, mid-frequency and high-frequency information to obtain the spatial components of different frequency bands, which were finally sent to CNN to extract features. (Li et al. 2021b) argued that fixed filters and hand-crafted features are not sufficient to extract the forgery feature from the frequency domain. Therefore, single-center loss (SCL) was proposed to make the intraclass features of the real class more compact. All the aforementioned frequency-based methods confine to exploring the subtle spatial artifact but pay little attention to the temporal frequency clue among different frames. Our proposed FCAN-DCT not only explores the spatial frequency feature but also augments the temporal clue in the frequency domain with the attention mechanism. Thus, The final spatialtemporal feature learned by our method can achieve a more comprehensive representation.

PROPOSED APPROACH
Aiming at filling the gap of analyzing the entire video sequence instead of a single image in the frequency domain, we introduce the Discrete Cosine Transform-based Forgery Clue Augmentation Network (FCAN-DCT) for forgery detection. FCAN-DCT consists of a Compact Feature Extraction (CFE) module in section and a Frequency Temporal Attention (FTA) module in section . Fig. 1 shows the framework of FCAN-DCT. The goal is to make full use of the rich information in the frame sequences which is not considered in the aforementioned methods, explore the spatial-temporal frequency clue among multiple frames, and finally achieve a more comprehensive feature representation in the frequency domain.

Preliminaries
Since the tampering area in the forgery task is only limited to the face area, in order to prevent other areas from affecting the final detection result, we first use the Facenet (Schroff, Kalenichenko, and Philbin 2015) to extract the landmarks of the face and then extract the face area in each frame. Then, we uniformly sample N cropped frames as the video input sequence of the model I = {f 1 , f 2 , f 3 , . . . f N }. Each selected frame is input to the backbone network and finally outputs the feature map representation of each frame M = {m 1 , m 2 , m 3 . . . m N }. We transform these resulting feature maps into the frequency domain using the Discrete Cosine Transform (DCT). Much like the Discrete Fourier Transform (DFT), DCT represents Figure 2: The weight matrix W for the 2D-DCT spectrum. We learn discriminative features by enhancing the intensity of midium and high-frequency components in the frequency domain to amplify the difference between real and fake images.
a sequence of points in space as the sum of cosine functions at different frequencies. Given a pixel matrix in RGB space f ∈ R N ×N , the 2D-DCT is defined as: where u, v are the coordinates of the DCT spectrum matrix D (u, v) ∈ R N ×N . Compared with DFT, DCT has better energy concentration in the frequency domain, and can directly crop those unimportant frequency domain regions and coefficients, so it is widely used in image compression tasks, such as jpeg compression. In recent years, with the rapid development of deep learning, the operation of DCT also attracted considerable interest in forgery detection tasks.

Compact Feature Extraction Module
We adopt Compact Feature Extraction (CFE) module to present a compact spatial feature representation. CFE is mainly divided into two steps: 1) Use the 2D-DCT in equation (1) to transform the input pixel feature map M = {m 1 , m 2 , m 3 . . . m N } to the DCT frequency domain D = {d 1 , d 2 , d 3 . . . d N }, while retaining more important information and discarding unimportant information. At the same time, artifacts that appear to be fake images are shown. 2) Compress the obtained frequency domain sequence to obtain a more compact feature representation. Let M ∈ R N ×C×H×W represent the feature map sequence of the frame set extracted by the backbone network. According to equation (1) mentioned above, we convert the obtained 2D spatial pixel feature map to the frequency domain, and the obtained 2D spectrum is recorded as D ∈ R N ×C×H×W . Inspired by (Frank et al. 2020;Wang et al. 2020b;Dzanic, Shah, and Witherden 2020;Chandrasegaran, Tran, and Cheung 2021;Li et al. 2021b), which think the component present on the spectrum in midium and high-frequency bands is pretty inconsistent between real and synthesized images, we believe that we should make use of the differences in the midium and high-frequency components as much as possible and increase the degree of influence of these parts in the frequency spectrum. Therefore, a weight matrix W is designed as Fig. 2. Here, β = √ 2. For 2D DCT, the equation can be expressed as: where u, v are the coordinates of the DCT spectrum, and H is the height of the spectrum. After multiplication with the coefficient matrix, the resulting spectrum sequence is F ∈ R N ×C×H×W . Considering the discrete nature of DCT transform, that is, some of the most valuable and effective information is often distributed in different parts, we adopt a block strategy and obtain the final compact spatial feature. First, the resulting DCT spectrum F (i) is divided into K blocks, then the maximum value of each block is used as the expression of the block, and the compact feature map F CFE ∈ R N ×C×K is obtained as the final representation.

Frequency Temporal Attention Module
To further explore the temporal frequency clue among multiple frames, we design a Frequency Temporal Attention (FTA) module. Similar to CFE, the input of the FTA module is also the DCT spectrum F ∈ R N ×C×H×W . The goal of this module is to construct an attention map A ∈ R N ×H×W based on the DCT frequency spectrum. First, we conduct the L2 normalization in the channel dimension.
where C is the number of channels. A (n, h, w) represents the attention score of the n th frame spectral feature map at position (h, w). Then in order to control all the values in the attention matrix to be between 0-1, the results are normalized as: Following the same principle in CFE, we also divide the feature map matrix into K blocks, and for each region, we use the sum of them as the attention score. The results are : A ′′ (n) = {A ′′ (n, 1) , A ′′ (n, 2) . . . A ′′ (n, K)} Finally, L1 norm based on A ′′ ∈ R N ×K is adopted to further explore the temporal clue between frame sequences. The 2D attention matrix is expressed as: In the end, the spatial frequency forgery clue F CF E ∈ R N ×C×K and the temporal frequency attention map A F T A ∈ R N ×K are merged together by applying the weights obtained with FTA module to the spatial feature in the CFE module. The final spatial-temporal frequency forgery feature f c ∈ R C is constructed as:

EXPERIMENTS AND RESULTS
In this section, we first introduce the benchmarks and the parameter settings in the experiments. Furthermore, we conduct sufficient experiments to demonstrate the effectiveness of our method FCAN-DCT. The results on the selfbuilt near-infrared modality dataset DeepfakeNIR for face forgery detection also demonstrate that FCAN-DCT has good generalization across NIR and VIS modalities.

Experimental Settings
Datasets: In this paper, we use four video-based datasets for forgery detection: WildDeepfake ( modality, we also experimented with our method in the near-infrared (NIR) modality. DeepfakeNIR contains 3,816 videos in total, where the ratio of real and fake videos is 1: 1, where the real videos are collected from CASIA NIR-VIS 2.0 (Li et al. 2013) and the fake videos are generated using a well-designed deepfacelab tool (Perov et al. 2020). Furthermore, we add some perturbations such as JPEG compression, Change in color contrast, Gaussian blur, and Gaussian white noise, etc. on the NIR videos to mimic the real-world scenarios. The new dataset are called DeepfakeNIR++. The example video frames in our DeepfakeNIR dataset are shown in Fig. 4. Preprocessing: We select 16 frames from each video, then for each frame, we extract facial key points and crop out the head region of the person through the Facenet (Schroff, Kalenichenko, and Philbin 2015). Finally, we normalize all the faces with ImageNet mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225], and resize them to a fixed size which is 256 × 256 for ResNet50 (He et al. 2016a) and 299 × 299 for Xception (Chollet 2017).
Training: In our experiments, we use ResNet50 (He et al. 2016a) or Xception (Chollet 2017) as the final backbone of our approach. For retaining more information on the video sequence, we just remove the last average pooling and fully connected layer. We optimize the networks by Adam optimizer with the beta1 = 0.9 and beta2 = 0.999 . The start learning rate lr = 0.0001 and it drops by 10 every time the accuracy does not increase after 5 consecutive epochs. The evaluation metrics in our experiments are accuracy and AUC.
Selection of division number K and the number of frames N in each video: We conduct experiments on a total of four combinations of K = 2 × 2, 4 × 4 and frames=8, 16. The results are shown in Table 1. It can be seen that when the number of blocks is K = 4 × 4 and the number of frames selected is 16, the accuracy of the model reaches the highest of 86.35%, and the AUC is also the second highest at 93.87%. However, this does not mean that more blocks or more frame selections are effective for model training. When K is fixed to 2 × 2, as the frames increase from 8 to 16 frames, the accuracy of the model dropped from 85.11% to 84.74%, and the AUC also dropped from 93.87% to 92.40%. Therefore, in the selection process of K and frames, these two parameters cannot be considered separately. When we fix N to 8, there is only a slight increase in accuracy but when N is 16, increasing K from 2 × 2 to 4 × 4, the accuracy increases by more than 1.5% from 84.74% to 86.35%. So we choose K = 4 × 4, N = 16 as the final parameter selection result.

Experiments of Video Forgery Detection in VIS Scenario
In this section, we conduct experiments to compare our method with recent methods on WildDeepfake (Zi et al. 2020) and Celeb-DF (v2) (Li et al. 2020c) datasets. The experimental results are shown in Table 2 and Table 8. Our method uses ResNet50 (He et al. 2016a) or Xception (Chollet 2017) as the backbone network which is commonly applied in forgery detection tasks, respectively. It can be seen that our method achieves state-of-the-art performance on both WildDeepfake and Celeb-DF (v2) datasets. The superior performance demonstrates the feasibility and effectiveness of our proposed method. Evaluation on WildDeepfake: We choose ResNet50 and Xception as the backbone respectively and conduct extensive experiments on the WildDeepfake dataset. We compare FCAN-DCT with several recent state-of-the-art methods for face forgery detection including MesoNet  The experimental results are shown in Table 2. It can be seen that our method achieves the highest accuracy of 86.35% and AUC of 93.74%. Compared to the framelevel method, ResNetV2-50 (He et al. 2016b) only achieves 63.99% accuracy, and our method improves the accuracy by 22.36%. Our method also increases the accuracy and AUC by 6.24% and 4.71% respectively on the basis of Xception (Chollet 2017). The superiority demonstrates the spatial-  temporal feature extracted by FCAN-DCT is efficient and effective. For the selection of the backbone network, when we select the more complex Xception network, the accuracy and AUC are slightly decreased by 0.12% and 0.17%. Meanwhile, several frequency-based approaches with competitive results should be noticed. F 3 -Net (Qian et al. 2020) found that fake images can show not only subtle forgery traces but compression errors in the frequency domain. PEL (Gu et al. 2022) extracted fine-grained information hidden in the frequency domain as auxiliary features, which were finally fused with the features extracted in the RGB space. The two aforementioned methods demonstrate the effectiveness of expressing features in the frequency domain. Compared with F 3 -Net (Qian et al. 2020) and PEL (Gu et al. 2022), the accuracy of our proposed method increased by 5.69% and 2.21% and AUC increased by 6.21% and 2.12% respectively, The remarkable progress demonstrates the superiority of FCAN-DCT. Evaluation on Celeb-DF (v2): Considering video-based face forgery detection tends to realize the inconsistency between frames, it seems more reasonable to use 3D-CNN directly considering the temporal relationship, we test some of the most popular networks utilizing temporal features including RCN (Güera and Delp 2018), R2Plus1D (Tran et al. 2018), I3D (Carreira and Zisserman 2017), MC3 (Tran et al. 2018) and R3D (Hara, Kataoka, and Satoh 2017) on Celeb-DF (v2) (Li et al. 2020c).
Experimental results are shown in Table 8. It can be seen that FCAN-DCT outperforms all existing temporalbased networks to achieve competitive accuracy and AUC with state-of-the-arts. Especially when ResNet50 is selected as the backbone network, the accuracy and AUC reached 99.8% and 99.99% respectively which is a near-ideal result. R3D (Hara, Kataoka, and Satoh 2017) is a 3D CNNs based on ResNet toward a better action representation. R2Plus1D (Tran et al. 2018) decomposes 3D convolutional filters into separate spatial and temporal components and improves the accuracy of action detection. However, few of these methods are specifically designed for video forgery detection and too many parameters of 3D-CNN models make it difficult to apply to lightweight scenarios. Our method not only notices the most important part in different frames but also explores temporal information in the frequency domain and finally presents a comprehensive spatial-temporal representation. In addition, apart from the backbone network, no additional parameters are added. Therefore, our method is more capable and effective in capturing spatial-temporal forgery inconsistencies than 3D-CNN methods.

Cross-dataset experiments
In this subsection, we conduct a series of cross-dataset experiments to verify the generalization ability of our proposed method. First, we train a classifier based on the highquality version of FF++ and WildDeepfake respectively, and evaluate on the Celeb-DF(v2). We compare FCAN-DCT with several recently state-of-the-art methods, including MesoNet (Afchar et al. 2018), Xception (Chollet 2017), Capsule (Nguyen, Yamagishi, and Echizen 2019), and DAM (Lin et al. 2021).
The comparative results are shown in Table 5 and Table  6 respectively. When trained on FF++ and tested on Celeb-DF (v2), our method achieves AUC metrics of 79.95% and 83.46% on ResNet50 and Xception backbone, respectively, outperforming other methods by at least 10%. In addition, When trained on WildDeepfake, our method also achieves the highest accuracy of 85.74% on Xception backbone. Our model leverages both spatial and temporal information in frequency domain and is more robust to the modality differences between VIS and NIR because it is in the frequency domain rather than the spatial domain, therefore generalizing better from one method to another which confirms the effectiveness and robustness of our proposed FCAN-DCT.

Experiment of Video Forgery Detection in NIR Scenario
In this subsection, we conducted the experiments with state-of-the-art (SOTA) Deepfake detection methods such as CNNDetection , GANFingerprint (Yu, Davis, and Fritz 2019) and our proposed FCAN-DCT on our newly constructed DeepfakeNIR and Deepfak-eNIR++ dataset, to evaluate the effectiveness and robustness of the proposed method. The experimental results are shown in Table 4. It can be seen that with the help of our method, the model trained only using sources based on the

Ablation Study
In this section, we conduct sufficient experiments to demonstrate the effectiveness of our method and analyze the impact of hyperparameter selection on the results. In addition, we perform ablation study on the proposed FCAN-DCT. We train and test CFE and FCAN-DCT separately, and the experimental result of each model is shown in Table 7. It should be noticed that the temporal clue extracted in FTA is a weight attention map for the CFE module which is designed for extracting the spatial frequency feature. The final spatialtemporal representation can only be obtained with the joint cooperation of both two branches. Therefore, we just operate the CFE module in this section while it is not available to experiment only with the attention weights obtained by the FTA branch.
Effectiveness of the cofficient weight matrix: Previous works (Frank et al. 2020;Wang et al. 2020b;Dzanic, Shah, and Witherden 2020;Chandrasegaran, Tran, and Cheung 2021;Li et al. 2021b)  on forgery detection in the frequency domain for a single image believe that there exists a significant difference between the real and fake images in the midium and high frequencies, and this difference is more obvious as the frequency increases. Inspired by these works, we construct a coefficient weight matrix whose height and width are the same as those of the DCT spectrum. While keeping the low-frequency components fixed, we increase the weights of the midium and high-frequency components so that the model pays more attention to the area for fake clues. The experimental results are shown in Table 7. When there is no cofficient weight matrix, the accuracy of merely applying the original DCT spectrum reaches 85.37% while adding the cofficient weight matrix in the frequency transform will increase the accuracy and AUC by 0.95% and 0.27%, respectively. The progress demonstrates the cofficient weight matrix is efficient and effective. Effectiveness of the temporal attention map: By comparing with and without the temporal attention map settings in FCAN-DCT, it is clear that the results of FCAN-DCT outperform the CFE branch with only the spatial compact feature. For example, when there is no temporal frequency attention map, the accuracy of merely applying the spatial frequency forgery clue reaches 85.13%. Furthermore, adding the temporal attention map to the spatial clue will increase the accuracy and AUC by 1.22% and 1.29%, respectively, which denotes that the temporal attention map learned by FTA is effective in video forgery detection.
Effectiveness of the maximum block strategy: In the CFE module, we use the maximum strategy to compress the obtained spectrm sequence. Due to the discreteness of DCT, the effective information embedded in DCT frequency spectrum is sparse. To make the frequency feature expression more compact, we first divide the whole spectrum F into K frequency band, and then use the max response to represent each corresponding frequency band. We conduct the strategy comparison experiment by taking the maximum, minimum, and average value within the block on the WildDeepfake datase. Our chosen framework backbone is ResNet50. The experimental results proved that the maximum strategy outperforms minimum as well as average strategy in both effectiveness and efficiency.

CONCLUSIONS
In this paper, we present a novel forgery detection method based on spatial-temporal frequency clue. We then performed comprehensive experiments on two widely used VIS video forgery datasets and our newly constructed NIR video forgery dataset DeepfakeNIR. The experimental re-sults show that our method outperforms existing methods on both accuracy and AUC, and has excellent robustness and generalization ability in heterogeneous video forgery detection scenarios. Furthermore, the proposed novel dataset DeepfakeNIR based on the NIR modality will further facilitate future research on forgery detection in the near-infrared modality. In the future, we intend to explore the generalizability of video forgery detection in the presence of adding more real-world perturbations as well as adversarial perturbations.