{'fraction_non_alphanumeric': 0.041963838235861056, 'fraction_numerical': 0.009560892864027141, 'mean_word_length': 4.797184042909822, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Shrinking the Semantic Gap: Spatial Pooling of Local Moment Invariants for Copy-Move Forgery Detection

Copy-move forgery is a manipulation of copying and pasting specific patches from and to an image, with potentially illegal or unethical uses. Recent advances in the forensic methods for copy-move forgery have shown increasing success in detection accuracy and robustness. However, for images with high self-similarity or strong signal corruption, the existing algorithms often exhibit inefficient processes and unreliable results. This is mainly due to the inherent semantic gap between low-level visual representation and high-level semantic concept. In this paper, we present a very first study of trying to mitigate the semantic gap problem in copy-move forgery detection, with spatial pooling of local moment invariants for midlevel image representation. Our detection method expands the traditional works on two aspects: 1) we introduce the bag-of-visual-words model into this field for the first time, may meaning a new perspective of forensic study; 2) we propose a word-to-phrase feature description and matching pipeline, covering the spatial structure and visual saliency information of digital images. Extensive experimental results show the superior performance of our framework over state-of-the-art algorithms in overcoming the related problems caused by the semantic gap.


I. INTRODUCTION
I N the last few years, multimedia forensics appears in full development with high demands from industry and society. This is mainly because fake multimedia has become an urgent problem in the real world, and they are often used to manipulate public opinion, commit fraud, discredit or blackmail people.
As a common and basic image manipulation methods, copy-move forgery [1] consists of copying and pasting specific patches from and to an image, in order to overemphasize or cover the objects of interest. Although copymove forgery can be easily performed by any individual with modern image editing software (e.g., Photoshop), detecting such manipulations is not a trivial task. The fundamental difficulty lies in finding the correspondences among copy-move regions, which exhibit near-identical semantic properties but differ at the digital level due to the trace removal operations such as geometric transformations and signal corruptions.
image patches, 2) finding the best counterpart of each patch by matching the corresponding features, and 3) converting such matches to the final detection result by post-processing.
As a critical phase, the feature extraction for copy-move detection forgery have been pursued by many researchers in two different branches: dense-field and sparse-field [3]. Dense-field approach extracts features from overlapping and fixed-size image blocks. For achieving higher robustness, some well-known transformations like moment invariants have been used to construct the block features. Despite their success in ideal scenarios, this line of approaches intrinsically lacks scaling invariance and efficient calculation [4]. On the contrary, sparse-field approach shows superiority in solving the above limitations, since the feature extraction works only on a relatively small set of patches (i.e., keypoints) with certain geometric invariance. In this branch, the main problem is lower accurate than the dense-field methods due to the sparse nature, especially when the forgery only involves small or smooth regions [4]. It is an obvious fact that above two branches are difficult to achieve satisfactory detection accuracy, efficiency and robustness at the same time.
Through various differences, we observe the following common phenomena in both state-of-the-art dense-field and sparse-field detection methods: 1) for the image with high self-similarity, they tend to produce large number of false matches due to the poor discriminability of features; 2) for the image with strong signal corruption, they are difficult to obtain a sufficient number of correct matches due to the poor robustness of features. Note that such limitations in feature extraction will in turn lead to inefficient matching/post-processing processes and unreliable detection results.
The fundamental cause for such phenomena lies in the well-known semantic gap. Specifically, copy-move forgery detection mainly focuses on checking the semantic integrity, so it is critical to effectively model local semantic properties from digital images. However, most copymove forgery detection methods rely heavily on the lowlevel visual representation of digital images, e.g., Scale Invariant Feature Transform (SIFT) [5], and there is a big gap between low-level features and high-level semantic concepts. With this in mind, one main explanation for the semantic gap is that detection algorithms ignore the spatial structure relationships and visual saliency differences of the extracted features. In other words, such independent and indifferent features are not sufficient to model the complex semantic properties.
Motivated by the above facts, this paper attempts to take a first step towards alleviating the semantic gap problem in arXiv:2207.09135v1 [cs.CV] 19 Jul 2022 copy-move forgery detection. Our work has the following two key contributions. • We introduce the Bag-of-Visual-Words (BoVW) model [6] into the field of copy-move forgery detection, meaning a new theoretical basis for feature extraction. In overcoming the semantic gap, the BoVW model may be the most competitive representation model in computer vision and pattern recognition. It is a hand-crafted algorithm in which low-level local features are extracted, encoded, and summarized into high-level image representation, with great success in pre-deep-learning era. • We propose a word-to-phrase feature description and matching pipeline inspired by BoVW model, covering the spatial structure and visual saliency information of digital images. In this hierarchical strategy, local features are first roughly described and matched at the visual-word level, followed by a fine description and matching of these matched features at the visual-phrase level. The core lies in a spatial pooling (spatial structure) and weighting (visual saliency) of low-level features for the midlevel representation, which is expected to alleviate the semantic gap. The rest of this paper is organized as follows. A review of the recent dense-field and sparse-field works is given in Section II. In Section III, the details of this work is provided. In Section IV, various experiments are carried out for effectiveness and efficiency verification. Finally, we draw conclusion in Section V.

II. RELATED WORK
In this section, we will discuss the two existing branches for copy-move forgery detection, with special attention on successful experiences and common limitations behind such works.

A. Dense-Field Approach
In this approach, most of the research efforts have focused improving the robustness of the features and the efficiency of the matching.
For feature extraction, it is a natural requirement that features can be robust against conventional signal corruption and geometric distortion. Some early works adopt classical image transform with high robustness to JPEG compression, noise addition, and other common distortions. The popular choices in the literature include Discrete Cosine Transform (DCT) [7], Wavelet Transform (WT) [8], Fourier Transform (FT) [9], and Singular Value Decomposition (SVD) [10]. However, the features extracted by such classical methods are vulnerable to geometric transforms, mainly rotation and scaling. This observation prompts the introduction of more complex geometric invariants. In this regard, a common strategy is to provide the rotation invariance by using the orthogonal moments, e.g., Zernike Moments (ZM) [11], Fourier-Mellin Transform (FMT) [12], Polar Cosine Transform (PCT) [13], and Polar Complex Exponential Transform (PCET) [14]. As for the scale invariance, however, effective generation methods are still lacking in dense-field approach. In fact, many scaleinvariant transforms (e.g., FMT) require the interest region for feature extraction to be adaptive or infinite. But the size of the image block is generally fixed in dense-field detection algorithms.
For feature matching, the processing of a large number of blocks (about 10 5 ∼ 10 6 blocks per image) usually results in expensive calculations, especially the brute-force method. A variety fast implementations for the generic nearest-neighbor (NN) searching have been introduced into this field, such as lexicographic sorting [15], kd-trees [16], and Locality Sensitive Hashing (LSH) [11]. However, these methods either cause a significant reduction in robustness/accuracy, or still fail to bring a satisfactory speed. One convincing reason is that such generic algorithms do not fully consider the specific background knowledge (i.e., priori) of the copy-move forgery detection problem. Inspired by this, more efficient NN searching have been proposed, e.g., modified PatchMatch [17] and enhanced Coherency Sensitive Hashing (CSH) [13], by exploiting the smoothness and self-similarity of the image. Note that, despite their success, such speed-up algorithms are still difficult to be used in time-critical scenarios, especially when the high-resolution image or high-reliability detection is required.
In summary, high computational burden and poor scaling invariance are the two intractable problems of densefield detection algorithms. This also promotes the development of sparse-field approach.

B. Sparse-Field Approach
Sparse-field approach is an alternative path for copymove forgery detection with inherent advantages in terms of complexity and invariance. Therefore, in this approach, the design of feature extraction, matching, and postprocessing have all attracted extensive research interest.
For feature extraction, regions of interest are first detected (i.e., keypoint detection) and then described by specific representation (i.e., keypoint description). A hot issue in the detection is the lack of keypoints in smooth or small regions, which may also contain copy-move forgery. Existing solutions include threshold adjustment [18], resolution enhancement [19], and Non-Maximum Suppression (NMS) [20]. With such strategies, the stateof-the-art forensic methods are able to detect the smooth or small copy-move region more effectively. As for the description, similar to the case of dense-field approach, achieving satisfactory descriptor is a long-lasting battle. The most commonly used SIFT do not perform well in terms of robustness and efficiency [21], especially the invariance to geometric changes. Therefore, the variations of SIFT with more invariance or lower complexity, such as SURF [22], KAZE [23], OpponentSIFT [20], affine-SIFT [24], mirror-SIFT [25], and binarized SIFT [26], were successively introduced into recent works. Also, moment invariants are used to describe the keypoints considering their geometric invariance and compactness, such as Exponent-Fourier Moments (EFM) [18], PCT [27], PCET [28], Radial Harmonic Fourier Moments (RHFM) [29], and Bessel-Fourier Moments (BFM) [30]. As the other side of the coin, such moment invariants are less informative than SIFT-like texture features, implying lower discriminative power. Thus, for all the above features, it is difficult to achieve satisfactory robustness and discriminability at the same time. This phenomenon is in fact a sign of the semantic gap between low-level visual representation and high-level semantic concept.
For feature matching, candidate features are first searched out from the set (i.e., NN searching), and then the true matches are screened out through certain testing (i.e., NN testing). In general, the NN searching dominates the computational cost of the matching stage. Efficiency gains can be achieved by introducing kd-trees [31] or hashing [26], similar to the case of dense-field approach. Other researchers have further accelerated the above NN searching methods by features grouping [19], based on certain prior knowledge. As for NN testing, major research efforts focus on the multi-feature matching problem, i.e., multiple correct matches for a patch/feature due to multiple copymove forgeries. The 2NN testing [31] in classical SIFT matching cannot handle such case due to the different assumption, thus its extension versions have been designed for this purpose, such as Generalized 2NN (G2NN) [32] and Reversed G2NN (RG2NN) [18], with different accuracy, speed and robustness.
For post-processing, typically, the geometric models of copy-move pairs are first evaluated from the matched keypoint set with noise (i.e., model estimation), and then the forgery regions are located in the dense-field (i.e., forgery localization). When multiple copy-move forgeries occur, the most popular global geometric model estimation algorithm, RANdom SAmple Consensus (RANSAC) [31], will fail. As it can only evaluate a single model, while multiple ones actually exist in such case. Therefore, separating these models is a natural idea. The existing solutions include the image-space segmentation strategy [33] and the concept-space clustering strategy [34]. Note that clustering strategy can better deal with multiple forgeries due to the higher discriminability, where commonly used discriminative information includes coordinates [25], [32], [34], angles [34], affine parameters [20], [35], and offset vectors [30]. As for localization, copy-move regions can be determined by the matched keypoint regions [19], or by window-wise similarity checking [31], e.g., Zero-mean Normalized Cross-Correlation (ZNCC) [31] and Structural SIMilarity (SSIM) [30]. The former has a high precision and a low recall, while the latter is just the opposite.
In summary, almost all of keypoint features are based on very shallow representations, which cannot effectively describe the local semantic properties. This leaves a fundamental flaw in the whole forensic algorithm, forcing the use of complicated matching and post-processing to remedy it.

A. Overview
For breaking the performance bottleneck caused by semantic gap problem, we propose a BoVW model inspired copy-move forgery detection algorithm. The overall workflow is shown in Fig. 1.
Note that this method inherits some techniques from our previous work [30], but with obvious differences in the theoretical basis and overall framework. Specifically, the existing sparse-field copy-move forgery detection methods (also in [30]) generally rely on a common pipeline, i.e., feature extraction (including keypoint detection and description), matching, and post-processing.

Normalizing image
Detecting keypoints with low contrast threshold  In this paper, such a classic three-step pipeline is generalized to a new four-step pipeline: keypoint detection, word-level feature description and matching, phrase-level feature description and matching, and post-processing, based on the theory of BoVW. The design aims to describe the mid-level image information via hierarchical feature extraction, and hence potentially addressing the problems related to the semantic gap.
• Keypoint detection. We extract sufficiently dense and uniform SIFT keypoints based on two strategies: scale normalization and contrast threshold reduction. They are effectiveness for detecting small/smooth copy-move regions [30]. • Word-level feature description and matching. Such keypoints are described by complex-valued moment invariants, and then matched by magnitude-phase hierarchical matching strategy. The theoretical and experimental analyses have shown their advantages in terms of invariance and efficiency [30]. In addition, we also propose an improved 2NN matching strategy for solving the multi-feature matching problem while reducing the time complexity. This new method is expected to replace the commonly used 2NN, G2NN and RG2NN due to the better performance on matching accuracy and efficiency. • Phrase-level feature description and matching.
The word-level matching results will be further integrated and encoded for capturing relevant information. First, the visual phrases are formed from the geometric relationships of matched keypoints, and followed by a max-pooling to construct corresponding phrase-level feature vector. Then, the features are weighted by an edge-information based heatmap, with certain foreground-background differences. Finally, such phrase-level features, associated with both spatial structure and visual saliency, will be matched as the overall matching results. • Post-processing. The matched keypoint pairs are first filtered and clustered, and then converted into localization results on the image plane. For the filtering and clustering, two priors for correct matches are taken into account, i.e., geometric constraint and content consistency, with the ability to reduce false positives and manage multiple forgeries [30].
For the localization, we propose a new fusion-based algorithm covering the information from geometric transformations, local similarity, and keypoint interest region. Existing works generally do not consider such rich information, and the proposed method works well in most scenarios. The following Sections III-B to III-E will introduce the above components respectively. Considering the focus of this paper, the techniques inherited from our previous work [30], mainly in Stages 2 and 4, will not be described in detail.

B. Keypoint Detection
Similar to many existing copy-move forgery detection algorithms, the classical SIFT [31] is adopted for detecting geometric invariant keypoints with corresponding interest regions. However, the general setting of SIFT cannot guarantee that sufficient keypoints are extracted in small or smooth regions, which in turn leads to the failure in detecting copy-move forgery for such regions. To address this challenge, two tricks have been explored in recent work [19], and they will also be used in our algorithm.
The first trick enhances the keypoint detection performance in smooth regions by reducing the contrast threshold. In many classical computer vision applications, such as image registration, ensuring a high repeatability of keypoints is a primary goal, rather than the coverage. For this reason, there is a step in SIFT to eliminate the unstable keypoints, i.e., passing only the keypoints with sufficiently high local contrast. However, for copy-move detection, keypoint coverage of the forgery regions plays an even more important role than the repeatability. Therefore, we skip such a step and directly set the contrast threshold T con = 0 in the implementation.
The second trick enhances the keypoint detection performance in small regions by increasing the image resolution. When the copy-move regions contain few pixels, it is difficult to achieve high keypoint coverage on such regions, even though the contrast threshold T con is set to 0. The simplest solution is to increase the resolution of the whole image, and thus also increase the number of pixels in the potentially forgery region, allowing more candidate keypoints to be detected. However, it is clear that this strategy requires a trade-off between coverage and efficiency. In the implementation of Li et al. [19], the scale factor is set to a fixed value (s = 2), resulting in expensive calculations for high-resolution images. For this reason, in our previous work [30], is a variable value and is determined according to the resolution of the input image. Although coverage and efficiency are basically balanced, the image size may changes over a wide interval, and some fixed parameters cannot adapt to it. In this paper, we refine this strategy and the scaling factor s defined as: where N I × M I is the resolution for the input image I. The (1) normalizes the long edge of the low-resolution (i.e., max(N I , M I ) < 3000) image to 3000 pixels, without changing the aspect ratio. The design aims to extract sufficient keypoints at a reasonable time cost, while reducing the sensitivity of parameters/thresholds to the resolution for the following stages. The reason why a similar normalization process is not performed on highresolution (i.e., max(N I , M I ) ≥ 3000) images is that down-sampling will inevitably produce a certain degree of information loss, which may be more harmful to feature description and post-processing. Through the above strategies, one can obtain a set of SIFT keypoints, where (x, y) is the coordinate, σ is the scale and N oK is the number of keypoints. For better visualization, two examples of the keypoint detection are shown in Fig. 2. As can be seen that, after the optimization based on two strategies, the keypoints are more uniformly and densely distributed over the image plane, ensuring the coverage of the copy-move regions.

C. Word-Level Feature Description and Matching
In this section, the SIFT keypoints will be roughly described and matched at the visual-word level. For this task, we continue to use the complex-valued moment invariant description and magnitude-phase hierarchical matching in our previous work [30], due to their advantages in representation ability and matching speed. Here, we will propose a new NN testing strategy to replace the 2NN in the matching, considering its difficulty in handling multiple copy-move forgeries.
Through the description, one can obtain a set of features F = {f k } N oK k=1 , which corresponds to the keypoint set Based on a specific distance metric, the matching starts with a search for several closest features to a given feature, i.e., NN searching. Mathematically, for a feature f (0) (w.r.t. keypoint k (0) ), its first n nearest neighbors in the feature set F are denoted as {f (1) Based on a certain prior, the final judgment of whether f (0) and {f (1) , f (2) , ..., f (n) } are matched pairs is formed, i.e., NN testing. In this regard, the existing NN testing strategies are mainly based on distance information in {d 1 , d 2 , ..., d n }, including two types [36]: • Absolute distance based NN testing. It simply regards f (0) (w.r.t. k (0) ) and f (1) (w.r.t. k (1) ) as a matched pair when d 1 is less than a predefined threshold. In the complex tasks (e.g., image registration), however, the distance between a feature and its counterpart varies widely, due to diversity of imaging conditions (e.g., illumination and viewing angle). Fig. 3. Illustration of the different keypoint matching strategies. Therefore, such method is usually unsatisfactory in matching accuracy [36]. • Relative distance based NN testing. This path is generally based on the ratio of the distances, which exhibits a higher tolerance for imaging conditions, and thus serving as a common choice [36]. In the field of copy-move forgery detection, relative distance based NN testing methods mainly include 2NN [31], G2NN [32], and RG2NN [18]. 2NN. For given f (0) , this strategy only searches the first two nearest neighbors, i.e., {f (1) It should be noted that the 2NN is based on the following assumption: there is at most one true match (semantically) for a keypoint. This assumption is in line with some computer vision tasks, but not applicable to copy-move detection due to the existence of multiple forgeries. For such scenarios, d 1 /d 2 ≈ 1 since both f (1) and f (2) are true matches.
G2NN. It is a generalized version of 2NN and is especially designed for multiple copy-move forgeries detection. By the order i = 1, 2, ..., n − 1, it calculates d i /d i+1 separately, until a position k allowing d k /d k+1 > T g2N N (general setting T G2N N = 0.5); then the f (0) and {k (1) , ..., k (k) } are considered as matched pairs. However, in the same way, when multiple forgeries exists with d 1 /d 2 ≈ 1, such a loop will terminate prematurely.
RG2NN. It is a reversed version of G2NN for solving the above premature termination problem. By the order ; then the f (0) and {k (1) , ..., k (k−1) } are considered as matched pairs. Compared with G2NN, RG2NN is theoretically more suitable for multi-feature matching. However, since k is generally much less than n, such NN testing often requires a large computational cost. Fig. 3 gives an illustration of 2NN, G2NN and RG2NN. Here, for given keypoint k (0) , its first NN k m1 and second NN k m2 are true matches, the remaining n − 2 neighbors {k u1 , k u2 , ..., k u(n−2) } are irrelevant keypoints, and the feature distances between k (0) and {k m1 , k m2 , k u1 , k u2 , ..., k u(n−2) } are {d 1 , d 2 , ..., d n }, respectively. Since d 1 /d 2 ≈ 1, the 2NN and G2NN directly treat k m1 and k m2 as irrelevant. The RG2NN is able to match k m1 and k m2 , but a total of n−2 tests for a keypoint results in a higher complexity.
To handle multiple copy-move forgeries detection with less complexity, we propose an Improved 2NN (I2NN) algorithm. Specifically, for the given f (0) (w.r.t. k (0) ) and its first two nearest neighbors {f (1) , when the following single-feature matching conditions, are satisfied, the features f (0) and f (1) (w.r.t. keypoints k (0) and k (1) ) are considered as a matched pair; or when the following multi-feature matching conditions, are satisfied, the features f (0) and {f (1) , f (2) } (w.r.t. keypoints k (0) and {k (1) , k (2) }) are all considered as matched pairs, where T sml−rat and T sml−abs are the relative and absolute thresholds for measuring the distances between the features (T sml−rat = 0.6 and T sml−abs = 0.1 in our implementation), and T dist is a threshold for measuring the distances between the positions (T dist = 50 in our implementation). Note that , since d 2 is only small when multiple true matches exist. Fig. 3 shows the comparison of I2NN and existing strategies. The proposed I2NN considers both absolute and relative distances, and thus achieves a limited ability for multi-feature matching, which draws a distinction between 2NN and G2NN. Moreover, the I2NN checks only the first two nearest neighbors, with less time complexity than RG2NN. Note that, as far as copy-move detection task is concerned, we actually do not need the full multi-feature matching ability like RG2NN. Since the number of the copy-move regions is rarely large, the ability of I2NN to match two nearest neighbors is generally sufficient in practice. As an illustration, the matching results of 2NN, G2NN, RG2NN, and I2NN for real image are given in Fig. 4. It can be seen that the proposed I2NN is able to deal with multiple copy-move forgeries, with much fewer mismatches and complexity.

D. Phrase-Level Feature Description and Matching
In this section, the above matching results will be further described and matched at the visual-phrase level. Existing methods generally feed the word-level results directly to post-processing, where the semantic gap inevitably leads to unsatisfactory forgery localization results. Therefore, we recommend performing a hierarchical feature extraction and matching process to capture more semantic properties. Specifically, the proposed phrase-level feature description and matching algorithm includes the following steps: • Step 1: Geometric phrase pooling based on spatial structure; • Step 2: Spatial weighting based on visual saliency; • Step 3: Direct matching of phrase-level features. 1) Geometric Phrase Pooling based on Spatial Structure: This step uses the geometric relationships of keypoints to form visual phrases, and then performs maxpooling to obtain the representation reflecting spatial structure. The main purpose is avoiding the isolation of the local descriptors in existing copy-move detection works, where the features are processed very independently, ignoring their relationships in the image plane. We design a mid-level connection between low-level descriptors and high-level concepts for the copy-move detection scenarios, inspired by the work of Xie et al. [37].
Going back to Sections III-C, with keypoint set K = {k k } N oK k=1 , one can obtain a set of word-level features F w = {f w k } N oK k=1 and a set of word-level matched pairs , where N oP −w is the number of pairs at the word level; the corresponding set of word-level matched keypoints denotes as ∈ K. Based on the above notations, a set of visual words can be defined: is the combination of keypoint k k and corresponding word-level feature f w k . For a visual word (k c , f w c ) in D, its visual phrase is defined as: where word (k c , f w c ) is named as central word of phrase G, resting {(k s1 , f w s1 ), (k s2 , f w s2 ), ..., (k sK , f w sK )} are the K visual words closest to the center word (k c , f w c ) in the image plane, called side words of phrase G (K = 3 in our implementation). To speed up the search for side words, we build a KD-tree using the keypoint location information from word set D, based on the VLFEAT package [38].
Suppose the dimension of word-level feature f w k is d, written as f w k = {f w k,1 , f w k,2 , ..., f w k,d }. Based on the geometric phrase G, the corresponding geometric phrase feature f g c for the keypoint k c can be obtained through geometric phrase pooling: The intuitive interpretation of (6) is to first find the maximum value among the side words {f w s1 , f w s2 , ..., f w sK } along each dimension of the feature, and then add the result to the corresponding dimension of the center word feature f w c , finally forming a phrase-level feature f g c . For more details on geometric phrase pooling, please refer to [37].
The generated phrase-level features are with following properties: 1) f g c maintains the same dimensions as f w c and thus does not increase the matching complexity; 2) f g c combines the complete information of the central word f w c with the partial information of the adjacent words {f w s1 , f w s2 , ..., f w sK }, exhibiting better discriminability than word-level representation; and 3) the max-pooling allows an order-independent information extraction for {f w s1 , f w s2 , ..., f w sK }, meaning the invariance w.r.t. geometric changes, especially rotation. Fig. 5 shows an intuition of the geometric phrase pooling operation in copy-move detection. Here, the central words of geometric phrases G 1 , G 2 , and G 3 have similar features, so they are regarded as matched pairs in the word-level process. Through geometric phrase pooling, such isolated features are upgraded to phrase-level features with spatial structure, and thus become more informative. For the case in Fig. 5, the correct matching between G 1 and G 2 is preserved in phrase-level process, due to the similarity in both center words and side words (even with different orders). As for G 1 and G 3 , this mismatched pair is passed in the word-level process, but is eliminated in the phrase-level process by the inconsistency of side words.
2) Spatial Weighting based on Visual Saliency: This step uses the edge information of image to form a heat map, and then weights the phrase-level features to strengthen the difference in visual saliency. The main purpose is avoiding the equality of the local descriptors in existing copy-move detection works, where the features are processed very equivalently, ignoring their visually differences in the image plane. We design a visual saliency based spatial weighting for the copy-move detection scenarios, with a corresponding acceleration method, inspired by the work of Xie et al. [37].
According to the relevant research on visual perception, the human visual system is more sensitive to the edge of object, which is the basis for the semantic interpretation of the shapes. Therefore, it is reasonable to generate a saliency map through edge information.  where w(i, j) is the visual saliency of image I at (i, j); G ij (i , j ) = exp(−T σ ·||(i, j)−(i , j )|| 2 ) is the Gaussian kernel function centered at (i, j); T σ is the smoothing threshold (T σ = 0.001 in our implementation). Here, the smaller the T σ , the larger the influence range of a specific response in the heat map. In practice, the (8) needs to enumerate all (i , j ) (a total of N I × M I ) for each position (i, j), resulting in a very high complexity O(N I 2 M I 2 ). For this, an approximate formula is given by ignoring the position (i , j ) with lower G ij (i , j ). Specifically, based on the three-sigma rule, the enumeration range is restricted to a square region S with center (i, j) and length L(T σ ) = 3 √ 2T σ : Then, the approximate version of (8) can be derived: where the complexity is reduced to O(4N I M I L 2 ). This is acceptable for low-resolution scenarios like in [37], but not for copy-move detection, due to the large N I ×M I and L(T σ ). An equivalent but efficient implementation of (10) can be achieved by the help of Fast Fourier Transform (FFT) and Convolution Theorem: Based on the geometric phrase feature f g c and the saliency heat map I w , the weighted geometric phrase feature f p c corresponding to the keypoint k c can be defined as: Fig. 6 shows the examples of edge map and saliency heat map. Here, the extracted edges are semantically relevant, and the resulting heat map well reflects the visual saliency differences in different regions. It is worth noting that the corresponding positions in the copy-move regions have similar weights, while the weights in other regions are more diverse. This fact clearly facilitates the further enhancement of phrase-level features in terms of discriminability.
3) Direct Matching of Phrase-Level Features: In this step, the obtained phrase-level features reflecting the spatial structure relationships and visual saliency differences will be directly matched as the final matching result. For the word-level matched keypoint set , it is reasonable to directly match such features without using the magnitude-phase hierarchical matching, due to the less complexity by 2N oP −w N oK. In terms of NN testing, the proposed I2NN strategy will also be used in the phrase-level matching. An illustration of phrase-level feature matching is given in Fig. 7. On the left, a matched pair of geometric visual phrases is selected as an example, where yellow indicates the central words and magenta indicates the side words. On the right, the phrase-level feature matching results are shown. Compared with the word-level results in Fig. 4, the number of incorrect pairs is greatly reduced, while the correct ones in the copy-move regions are well preserved. This should be attributed to the enhanced description ability of the phrase-level features.

E. Post-processing
In this section, the sparse matches will be transformed into the dense localization results through post-processing. Specifically, the proposed post-processing algorithm includes the following steps: Here, the detailed description of Steps 1 to 4 can be found in our previous work [30]. As a new concept, the combination of Steps 4 to 6 is able to provide better localization recall and precision, which is not available in previous methods. Next, the research motivation and technical details of such new components are presented.
Currently, typical localization strategies in the postprocessing stage are mainly divided into the following two categories: • Localization strategy based on homography and local similarity [30], [31]. It first evaluates the geometric transformation relationship of each pair of copy-move regions. Then, the same transformation performed forward and backward on the entire image, allowing the copy-move regions to overlap. Finally, by a pixel-by-pixel correlation check, the regions with stronger correlation are highlighted as the localization results. Ideally, this strategy is able to offer accurate forgery regions with clear boundary. However, due to the smoothness and self-similarity of natural images, high correlation response often appears in the similarbut-genuine regions, leading to false positives. • Localization based on matched keypoint regions [19]. It generally takes the union of matched keypoint regions as the basic localization results, and then through certain enhancement technology like morphological processing to improve the accuracy. Obviously, this method has a lower cost because it does not require the dense operations. However, when noise-like attacks exist, the number of matched keypoints may be significantly reduced, leading to false negatives.
Taken together, the above analysis shows that both two methods are difficult to achieve satisfactory localization accuracy. This is especially true especially when input image is with high self-similarity or strong signal corruption. For this reason, we proposes a fusion-based localization algorithm using the information from homography, local similarity, and keypoint region, with ability to reduce both false positives and false negatives.
To clarify, we denote the set of phrase-level matched pairs as , the corresponding set of phrase-level matched keypoints as ∈ K m−w , and the set K m−p after two-stage mismatch filtering as where the Gaussian kernel G ij (i , j ) and the enumeration region S are defined in Section III-D2, and the similar fast calculation is also used; the morphological dilation D is based on a circular element with radius of T D (T D = 50 in our implementation). Subsequently, the response in the ROI heat map is normalized to  where T nor is the normalization threshold (T nor = 25000 in our implementation). The SSIM correlation map ρ k under cluster C k is weighted by the normalized ROI heat map I ROI to merge the information of homography, local similarity, and keypoint region: Next, based on the correlation threshold T cor (T cor = 0.4 in our implementation), ρ k is binarized into the localization resultρ k . After performing the homography estimation and forgery localization for all clusters {C k } N oC k=1 , the union of such localization maps, R = N oC k=1ρ k , is taken as the final copy-move detection result.
An example of our localization strategy is given in Fig. 8. As can be seen, the normalized ROI heat map is able to draw the potential regions, but such a result is rough. The SSIM map gives a more accurate boundary of the potential regions, but the response values is quite high for some similar-but-genuine regions, tending to raise false positives. For above facts, the proposed strategy merges such two types of results with complementary information, and hence exhibits lower false positives and false negatives.

IV. PERFORMANCE EVALUATION
In this section, we will conduct a number of benchmark and robustness experiments for evaluating the performance of our work w.r.t. the state of the art. In addition, we also present the performance analysis for key components and under challenging scenarios.
The F1 score is introduced as a main quantitative indicator of detection accuracy. The Precision and Recall scores are also considered as additional accuracy indicators, reflecting false positives and false negatives, respectively.

B. Benchmark Experiments
In this round of experiments, we provide the performance statistics on five benchmarks for positioning our algorithm w.r.t. the current state-of-the-art approaches.
In Fig. 9, some detection samples on benchmark FAU, GRIP, MICC, CMH, and UNIPA are illustrated. Here,   It can be seen that the number of false matches in (c1)-(c5) is significantly reduced compared to (b1)-(b5), while the correct matches in the copy-move regions are well preserved. This phenomenon indicates that the proposed phrase-level features can better describe the image local behavior, compared to the word-level features used in traditional copy-move detection methods. In addition, the normalized ROI heat maps in (d1)-(d5) are roughly consistent with the ground-truth forgery regions; the fusion localization strategy is able to give more accurate results (e1)-(e5), i.e., with high precision and high recall at the same time.
In Table I, we list the Precision, Recall, and F1 scores of various competing algorithms on benchmark FAU, GRIP, MICC, CMH, and UNIPA. The three most representative copy-move forgery detection algorithms are selected as the reference techniques: PatchMatch [17], Iteration [27], and HFPM [19]. The quantitative results in Table I are derived from the code they released. Here, Iteration and HFPM are the state-of-the-art algorithms in sparse-field approach; PatchMatch is the state-of-the-art algorithm in dense-field approach.
As we mentioned in Section II, the main advantage of the dense-field approach lies in the localization accuracy (especially for rigid translation), while the sparse-field approach provides higher robustness (especially for rotation and scaling). The results in Table I generally support above conclusion, i.e., the PatchMatch exhibits relatively better accuracy than Iteration and HFPM in plain copymove scenarios. Although the proposed method works in sparse domain, it still provides comparable performance with PatchMatch in the Precision, Recall and F1 scores. Meanwhile, our work is generally more accurate compared to the sparse Iteration and HFPM, especially on CMH and UNIPA. Over a variety of benchmarks, the average scores indicate that the proposed method offers the best overall localization accuracy at the levels of Precision, Recall, and F1. The main reasons are as follows. The proposed algorithm is able to deal with small/smooth forgery regions (in GRIP and UNIPA), multiple copy-move forgeries (in FAU and MICC), and rotation/scaling attacks (in MICC and CMH), while better overcoming the semantic gap problem such as high false positives in similar-but-genuine regions.

C. Robustness Experiments
In this round of experiments, we provide the performance statistics in the scenarios with geometric transformations or signal corruptions. Specifically, the copy-move images in the benchmark FAU are further attacked by the following four operations: scaling, rotation, additive Gaussian white noise, and JPEG compression. Their parameter settings are given by Table II. In Fig. 10, we show the Precision, Recall, and F1 curves of various competing algorithms w.r.t different attack parameters. This part covers the following popular competing algorithms: FROGS [41], Segment [42], ECSS [13], Iteration [27], and HFPM [19]. The quantitative results of Iteration and HFPM are derived from the code they released, and the other results are cited directly from the reference [4]. Here, FROGS and ECSS are dense-field algorithms developed from the representative PatchMatch [17]; Iteration and HFPM are the state-ofthe-art algorithms in sparse-field approach; Segment tries to combine the advantages of dense-field and sparse-field through superpixel segmentation. As we mentioned in Section II, the sparse-field approach generally exhibit better robustness accuracy than densefield approach, especially for the scenarios with large geometric variations. The results in Fig. 10 generally support above conclusion. While the dense FROGS and ECSS are more accurate in most cases with light attacks, the sparse Iteration and HFPM show increasing performance advantages as the attack level increases, mainly for scaling, rotation, and noise. In contrast, the proposed algorithm achieves comparable performance with the dense-field approach for the scenario with light attacks. Also, for highlevel attacks, the proposed algorithm exhibits similar or better accuracy than the sparse-field approach. The above facts illustrate that our algorithm can offer better overall robustness accuracy. A noteworthy accuracy decrease is observed at the 60 and 180 degree rotations. The possible reason for this lies in the imperfection of the adaptive clustering. In the future, it is expected to solve this problem by designing more discriminative representation of matched pairs for clustering. In the compression and scaling scenarios, the proposed algorithm exhibits a significant advantage over the existing state-of-the-art works. This should be mainly attributed to the proposed feature and post-processing strategies, both of which have satisfactory robustness. It is worth noting that scaling causes changes in the density of keypoints and rotation causes changes in the order of keypoints. The proposed phraselevel feature descriptions (i.e., geometric phrase pooling based on spatial structure and spatial weighting based on visual saliency) are adaptive to both changes.

D. Performance of Key Components
In this round of experiments, we provide the performance statistics of the key components in our algorithm, verifying their effectiveness directly.
In Table III, we list the CPU time, number of correct matches, and number of all matches for different NN testing strategies discussed in Section III-C. The testing image is from Fig. 4 with multiple copy-move forgeries. Its size is 3264 × 2448, and a total of 43888 keypoints are detected. We use the direct NN searching to better reflect the accuracy and efficiency of NN testing strategy itself.
Note that the number of correct matches derived by the 2NN strategy is significantly less than the others, for its failure to deal with the multi-feature matching problem. In contrast, the G2NN, RG2NN, and I2NN yielded similar numbers of correct matches, implying that they can fulfill the multi-feature matching to varying degrees. However, the G2NN and especially the RG2NN strategies produce a considerable number of false matches (see also Fig.  4), resulting in a small percentage of correct matches in the set of matched pairs. Such fact obviously increases the difficulty of the post-processing algorithm, potentially leading to false positives in the localization result. On the contrary, the proposed I2NN exhibits the highest percentage of correct matches w.r.t all matches, also the number of correct matches remains at the same level as G2NN and RG2NN. Consequently, our I2NN is more capable in both overcoming false matches and dealing with multi-feature matching.
In Table IV, we list the CPU time and number of all matches for the different feature description/matching levels (i.e., word and phrase levels) discussed in Sections III-C and III-D. The testing images are from Fig. 9, where the average statistics are used.
It can be seen that the proposed phrase-level feature description/matching strategy can significantly exclude the false matches in the word-level, and the time cost of the phrase-level strategy is only one-third of the word-level strategy. Certainly, it must be acknowledged that the above process may remove some correct matches of the wordlevel. However, as shown in Fig. 9, the deleted matches are mainly false matches, and the loss of a small number of correct matches is acceptable.  Fig. 11. Several challenging samples of the copy-move forgery detection.

E. Performance for Challenging Cases
In this round of experiments, we provide case studies in challenging scenarios for comparing our algorithms with state-of-the-art methods.
For (a1)-(g1), the small/smooth copy-move regions are involved. In such regions, the number of the pixels is 0.78% w.r.t. the entire image, and the variance of the pixel values is about 0.88. It can be seen that Segment and PatchMatch fail completely. For the former, the main reason lies in the fact that not enough keypoints are detected in such smooth regions. As for the latter, the random nature of PatchMatch algorithm makes it difficult to handle small regions. The Iteration and HFPM successfully detect the copy-move regions, but have a high false-negative rate, as the imperfection of their post-processing algorithms. The proposed method, with well-designed keypoint detection (Section III-B) and post-processing (Section III-E) algorithms, thus exhibits better accuracy for this challenging case.
For (a2)-(g2), the multiple copy-move forgeries are involved, where one region is selected and pasted three times. It can be seen that Segment and PatchMatch are also fail here, possibly for the similar reasons as before. The Iteration and HFPM are able to detect all the four regions, but both exhibit high false-positive rate and high falsenegative rate, respectively. For the Iteration, its absolute distance based NN testing (Section III-C) allows a well multi-feature matching, also leading to low discriminability for similar-but-genuine regions. For the HFPM, its 2NN strategy (Section III-C) should be primarily responsible for the false negatives, as we shown in Section IV-D. In contrast, the proposed method works better in this challenging case, further confirming the effectiveness of our I2NN (Section III-C).
For (a3)-(g3), the large scaling is involved, where the scaling factor is 200%. It can be seen that Segment, PatchMatch, and Iteration all fail, due to the fact that the descriptors they construct are not scaling invariant. The HFPM and our method successfully detect such copyscale-move regions. The HFPM exhibits a high falsepositive rate, arising from the defects of its matched-region based localization algorithm. As for the proposed method, both word-level and phrase-level features are designed towards rotation and scaling invariance, and hence it able to handle this challenging case. In addition, our fusionbased algorithm, using the information from homography, local similarity, and keypoint region, allows for higher accuracy.
For (a4)-(g4) and (a5)-(g5), severe noise (with standard deviation 0.1) and JPEG compression (with quality factor 20) are considered, respectively. Under such signal corruptions, the localization maps of these comparison methods exhibit varying degrees of inaccuracy, i.e., they are with notable false positives and/or false negatives. In general, the poor robustness/discriminability of their lowlevel features fundamentally contribute to the above phenomenon. Instead, we focus on building more expressive features through the BoVW model (Section III-D), with clear advantages in overcoming such challenging case as shown in Fig. 11.

V. CONCLUSION
Although copy-move is a basic and common operation in image forgery, detecting such manipulations can be very difficult, especially for the scenarios with high selfsimilarity or strong corruption. The fundamental difficulty lies in the following semantic gap problem: features should be robust to geometric/signal distortions in copy-move regions, while retaining discriminability for similar-butgenuine regions. The low-level visual representation employed in existing copy-move forgery detection algorithms is inherently constrained by the above semantic gap problem.
The algorithm designed here is a very early step towards bridging the semantic gap in copy-move forensics. In this regard, we first introduce the bag-of-visual-words model into this field, as a new perspective for the representation in copy-move forgery detection. With this perspective, we then propose a word-to-phrase feature description and matching pipeline, covering the spatial structure and visual saliency information of images. The core lies in a spatial pooling and weighting of local moment invariants for robust and discriminative representation, which is expected to alleviate the semantic gap. In addition to the above core contributions, we also give some useful modules that can complement the existing works. Specifically, our I2NN is able to deal with multiple copy-move forgeries, with much fewer mismatches and complexity; our fusionbased localization strategy using the information from homography, local similarity, and keypoint region, with ability to reduce both false positives and false negatives.
We provide statistical comparisons with both state-ofthe-art dense-field and sparse-field forensics methods, by both benchmark experiments and robustness experiments. In general, the extensive experimental results confirm our claims, exhibiting quite satisfactory accuracy and robustness w.r.t. existing works.