{'fraction_non_alphanumeric': 0.03426083938333382, 'fraction_numerical': 0.0149104792757354, 'mean_word_length': 5.328665568369028, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Making Images Real Again: A Comprehensive Survey on Deep Image Composition

As a common image editing operation, image composition aims to cut the foreground from one image and paste it on another image, resulting in a composite image. However, there are many issues that could make the composite images unrealistic. These issues can be summarized as the inconsistency between foreground and background, which includes appearance inconsistency (e.g., incompatible illumination), geometry inconsistency (e.g., unreasonable size), and semantic inconsistency (e.g., mismatched semantic context). Previous works divide image composition task into multiple sub-tasks, in which each sub-task targets at one or more issues. Specifically, object placement aims to find reasonable scale, location, and shape for the foreground. Image blending aims to address the unnatural boundary between foreground and background. Image harmonization aims to adjust the illumination statistics of foreground. Shadow generation aims to generate plausible shadow for the foreground. By putting all the abovementioned efforts together, we can acquire realistic composite images. To the best of our knowledge, there is no previous survey on image composition. In this paper, we conduct comprehensive survey over the sub-tasks of image composition. For each sub-task, we summarize the traditional methods, deep learning based methods, datasets and evaluation. We also point out the limitations of existing methods in each sub-task and the problem of the whole image composition task. Datasets and codes for image composition are summarized at https://github.com/bcmi/Awesome-Image-Composition.


I. INTRODUCTION
During image composition, we first cut the foreground from one image using image segmentation [88] or matting [131,31] techniques. Then, we paste the foreground on another image to form a composite image, as illustrated in Fig. 1(a). More generally, image composition can be used for combining multiple visual elements from different sources to construct a realistic image. As a common image editing operation, image composition could benefit various applications in the realm of entertainment, art, and commerce [12,127,147] and data augmentation [23,99,91] for downstream tasks. For example, people can replace the backgrounds of selfportraits and make the obtained images more realistic using image composition techniques. Another example is artistic creation, in which image composition can be used to create fantastic artworks that originally only exist in the imagination. Image composition could also be used for automatic advertising, which helps advertisers with the product insertion in the background scene [147]. Similarly, advertisement logo Li   Specifically, we extract the foreground from one image use image segmentation or matting techniques and paste the foreground on another background image. There could be many issues (e.g., unreasonable location and size, incompatible illumination, missing shadow) for the obtained composite image. These issues can be addressed by different sub-tasks (e.g., object placement, image blending, image harmonization, shadow generation) as shown in Subfigure (b), in which the composite image and foreground mask are taken as input to produce a realistic-looking image.
compositing [69] targets at embedding some specified logos in target images. The obtained composite images can be taken as design renderings or blueprint to help the designer and the client choose their preferable versions. After compositing a new image with foreground and background, there exist many issues that could make the composite image unrealistic and thus significantly degrade its quality. These issues can be summarized as the inconsistency between foreground and background, which can be divided into appearance inconsistency, geometric inconsistency, and semantic inconsistency. Each type of inconsistency involves a number of issues to be solved. Previous works [108,2,138,137] on image composition target at one or more issues. Since each individual issue could be very challenging, there has been no attempt to address all issues with one unified model. Instead, several divergent research directions are emerging, with each research direction focusing on partial issues (see Fig. 1(b)). Next, we will introduce each type of inconsistency one by one. Fig. 2. The quality of composite image is degraded by the appearance inconsistency, geometric inconsistency, and semantic inconsistency. Each type of inconsistency involves a number of issues. Each sub-task targets at addressing one or more issues. The appearance inconsistency is including but not limited to: 1) unnatural boundary between foreground and background; 2) incompatible illumination statistics between foreground and background; 3) missing or implausible shadow and reflection of foreground. For the first issue, the foreground is usually extracted using image segmentation [88] or matting [131,31] algorithms. However, the foregrounds may not be precisely delineated, especially at the boundaries. When pasting the foreground with jagged boundaries on the background, there would be obvious color artifacts along the boundary. To solve this issue, image blending [128,143] aims to address the unnatural boundary between foreground and background, so that the foreground could be seamlessly blended with the background. For the second issue, since the foreground and background may be captured in different conditions (e.g., weather, season, time of the day, camera setting), the obtained composite image could be inharmonious (e.g., foreground captured in the daytime and background captured at night). To solve this issue, image harmonization [116,19,14] aims to adjust the illumination statistics of foreground to make it more compatible with the background, so that the whole image looks more harmonious. For the third issue, when pasting the foreground on the background, the foreground may also affect the background with shadow or reflection. To solve this issue, shadow generation [80,146,102] or reflection generation [87] focus on generating plausible shadow or reflection for the foreground according to both foreground and background information. As far as we are concerned, there are only few works [87] on generating reflection for the inserted object probably due to the limited application scenarios, so we focus on shadow generation in this paper.
The geometric inconsistency is including but not limited to: 1) the foreground object is too large or too small; 2) the foreground object does not have reasonable supporting force (e.g., hanging in the air); 3) unreasonable occlusion; 4) inconsistent perspectives between foreground and background. In summary, the location, size, and shape of the foreground may be irrational considering the geometric constraints. Object placement [2,21,59,114,142] tends to seek for reasonable location, size, and shape by predicting the foreground transformation to avoid the abovementioned inconsistencies. Previous object placement methods [142,114] mainly predict Fig. 3. Given a pair of foreground and background, we can obtain a high-quality composite image after going through object placement, image blending, image harmonization, and shadow generation. simple form of spatial transformation, that is, shifting and scaling the foreground to achieve reasonable location and size. Some other methods [59,76] predict more advanced form of spatial transformation (e.g., affine transformation, perspective transformation) to warp the foreground. When placing the object on the background, unreasonable occlusion may occur. Most previous methods seek for reasonable placement to avoid unreasonable occlusions, while some methods [2,138,109] aim to fix unreasonable occlusion by removing the occluded regions of foreground based on the estimated depth information.
The semantic inconsistency is including but not limited to: 1) the foreground appears at a semantically unreasonable place (e.g., a zebra is placed in the living room); 2) the foreground has unreasonable interactions with other objects or people (e.g., a person is riding a motorbike, but the person and the motorbike are facing towards opposite directions). The semantic inconsistency is judged based on commonsense knowledge, so the cases of semantic inconsistency may be arguable according to subjective judgement. For example, when a car is placed in the water, it can be argued that a car is sinking into the water after a car accident. However, such event has rather low probability compared with commonly seen cases, so we can claim that the car appears at an unreasonable place, which belongs to semantic inconsistency. The solution to semantic inconsistency often falls into the scope of object placement. To be exact, by predicting suitable spatial transformation for the foreground, we can relocate the foreground to a reasonable place or adjust the pose of foreground to make its interactions with environment more convincing. Therefore, object placement assumes the duty of addressing geometric inconsistency and semantic inconsistency simultaneously.
Given a pair of foreground and background, after addressing the abovementioned inconsistencies, we can obtain a photorealistic composite image. In a broader sense, image composition can be used to combine visual elements from different real images to produce a high-fidelity composite image, which has a broad spectrum of applications (e.g., augmented reality, artistic creation, automatic advertising). In recent years, image generation [8,55] has achieved remarkable progress and demonstrated great potential to create fantastic images. We believe that image composition is irreplaceable by image generation due to its uniqueness. For example, with the goal of inserting a cat into the complex background, image generation methods can hardly produce the image with the cat exactly the same as expected, even if detailed conditional information is provided. Instead, we can paste the desired cat on the background and employ image composition techniques to obtain a natural-looking image. Therefore, image composition and image generation could be collaboratively used to create images that satisfy our requirement.
In the following sections, we will first introduce different sub-tasks arranged in the sequential order of creating a composite image. Specifically, as illustrated in Fig. 3, given a pair of foreground and background, we first use object placement to find suitable scale and location for the foreground (see Section II), and use image blending (see Section III) to refine the boundary between foreground and background. Then, we use image harmonization (see Section IV) to adjust the foreground illumination and shadow generation (see Section V) to generate plausible shadow for the foreground. In each section corresponding to one sub-task, we will introduce the traditional methods and deep learning based methods for this sub-task. Because different sub-tasks employ different datasets, we will also review the commonly used datasets and evaluation metrics in each section. Previous works on image composition may focus on one sub-task or multiple sub-tasks, aiming to address one issue or multiple issues. For ease of locating the suitable method to use, we summarize all issues and the corresponding methods to solve them in Table I. Furthermore, we will point out some limitations and suggest some potential directions in Section VII. Finally, we will conclude the whole paper in Section VIII. The contributions of this paper can be summarized as follows, • To the best of our knowledge, this is the first comprehensive survey on deep image composition. • We summarize the issues in image composition as three types of inconsistencies. We clarify the relation between inconsistency, issue, and sub-task in Figure 2. We also summarize the issues that previous works attempt to solve in Table I. All the above summaries give rise to a large picture for deep image composition. • For each sub-task, we survey the traditional methods and deep learning based methods. We also discuss their advantages and disadvantages, followed by comparison results. We believe that this comprehensive survey can serve as the roadmap for the future research in a broad community.
II. OBJECT PLACEMENT Object placement aims to paste the foreground on the background with suitable location, size, and shape. As shown  In the left subfigure, we compare category-specific object placement with instance-specific object placement. In the right subfigure, we show the taxonomy of existing object placement methods.
in Fig. 4, the cases of unreasonable object placement are including but not limited to: a) the foreground object has inappropriate size (e.g., the dog is too large); b) the foreground object has unreasonable occlusion with background objects (e.g., the fences are unreasonably occluded by the giraffe); c) the foreground object does not have reasonable force condition (e.g., the suitcase is floating in the air); d) the foreground object appears at a semantically unreasonable place (e.g., the boat appears on the land); e) inconsistent perspectives between foreground and background (e.g., the car and the bus have inconsistent perspectives). By taking all the above factors into consideration, object placement is a very challenging task.

A. Traditional Methods
Some object placement methods design explicit rules to find reasonable location and scale for the foreground object. For example, Remez et al. [99] proposed to move the foreground object of fixed scale along the same horizontal scanline on the background. They assume that the locations along the same horizontal scanline have similar depth, so that the true scale of foreground object can be well-preserved. Wang et al. [120] designed the instance-switching strategy to generate new images through switching different instances of the same class with similar shape and scale. To better refine the position where the object is pasted, Fang et al. [27] explored appearance consistency heatmap to guide the object placement, based on the intuition that an object could be moved to another location which has similar visual context to its original visual context. Specifically, one element in the appearance consistency heatmap measures the similarity between the visual context at this point and original visual context. Georgakis et al. [37] proposed to combine support surface detection and semantic segmentation to find proper location for placing the object. With the determined location, the size of the object is decided in the light of the depth at this location and the original scale of the object. Zhang et al. [147] proposed to model the probability distribution of bounding box information conditioned on background image and foreground category using Gaussian mixture model (GMM).
Although these rules are effective in some cases, they are incomplete and sometimes inaccurate, which is far below the requirement to handle the diverse and complicated challenges in object placement task.

B. Deep Learning Methods
Apart from the above methods which design explicit rules to infer the reasonable placement for the foreground object, some methods [136,2,138,109,122] employ deep learning techniques to predict the placement and generate the composite image automatically.
The existing deep learning based object placement methods can be divided into category-specific object placement and instance-specific object placement. For category-specific object placement, the model aims to predict plausible bounding Fig. 6. We show three types of methods for category-specific object placement. Generative model: given the foreground category and background image, the model generates a reasonable bounding box (e.g., location (x,y) and scale (w,h)). Slow discriminative model: given the foreground category, foreground bounding box, and background image, the model predicts a rationality score. Fast discriminative model: given the foreground category and background image, the model uses sliding window on the feature map to get the rationality score for each bounding box. Fig. 7. We show three types of methods for instance-specific object placement. Generative model: given the foreground, foreground object mask, and background, the model generates a reasonable placement (e.g., location (x,y) and scale (w,h)) for the foreground. Slow discriminative model: given the composite image and composite foreground mask, the model predicts a rationality score. Fast discriminative model: given the foreground, foreground object mask, and background, the model predicts a rationality score map containing the rationality scores for all locations. boxes given a background image and a foreground category. This group of methods assume that the predicted bounding boxes are suitable for all instances belonging to the same foreground category. Nevertheless, this assumption is too restrict, because different instances from the same category may have distinct properties (e.g., geometry, fine-grained semantics) and thus require bounding boxes with different scales/locations. In contrast, instance-specific object placement methods aim to predict plausible spatial transformations given a background image and a specific foreground object. The difference between category-specific object placement and instance-specific object placement is shown in Fig. 5. Next, we will introduce these two groups of works separately.
1) Category-specific Object Placement: Category-specific object placement methods can be categorized into generative approach and discriminative approach. The generative approach targets at predicting one or multiple reasonable bounding boxes for the foreground category, whereas the discriminative approach aims to predict the rationality score of a bounding box for certain foreground category. The discriminative approach can be further divided into slow discriminative approach and fast discriminative approach. The slow discriminative approach takes in a background image with foreground bounding box and predicts a rationality score. The fast discriminative approach takes in a background and produces a feature map, based on which sliding window is used to predict the rationality score of each bounding box. The comparison between generative model, slow discriminative model, and fast discriminative model is illustrated in Fig. 6.
Generative approaches: Tan et al. [108] proposed to predict the location and scale of inserted object by taking the background image and object layout as input. Besides, the bounding box prediction task is converted to a classification task by discretizing the locations and scales. Lee et al. [64] studied on taking a background semantic map instead of a background image as input. Given a background semantic map, they designed a network consisting of two generative modules, in which the first module accounts for the bounding box of inserted object and the second module accounts for the mask shape of inserted object.
Discriminative approaches: The methods in [21,22] used a network to predict whether a bounding box is suitable for certain foreground category, based on the contextual information surrounding the bounding box. This approach needs to pass through the network once for each bounding box, which is very time-consuming. To accelerate this process, Volokitin et al. [119] employed masked convolutions to aggregate the contextual information along four directions as context feature maps, based on which the contextual information excluding each bounding box can be obtained efficiently to predict the  [82]. From left to right in each row, we show the foreground, background, and composite images obtained by TERSE [114], PlaceNet [142], SimOPA (SOPA) [82], FOPA [90]. rationality score for this bounding box.
2) Instance-specific Object Placement: Instance-specific object placement methods can also be categorized into generative approach and discriminative approach. The generative approach targets at predicting one or multiple reasonable placements (i.e., spatial transformations) for the foreground object, whereas the discriminative approach aims to predict the rationality score of a composite image in terms of the foreground object placement. The discriminative approach can be further divided into slow discriminative approach and fast discriminative approach. The slow discriminative approach takes in a composite image and predicts a rationality score. The fast discriminative approach takes in a pair of foreground and background, and predicts a rationality score map. The comparison between generative model, slow discriminative model, and fast discriminative model is illustrated in Fig. 7.
Generative approaches: Generative approaches [114,142,76,70,69] predict different types of spatial transformations (e.g., shifting and scaling, affine transformation, perspective transformation) for the foreground object, which is more flexible and powerful than category-specific object placement methods. For instance, Tripathi et al. [114] developed a model with generator, discriminator, and target network. Given a pair of background and foreground, the generator predicts the affine transformation for the foreground object to produce a composite image. The produced composite image is expected to fool the discriminator and fit the target network corresponding to a downstream task (e.g., object detection). Similarly, Zhan et al. [136] adopted spatial transformer network (STN) [52] to predict the warping parameters under an adversarial learning framework. To produce multiple reasonable placements, Zhang et al. [142] combined the foreground feature, background feature, and a random vector to predict the object placement. Moreover, they ensure the diversity of object placement by enforcing the pairwise distances between predicted placements to approach those between corresponding random vectors. To promote the diversity of generated placements, Zhou et al. [149] established the bijection between random vector and positive composite image. Moreover, they reformulated object placement as a graph completion task. In particular, background nodes have both content features and placements, while the inserted foreground node only has content feature, giving rise to an incomplete graph. Hence, they estimated the missing placement for the foreground node to complete the graph. Azadi et al. [2] employed STN to warp the foreground and relative appearance flow network to change the viewpoint of foreground. Additionally, they investigated on self-consistency constraint, that is, the generated composite image could be decomposed back to the foreground and background. ST-GAN [76] proposed to warp a foreground object into a background image with iterative spatial transformations predicted by STN. As a follow-up work, Kikuchi et al. [59] replaced the iterative spatial transformations in [76] with one-shot spatial transformation.
Discriminative approaches: Liu et al. [82] proposed a discriminative approach named SimOPA to verify whether a composite image is rational in terms of the foreground object placement. Particularly, they feed the concatenation of composite image and foreground mask into a binary classification network to predict a rationality score. However, this discriminative approach is very inefficient, because they need to go through the discriminative network multiple times to find a reasonable object placement. To address this issue, Niu et al. [90] dubbed SimOPA as slow object placement assessment (SOPA) model and proposed a fast object placement assessment (FOPA) model, which can predict the rationality scores at all locations by going through the model only once. Precisely, they take in a pair of background and scaled foreground, and produces a rationality score map, in which each entry represents the rationality score of the composite image obtained by pasting the foreground at this location. They developed several innovations (e.g., background prior transfer, feature mimicking) to bridge the performance gap between FOPA and SOPA, reaching the conclusion that FOPA can achieve comparable performance with SimOPA at significantly reduced cost.
In the end, we briefly discuss the occlusion issue. Most of the above methods seek for reasonable placements to avoid the occurrence of occlusion, i.e., the inserted foreground is not occluded by background objects. Differently, a few methods [2,138,109] attempt to address the unreasonable occlusion when it occurs. Specifically, they first estimate the relative depth relation between the foreground object and the surrounding background objects. Then, they remove the occluded part of foreground object. In this way, they are able to generate composite images with reasonable inter-object occlusions.

C. Datasets and Evaluation Metrics
In some previous works [114,27], object placement is used as data augmentation strategy to facilitate the downstream tasks (e.g., object detection, instance segmentation). Therefore, they make use of existing object detection and instance segmentation datasets [77,26,18,36]. In particular, the foregrounds are cropped out based on the annotated segmentation masks. After removing the foreground objects, the remaining incomplete background images are restored to complete background images by using image inpainting techniques [134,81,135]. In this manner, triplets of foregrounds, backgrounds, and ground-truth composite images can be obtained. Some other works focus on specific applications like 2D virtual try-on [76,59,70] (e.g., placing glasses/hats on human faces) or logo composition [69] (e.g., attaching logo to product image), so they need to collect foregrounds and backgrounds specifically for these applications. More recently, Liu et al. [82] released a large-scale object placement assessment dataset named OPA, which consists of 73,470 composite images and their binary rationality labels. OPA dataset is constructed by compositing the foregrounds and backgrounds from COCO dataset [77], followed by manually labelling the rationality of obtained composite images. A large number of annotated composite images could greatly facilitate the research on object placement.
To evaluate the quality of generated composite images, previous object placement works usually adopt the following three schemes: 1) Some works measure the similarity between real images and composite images. For example, Tan et al. [108] score the correlation between the distributions of predicted boxes and ground-truth boxes. Zhang et al. [142] calculate Frechet Inception Distance (FID) [43] between composite images and real images. However, they cannot evaluate each individual composite image. 2) Some works [114,27] utilize the performance improvement of downstream tasks (e.g., object detection) to evaluate the quality of composite images, where the training sets of the downstream tasks are augmented with generated composite images. However, the evaluation cost is quite huge and the improvement in downstream tasks may not reliably reflect the quality of composite images, because it has been revealed in [38] that randomly generated unrealistic composite images could also boost the performance of downstream tasks. 3) Another common evaluation strategy is user study, where people are asked to score the rationality of object placement [64,108]. User study complies with human perception and each composite image can be evaluated individually. However, due to the subjectivity of user study, the gauge in different papers may be dramatically different. There is no unified benchmark dataset and the results in different papers cannot be directly compared. 4) Finally, with the recently released OPA dataset [82], we could use the annotated composite images for evaluation. Nevertheless, the sparse annotations only cover a small proportion of locations and scales, which limits the universal evaluation of arbitrary composition results.

D. Experiments
In this section, we focus on instance-specific object placement and compare existing object placement methods for generating a reasonable composite image. For ease of comparison, we fix the foreground scale and only predict the reasonable location for the foreground object. Recall that instance-specific object placement methods are divided into generative approaches and discriminative approaches. For generative approach, we choose TERSE [114] and PlaceNet [142], which can directly predict one placement. For discriminative approach, we report the results of SimOPA [82] and FOPA [90]. We use SimOPA and FOPA to generate rationality score map, based on which the location with the largest rationality score is chosen as the optimal placement. We train and evaluate different methods on OPA dataset [82]. The test results are shown in Fig. 8, from which it can be seen that discriminative approaches usually achieve better results than generative approaches. One possible explanation is that TERSE [114] and PlaceNet [142] only utilize the annotated composite images to update the discriminator, without fully using the annotations to train the generator. Nevertheless, discriminative approaches also have failure cases when dealing with occlusion and complex scenes (e.g., unreasonable occlusion between fire hydrant and fallen branches in row 4).

III. IMAGE BLENDING
During image composition, the foreground is usually extracted using image segmentation [88] or matting [131] methods. However, the segmentation or matting results may be noisy and the foregrounds are not precisely delineated. When the foreground with jagged boundaries is pasted on the background, there will be abrupt intensity change between the foreground and background. To refine the boundary and reduce the fuzziness, image blending techniques have been developed.

A. Traditional Methods
Traditional image blending methods aim to smooth the transition between foreground and background. Alpha blending [97] proposed to assign alpha values for boundary pixels indicating what fraction of the colors are from foreground or background, in which the alpha values need to be manually set. Alpha blending is a simple and fast method, but it blurs the fine details and brings in ghost effects. Considering multiscale information, Laplacian pyramid blending [9] proposed to build multi-scale Laplacian pyramids for two images and perform alpha blending at each scale. Then, the final output is obtained by adding up the blended results of different scales.
Another group of methods attempt to achieve smooth boundary transition by enforcing gradient domain smoothness [29,58,65,107]. The earliest work along this research direction is Poisson image blending [95]. Poisson image blending [95] proposed to enforce the gradient domain consistency with respect to the source image containing the foreground, where the gradient of inserted foreground is computed and propagated from the boundary pixels in the background. Although Poisson image blending can yield more pleasant results than simple alpha blending, it is very time-consuming to solve the Poisson equation. Therefore, there are many follow-up works [107,110,58] to accelerate Poisson image blending by using different techniques. Based on the observation that the effectiveness of Poisson image blending seriously depends on the boundary condition, [53] designed a method to optimize the boundary condition. To avoid the color bleeding and halo effect brought by Poisson image blending, Tao et al. [112] developed a two-step algorithm: first processing the gradient values on the boundary and then employing a weighted integration scheme to reconstruct the image from its gradient field.
The above methods based on gradient domain smoothness can smooth the transition between foreground and background to some extent. However, the colors of background may seep through the foreground too much and distort the foreground color, which would bring significant loss to the foreground content.

B. Deep Learning Methods
Inspired by traditional image blending methods [95,9], some recent works [128,143] explored incorporating the function of smoothing boundary into deep learning network. Among them, the works [128,143,140] not only enable smooth transition over the boundary, but also reduce the illumination discrepancy between foreground and background, in which the latter one is the goal of image harmonization in Section IV. In this section, we only introduce the way they enable smooth transition over the boundary. These two works [128,143] are both inspired by [95]. Specifically, they add the gradient domain constraint to the objective function according to Poisson equation, which can produce a smooth blending boundary with gradient domain consistency. They both optimize over the input composite image to minimize the gradient domain loss. Differently, [128] has a closeform solution, while [143] converts the gradient domain loss to a differentiable loss function and uses gradient descent algorithm.
Different from [128,143], Zhang et al. [140] proposed a new learnable image blending paradigm. Specifically, the fusion network proposed in [140] takes in a pair of foreground image and background image, producing a seamlessly blended image. Inspired by [9], two separate encoders are used to extract and fuse multi-scale features from foreground and background. Because the fusion network relies on groundtruth composite images obtained by using accurate alpha matte as supervision, the work [140] also proposed an easy-to-hard data-augmentation scheme to relieve the burden of annotating ground-truth alpha matte.

C. Datasets and Evaluation Metrics
To the best of our knowledge, there are only few deep learning methods [128,143,140] for image blending and there is no unified benchmark dataset. Zhang et al. [143] do not mention the source of used images. Wu et al. [128] manually crop objects from Transient Attributes Database [62] to create input composite images. Similarly, Zhang et al. [140] take foreground images from segmentation datasets [46,101] and random background images to construct input pairs.
The existing deep image blending works [128,143,140] adopt the following evaluation metrics: 1) calculating realism score using the pretrained model [150] which reflects the realism of a composite image; 2) conducting user study by asking engaged users to select the most realistic images; 3) Zhang et al. [140] deem the composite images obtained using ground-truth alpha matte as ground-truth composite image, and calculate Peak Signal-to-Noise Ratio (PSNR) between resultant image and ground-truth composite image. Fig. 9. The leftmost column is the initial composite image obtained using the alpha matte predicted by LFPNet [83]. The rightmost column is the ground-truth composite image obtained using ground-truth alpha matte. The middle columns are the refined results obtained by Poisson image blending [95], GP-GAN [128], Zhang et al. [143], and MLF [140]. The odd rows display the whole image, while the even rows zoom in the red bounding boxes in the odd rows for better observation.

D. Experiments
We evaluate different image blending methods conditioned on the matting results. First, we create composite images using the alpha matts predicted by the state-of-the-art trimap-based image matting methods [20,84,83]. Then, we hope that image blending methods can refine the obtained composite images. We collect 4500 foreground images from recent image matting datasets [83,67,66], which are divided into 4000 for training and 500 for testing. For each foreground image, we randomly select two background images from BG20K [68]. The background images for the training set and test set have no overlap. The training set is used to train the fusion network in MLF [140].
By taking LFPNet [83] as an example matting method, we predict the alpha mattes and obtain the composite images. We observe that LFPNet can generally achieve satisfactory results except some challenging cases. We pick out its several failure cases to verify the effectiveness of image blending methods. We report the results of Poisson image blending [95], GP-GAN [128], Zhang et al. [143], and MLF [140]. We also report the ground-truth composite image obtained using ground-truth alpha matte for comparison. From Fig. 9, it can be seen that the obtained composite images using predicted alpha mattes are very close to the ground-truth composite image except partial boundary regions. We observe that Poisson image blending [95] smooths the transition boundary to some extent, but unexpectedly distorts the foreground content by seeping through the foreground. GP-GAN [128] and Zhang et al. [143] are inspired by Poisson image blending, but use content loss to preserve the original foreground content. Therefore, they strike a balance between preserving the foreground content and smoothing the boundary. However, some smoothed boundary regions (e.g., hair in the last row) are still not satisfactory. MLF [140] can obtain visually appealing results in some cases. Nonetheless, it may erase detailed information (e.g., the small leaves of pineapple) and fail in handling transparent foreground objects (e.g., plastic bag).

IV. IMAGE HARMONIZATION
Given a composite image, its foreground and background are likely to be captured in different conditions (e.g., weather, season, time of day, camera setting), and thus have distinctive illumination characteristics, which make them look incompatible. Image harmonization aims to adjust the appearance of composite foreground according to composite background to make it compatible with the composite background.

A. Traditional Methods
The general thought of traditional image harmonization methods [132,106,63,105] is performing color transformation for the foreground to match the low-level color statistics between foreground and background. The difference between different methods mainly lies in the matching details. For example, Xue et al. [132] proposed to train a classifier to predict the zone (e.g., low, middle, high) of the histogram which can be best matched between foreground and background, and then adjust the foreground color to match the selected zone between foreground and background. [106] explored decomposing an image into a multi-resolution pyramid with multiple subbands, and performing histogram matching for each subband between foreground and background. Lalonde and Efros [63] proposed to represent foreground and background with color clusters, followed by matching foreground and background color clusters. Song et al. [105] proposed to calculate the color transformation (channel-wise scales) based on gray pixels of foreground and background, because normalized illumination color can be directly derived from the pixel values of gray pixels. In some works on sky replacement [152,111,115], they attempted to match the color statistics (e.g., mean, variance) between sky region and non-sky region. Broadly speaking, traditional color transfer methods [98,129,30,96,28,1] can also be used for color matching between foreground and background to produce a harmonized image.

B. Deep Learning Methods
Early deep learning based image harmonization methods target at making the harmonized images indistinguishable from real images. For instance, Zhu et al. [150] explored predicting the realism of an image using a CNN classifier. With such realism predictor, they learn the color transformation for the foreground to achieve high realism score, and also enforce the color variation in different channels to be close. Similar to [150], the works [137,12] used adversarial learning to make the harmonized images indistinguishable from real images. Bhattad and Forsyth [5] drew inspiration from Retinex theory [60] that an image can be decomposed into albedo (reflectance) and shading (illumination). On the premise of this assumption, an image harmonization model is trained so that the harmonized result should have consistent albedo and consistent background shading with input composite image.
With the emergence of image harmonization datasets consisting of paired training data (see Section IV-E), abundant image harmonization methods [116,104,19,14,16,42,40,48,74,151] using paired supervision have been developed. Tsai et al. [116] proposed the first end-to-end CNN network for image harmonization and leveraged auxiliary semantic segmentation branch to enhance the basic image harmonization network. Another work Sofiiuk et al. [104] also utilized highlevel semantic features, which are inserted into the encoder to provide auxiliary information.
Cun and Pun [19] designed an additional Spatial-Separated Attention Module to deal with foreground and background feature maps separately. Hao et al. [42] employed selfattention [123] mechanism to propagate relevant information from background to foreground.
By treating different capture conditions as different domains, Cong et al. [14] proposed a domain verification discriminator to pull close the foreground domain and background domain. Similarly, Cong et al. [16] formulated image harmonization as background-guided domain translation task, in which the domain code of background is directly used to guide the harmonization process. One byproduct of [16] is predicting the inharmony level of an image by comparing the domain codes of foreground and background, so that we can selectively harmonize those apparently inharmonious composite images.
Analogous to [5] using Retinex theory, Guo et al. [40] also developed a model to disentangle a composite image into reflectance map and illumination map, in which the illumination map is harmonized by transferring lighting information from background to foreground. Another work [39] also adopted the similar decomposition network and explored integrating transformer block [118] into the network. Following the disentanglement technical route, Jiang et al. [54] proposed to disentangle an image into content representation and appearance representation. Then, the appearance representation of foreground is superseded by that of background to accomplish the goal of image harmonization.
In [78], they reframed image harmonization as a background-to-foreground style transfer problem and introduced region-aware adaptive instance normalization (AdaIn) to transfer visual style from the background to the foreground. A succeeding work [41] extended [78] by searching foreground-relevant background regions and transferring foreground-relevant style from background to foreground. They also extended the triplet loss in [16] to contrastive loss.
Inspired by traditional image harmonization methods [132,106] which applied color transformation to adjust the foreground appearance, Cong et al. [17] proposed to learn color transformation using deep learning for image harmonization. They combined color-to-color transformation and pixel-topixel transformation in a unified framework coherently.

C. Variants of Image Harmonization Task
In this subsection, we discuss two variants of standard image harmonization task. Fig. 10. In the left subfigure, we summarize three ways to construct image harmonization dataset and list the corresponding datasets: RealHM [54], iHarmony4 [15] (HCOCO, HFlickr, HAdobe5k, Hday2night), GMS [105], HVIDIT [40], RHHarmony [15]. We also mark the dataset based on real (resp., rendered) images with red (resp., green) box. In the right subfigure, we provide the illustration of three ways to construct image harmonization dataset. Fig. 11. In the first (resp., second, third) row, we show two examples from RealHM [54] (resp., HFlickr in iHarmony4 [15], HVIDIT [40]) dataset. From left to right in each example, we show the composite image, the foreground mask, and the ground-truth harmonized image.
Blind image harmonization: Most image harmonization methods require the foreground mask as input, which means that the inharmonious region is known in advance. However, in real-world applications, we may not know the exact inharmonious region in advance. Image harmonization without foreground mask is called blind image harmonization. Cun and Pun [19] considered the problem of blind image harmonization. They proposed to predict the inharmonious region mask in the attention block, which deals with the foreground and background separately according to the predicted mask. Subsequently, some works [72,73] focus on inharmonious region localization task, which aims to localize the inharmonious region in an image. Liang et al. [72] explored aggregating multi-scale contextual information and suppressing redundant information to enhance the inharmonious region localization performance. Liang et al. [73] designed a color mapping module to map the input image to another color space, which can magnify the domain discrepancy between foreground and background.
Painterly image harmonization: In standard image harmonization, both foreground and background are from realistic images. There exist certain application scenarios that the background is an artistic image while the foreground is from a realistic image, in which case the standard image harmonization models may not work well. To overcome this problem, painterly image harmonization [86] has been studied to harmonize the realistic foreground according to the artistic background to obtain a uniformly stylized composite  [14]. From left to right in each row, we show the input composite image, the harmonization results of DoveNet [14], RainNet [78], IIH [40], iSSAM [104], CDTNet [17]), and the ground-truth harmonized image.
image. The relation between painterly image harmonization and standard image harmonization is like the relation between photorealistic style transfer and artistic style transfer. Luan et al. [86] proposed to optimize the input image with two passes to minimize the style loss and content loss, in which the first pass aims at robust coarse harmonization and the second pass targets at high-quality refinement. Analogous to [78], Peng et al. [94] applied adaptive instance normalization to match the means and variances between the feature map of composite image and that of artistic background.

D. Related Research Fields
Here, we briefly discuss two research fields related to image harmonization.
Image relighting: Image relighting [92,103,124,144,113] aims to adjust the appearance of an image or the object in an image as lit by novel illumination. With some adaptation, image relighting can also be used to adjust the foreground appearance according to the illumination of a new background [92,144], which bears some resemblance to image harmonization. However, they usually achieve this goal by inferring explicit illumination condition, material properties, and 3D geometry, in which the supervision for these information is difficult and expensive to acquire. Besides, they generally have strong assumption for the light source, which may not generalize well to complicated real-world scenes.
Style transfer: Both artistic style transfer [35,49,93] and photorealistic style transfer [85,71] are referred to as style transfer. Image harmonization is closer to photorealistic style transfer, which transfers the style of a reference photo to another input photo. There are two main differences between image harmonization and photorealistic style transfer. 1) Firstly, image harmonization adjusts the foreground appearance according to the background, which needs to take the foreground location into consideration due to the locality property. In contrast, photorealistic style transfer adjusts the appearance of a whole input image according to another whole reference image. 2) Secondly, the definition of "style" in photorealistic style transfer is unclear and coarsely depends on the employed style loss (e.g., Gram matrix loss [35], AdaIn loss [49]). Differently, the goal of image harmonization is clearly adjusting the illumination statistics of foreground, so that the resultant foreground looks like the same object captured in the background illumination condition.

E. Datasets and Evaluation Metrics
A large amount of composite images can be easily obtained by pasting the foreground from one image on another background image, but it is not easy to obtain the ground-truth harmonized image for the composite image. Training deep learning models requires abundant pairs of composite images and ground-truth harmonized images. Existing works have designed different schemes to construct image harmonization dataset. We categorize the existing schemes into three groups: forward adjustment, backward adjustment, and replacement. Note that some datasets are constructed based on real images while some other datasets are constructed using rendering techniques. We summarize the existing image harmonization datasets in Fig. 10. Moreover, we show one representative dataset from each group in Fig. 11.
Forward adjustment: Jiang et al. [54] released a smallscale RealHM dataset with 216 image pairs, which is constructed by manually harmonizing the composite image. However, manually adjusting the foreground according to the background to obtain the ground-truth is time-consuming, labor-intensive, and unreliable.
Backward adjustment: In contrast with manually adjusting the foreground of composite image to create harmonized image, some other works [116,19,14] adopted an inverse approach, i.e., adjusting the foreground of real image to create synthetic composite image. Specifically, they treat a real image as harmonized image, segment a foreground region, and adjust this foreground region to be inconsistent with the background, yielding a synthetic composite image. Cong et al. [14] released the first large-scale image harmonization dataset iHarmony4 with 73146 pairs of synthetic composite images and groundtruth real images. iHarmony4 consists of four sub-datasets, in which three sub-datasets (HCOCO, HFlickr, HAdobe5k) are constructed using the aforementioned scheme. HCOCO (resp., HFlickr) sub-dataset is built upon COCO [77] (resp., crawled images from Flickr website), in which the foregrounds in real images are adjusted using traditional color transfer methods [98,129,30,96]. HAdobe5k sub-dataset is generated based on MIT-Adobe FiveK dataset [10], in which the foregrounds in real images are manually edited. When using automatic color transfer for foreground adjustment (e.g., HCOCO, HFlickr), a large number of pairs of synthetic composite images and real images can be easily obtained.
Replacement: A natural way to build image harmonization dataset is collecting a set of foreground images captured in different illumination conditions, followed by replacing one foreground with another counterpart. For example, Transient Attributes Database [62] contains 101 sets, in which each set has well-aligned images for the same scene captured in different conditions (e.g., weather, time of the day, season). This dataset has been used to construct pairs in Hday2night sub-dataset in iHarmony4 [14]. However, collecting the dataset like [62] calls for capturing the same scene with a fixed camera for a long time, which is hard to be realized in practice. To obtain the foregrounds in different capture conditions, Song et al. [105] proposed an interesting way to construct GMS Dataset. Specifically, they place the same physical model (3D foreground object) in different lighting conditions to capture different images and aligned the foregrounds in different images. Nevertheless, the collection cost is still very high and the diversity of foreground is restricted. In lieu of capturing images in the real world, an easy alternative is varying the illumination condition in a virtual environment. Cong et al. [15] constructed RHHarmony dataset by varying the lighting condition of the same scene using 3D rendering techniques. Within a set of images with the same scene yet various lighting conditions, the composite images could be obtained by exchanging the foregrounds between two images. Similarly, Guo et al. [40] constructed HVIDIT dataset based on the rendered dataset [25]. However, the rendered images have large domain gap with real images, so the harmonization model trained on rendered images cannot be directly applied to real test images.
For quantitative evaluation, existing works adopt metrics including Mean Square Error (MSE), Peak Signal-to-Noise Ratio (PSNR), Structural SIMilarity index (SSIM) [100], Learned Perceptual Image Patch Similarity (LPIPS) [145] to calculate the distance between harmonized result and groundtruth. These metrics can also be calculated only within the foreground region. For qualitative evaluation, they conduct user study on real composite images by asking engaged users to select the most realistic images and calculate the metric (e.g., B-T score [7]).

F. Experiments
Since iHarmony4 [14], which includes HCOCO, HFlickr, HAdobe5k, and Hday2night, is the most commonly used dataset for image harmonization, we evaluate different methods on iHarmony4 dataset. All methods are trained on the combination of training sets from four sub-datasets, and evaluated on the test set from each sub-dataset. In Fig. 12, we show the harmonized results of different methods (DoveNet [14], RainNet [78], IIH [40], iSSAM [104], CDTNet [17]). We observe that some competitive methods can generally produce visually appealing results that are close to the groundtruth images. However, when the background illumination is very complex or the composite foreground and background have dramatically divergent illumination statistics, the existing methods are still struggling to harmonize the foreground to approach the ground-truth.

V. SHADOW GENERATION
In the previous section, image harmonization methods could adjust the foreground appearance to make it compatible with the background, but they ignore the fact that the inserted object may also have impact on the background (e.g., reflection, shadow). For example, if background objects cast shadows on the ground but the inserted object does not have shadow, the composite image would look unrealistic. To address this issue, shadow generation task aims to generate plausible shadow for the foreground object according to background illumination information to make the composite image more realistic.

A. Traditional Methods
The traditional methods [56,57,79,75] usually use rendering techniques to generate shadow for the inserted foreground object, which needs to collect or estimate the scene geometry, foreground object geometry, and scene illumination. For example, [56] proposed to collect the rough geometry information and lighting information from users, based on which rendering techniques could be employed. However, it is very tedious and sometimes impossible to collect all the required information. In [57,79,75], they attempted to estimate the missing information (e.g., scene geometry, lighting information) automatically. With the recovered information, the local region to place the inserted 3D object is rendered with and without the inserted foreground object. The difference between these two rendered images reveals the impact of foreground object on the background, which is added to the input composite image to produce the target image with foreground shadow. Geometry estimation and lighting estimation based on a single image have been long studied, and many different technical approaches have been developed [79,61]. More recently, some methods [75,33,141,34,44,126] endeavored to estimate illumination condition and scene geometry based on a single image using deep learning models, which could achieve better performance than traditional estimation models. Nevertheless, it is still very challenging to accurately estimate the geometry and lighting information in complex real-world scenes. Erroneous estimation may mislead the rendering process and produce terrible results [146].

B. Deep Learning Methods
Recently, some works treat shadow generation as an imageto-image translation task, and develop deep networks which translate input composite image without foreground shadow to the target image with foreground shadow. For instance, Zhan et al. [137] used an auto-encoder to predict the shadow mask with a pretrained illumination model [32,13] to provide illumination information. The generated images are pushed towards real images with foreground shadows using adversarial learning.
Other methods [146,50,80,45] utilized paired training data (paired images with and without foreground shadow) to generate better shadow images. ShadowGAN [146] employed standard conditional GAN with reconstruction loss, local adversarial loss, and global adversarial loss to generate shadow for the inserted 3D foreground objects. Inoue et al. [50] developed a multi-task framework with two decoders accounting for depth map prediction and ambient occlusion map prediction respectively. ARShadowGAN [80] proposed an attention-guided residual network. The network predicts two attention maps for background shadow and occluder respectively, which are concatenated with composite image and foreground object mask to produce a residual shadow image. SGRNet [45] designed a two-stage shadow generation network. In the first stage, foreground features and background features are interacted using cross-attention to predict a shadow mask. In the second stage, they predict shadow parameters which are used to darken the input composite image. Then, the darkened image is combined with the input composite image with shadow matte.
Some other shadow generation methods are not designed for our task, i.e., generating shadow for the foreground object in a composite image, but they can be somehow adapted to our task. Mask-ShadowGAN [47] explored conducting shadow removal and shadow generation with unpaired data at the same time, which satisfies cyclic consistency. The shadow generation branch can be directly extended to generate foreground shadow. Sheng et al. [102] designed a shadow generation network to generate soft shadow for foreground object with user control. They first predict ambient occlusion map, which is jointly used with user-provided light map to produce soft shadow mask. When adapted to our task, an environment light map needs to be inferred from background before using their network.
The above methods can generate reasonable shadows for foreground objects in the composite images with simple scene and illumination condition, but often fail to generate reasonable shadows for the composite images with complex scene and illumination condition. Moreover, the generated shadows have roughly correct locations and shapes, but lack realistic contours and details matching the foreground objects.

C. Datasets and Evaluation Metrics
Similar to image harmonization in Section IV, composite images without foreground shadows can be easily obtained. Nonetheless, it is very difficult to obtain paired data, i.e., a composite image without foreground shadow and a groundtruth image with foreground shadow, which are required by supervised deep learning methods on shadow generation [146,80,45]. Some works [146,80] construct rendered datasets with paired data by inserting a virtual object into 3D scene and generating shadow for this object with rendering technique. AR- Fig. 13. In the first row, we show two examples from Shadow-AR dataset [80], which is constructed based on rendered images. In the second row, we show two examples from DESOBA dataset [45], which is constructed based on real images. From left to right in each example, we show the composite image without foreground shadow, the foreground mask, and the ground-truth image with foreground shadow.
ShadowGAN [80] released a rendered dataset named Shadow-AR by inserting a foreground object into real background image and generating its corresponding shadow with rendering technique. Shadow-AR dataset contains 3, 000 quintuples, in which each quintuple consists of a composite image without foreground shadow, its corresponding ground-truth image with foreground shadow, foreground object mask, background object mask, and background shadow mask. Shadow-AR dataset only uses 13 foreground objects from ShapeNet [11] and Stanford 3D scanning repository, so the diversity of dataset is very limited. Some examples in Shadow-AR dataset are exhibited in the first row in Fig. 13, in which we show the composite image without foreground shadow, foreground object mask, and ground-truth image with foreground shadow. Similar to ARShadowGAN [80], ShadowGAN [146] also adopted rendering technique to construct a rendered dataset, which uses 9, 265 foreground objects from ShapeNet [11] and 110 background textures (e.g., woollen, stone, tablecloth) collected from Internet.
Although it is feasible to generate paired data using rendering technique, the rendered images have large domain gap with real images. When applying the model trained on rendered images to real images, the performances are usually significantly degraded. To overcome this drawback, Hong et al. [45] constructed paired data by manually removing the foreground shadows from real shadow images in SOBA dataset [121] to produce synthetic composite images without foreground shadows, leading to DESOBA dataset. This strategy to create synthetic composite image is similar to the backward adjustment for constructing image harmonization dataset (see Section IV). In particular, Hong et al. [45] first remove all shadows from a shadow image to create a shadow-free image. Then, one foreground shadow region in the shadow image is overlaid by the counterpart in its corresponding shadow-free image, yielding a synthetic composite image with one missing foreground shadow. DESOBA dataset contains 839 training images with totally 2,995 object-shadow pairs and 160 test images with totally 624 object-shadow pairs. Some examples in DESOBA dataset are exhibited in the second row in Fig. 13, in which we show the composite image without foreground shadow, foreground object mask, and ground-truth image with foreground shadow.
To evaluate the quality of generated composite images with foreground shadows, existing shadow generation works [137] without paired data adopt Frechet Inception Distance (FID) [43] and Manipulation Score (MS) [12] to measure the realism of generated shadow images. For the works [80,45] with paired data, they adopt Structural SIMilarity index (SSIM) [100] and Root Mean Square Error (RMSE) [4] to measure the difference between generated image and groundtruth image. SSIM and RMSE can also be calculated only within the ground-truth foreground shadow region. Liu et al. [80] also use Balanced Error Rate (BER) [89] to evaluate the quality of predicted shadow mask based on ground-truth shadow mask.

D. Experiments
We compare existing shadow generation methods Shadow-GAN [146], ARShadowGAN [80], and SGRNet [45] on the DESOBA dataset [45]. We also include classic paired imageto-image translation method Pix2pix [51] for comparison. We show the shadow images generated by different methods in Fig. 14. It can be seen that most methods are struggling to produce reasonable shadow for the foreground object, or even produce no shadow at all, which implies that shadow generation for the inserted foreground object is a very tough task. Among the existing methods, SGRNet [45] achieves relatively compelling results, but the shapes of generated shadows are unrealistic and there is still plenty of room for improvement.

VI. OTHER ISSUES
As mentioned in [68], apart from the appearance inconsistency, geometric inconsistency, and semantic inconsistency between foreground and background, there could be some  [45]. From left to right in each row, we show the input composite image, the composite foreground mask, the generated results of Pix2pix [51], ShadowGAN [146], ARShadowGAN [80], SGRNet [45], and the ground-truth shadow image. other issues in a composite image, like the resolution inconsistency and sharpness inconsistency between foreground and background.
Resolution inconsistency: The foreground and background may be from two images with huge resolution gap, leading to obvious artifacts in the composite image. We could carefully select foreground and background with comparable resolutions to circumvent this issue. Otherwise, we could perform downsampling to lower the resolution of foreground/background, or adopt super resolution methods [125,133] to raise the resolution of foreground/background, with the goal of matching the resolution between foreground and background.
Sharpness inconsistency: To distinguish the important region in an image, photographers are likely to sharpen the import region and blur the unimportant region, by adjusting the aperture size, focal length, and depth-of-field. One common case is that in an image, the foreground is sharp and salient, while the background is blurry with out-of-focus effect. In this situation, when inserting a sharp object into the blurry background, the inserted object is apparently incompatible with the other background area. One naive solution is using proper smoothing filters to blur the inserted object.

VII. LIMITATIONS AND FUTURE DIRECTIONS
The past few years have witnessed tremendous progress achieved in the realm of image composition. However, there exist several limitations which severely hinder the development of image composition techniques: 1) Most existing image composition methods focus on a single unoccluded foreground and only a few works [138,142] discuss multiple foregrounds. However, in real-world applications, we are likely to paste multiple foregrounds on the background and the foregrounds may be occluded with each other. Therefore, more efforts are in high demand for adapting image composition methods to real-world applications.
2) There are many issues to be solved to obtain a realistic composite image. As shown in Table I, existing methods attempt to solve one issue or multiple issues simultaneously. Since each issue could be very complicated and hard to solve, different research directions (e.g., object placement, image harmonization) diverge from image composition to solve different issues. There is no unified system to address all the issues in image composition, which increases the difficulty in deploying image composition model in real-world applications.
3) For image composition task, the ground-truth images are very difficult to acquire. For each sub-task (e.g., object placement, image harmonization), researchers have proposed different schemes to build benchmark datasets. There is no unified benchmark dataset across different sub-tasks, which also obstructs developing a unified model for the holistic image composition task. Additionally, for some sub-tasks (e.g., image harmonization, shadow generation), the quantitative evaluation metrics (e.g., MSE) may not coincide with human perception. Therefore, it is imperative to set up unified benchmark dataset and design more reliable evaluation metrics.
At last, we suggest some potential research directions for image composition: 1) As mentioned above, in real-world applications, we are likely to paste multiple foregrounds on the background and the foregrounds may be occluded with each other. To handle the occluded foregrounds, we probably need to complete the occluded foregrounds beforehand using image outpainting or object completion techniques [117,148,6,139,130,24]. To handle multiple foregrounds, we need to consider their relative depth and complicated interaction to compose a realistic image.
2) Most existing image composition methods are from image to image, i.e., 2D → 2D. In previous rendering based works, numerous efforts have been made to infer the 3D geometric information of foreground/background as well as the illumination information. Ideally, with the complete geometric and illumination information, we can harmonize the foreground and generate foreground shadow accurately. However, it is still very challenging to reconstruct the 3D world and estimate the illumination condition based on a single 2D image. During this procedure, the induced noisy or redundant information may adversely affect the final performance. Nevertheless, 2D → 3D → 2D is an intriguing and promising research direction. We can locate an applicable middle ground between 2D → 2D and 2D → 3D → 2D to take the advantages of both.

VIII. CONCLUSION
In this paper, we have conducted a comprehensive survey on image composition. Although the image composition task sounds simple, it actually requires a variety of techniques to produce a realistic composite image. We have divided the research works on image composition into object placement, image blending, image harmonization, shadow generation, based on the issues they aim to solve. In the future, we will extend the survey to more general composition in other fields such as video composition and 3D composition.