{'fraction_non_alphanumeric': 0.02768437570051558, 'fraction_numerical': 0.019054023761488455, 'mean_word_length': 4.652518213493823, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Depth-aware Blending of Smoothed Images for Bokeh Effect Generation

Bokeh effect is used in photography to capture images where the closer objects look sharp and every-thing else stays out-of-focus. Bokeh photos are generally captured using Single Lens Reflex cameras using shallow depth-of-field. Most of the modern smartphones can take bokeh images by leveraging dual rear cameras or a good auto-focus hardware. However, for smartphones with single-rear camera without a good auto-focus hardware, we have to rely on software to generate bokeh images. This kind of system is also useful to generate bokeh effect in already captured images. In this paper, an end-to-end deep learning framework is proposed to generate high-quality bokeh effect from images. The original image and different versions of smoothed images are blended to generate Bokeh effect with the help of a monocular depth estimation network. The proposed approach is compared against a saliency detection based baseline and a number of approaches proposed in AIM 2019 Challenge on Bokeh Effect Synthesis. Extensive experiments are shown in order to understand different parts of the proposed algorithm. The network is lightweight and can process an HD image in 0.03 seconds. This approach ranked second in AIM 2019 Bokeh effect challenge-Perceptual Track.


Introduction
Depth-of-field effect or Bokeh effect is often used in photography to generate aesthetic pictures. Bokeh images basically focus on a certain subject and out-of-focus regions are blurred. Bokeh images can be captured in Single Lens Reflex cameras using high aperture. In contrast, most smartphone cameras have small fixed-sized apertures that can not capture bokeh images. Many smartphone cameras with dual rear cameras can synthesize bokeh effect. Two images are captured from the cameras and stereo matching algorithms are used to compute depth maps and using this depth map, depth-of-field effect is generated. Smartphones with good auto-focus hardware e.g. iPhone7+ can generate depth maps which helps in rendering Bokeh images. However, smartphones with single camera that don't have a good auto-focus sensor have to rely on software to synthesize bokeh effect.
Also, already captured images can be post-processed to have Bokeh effect by using this kind of software. That is why generation of synthetic depth-of-field or Bokeh effect is an impor-tant problem in Computer Vision and has gained attention recently. Most of the existing approaches (Shen et al., 2016;Wadhwa et al., 2018;Xu et al., 2018) work on human portraits by leveraging image segmentation and depth estimation. However, not many approaches have been proposed for bokeh effect generation for images in the wild. Recently, Purohit et al. (2019) proposed an end-to-end network to generate bokeh effect on random images by leveraging monocular depth estimation and saliency detection.
In this paper, one such algorithm is proposed that can generate Bokeh effect from diverse images. The proposed approach relies on a depth-estimation network to generate weight maps that blend the input image and different smoothed versions of the input image. The generated bokeh images by this algorithm are visually pleasing. The proposed approach ranked 2nd in AIM 2019 challenge on Bokeh effect Synthesis-Perceptual Track (Ignatov et al., 2019).

Monocular Depth Estimation
Depth estimation from a single RGB image is a significant problem in Computer Vision with a long range of applications arXiv:2005.14214v1 [cs.CV] 28 May 2020 including robotics, augmented reality and autonomous driving. Recent advances in deep learning have helped in the progress of monocular depth estimation algorithms. Supervised algorithms rely on ground truth Depth data captured from depth sensors. Luo et al. (2018) formulated monocular depth estimation as a combination of two sub-problems: view synthesis and stereo matching. View synthesis network creates another view of the given RGB image which is utilized by the stereo matching network to estimate the depth map. Chen et al. (2016) train a multiscale deep network that predicts pixelwise metric depth with relative-depth annotations from images in the wild. Atapour-Abarghouei and Breckon (2018) trains a depth estimation network on large amount of synthetic data and use image style transfer to map real world images to synthetic images. Godard et al. (2017) poses Monocular Depth Estimation as an image reconstruction problem. They train a Convolutional Neural Network on easy-to-obtain binocular stereo data rather than training on depth ground truth and introduce a novel loss function that checks left-right consistency in generated disparity maps. Godard et al. (2019) introduces a minimum reprojection loss, an auto masking loss and a full-resolution multi-scale sampling method to train a deep network in self-supervised way. Their minimum reprojection loss handles occlusion in a monocular video and auto-masking loss helps network to ignore confusing and stationary pixels. Li and Snavely (2018) created a training dataset using Structure from Motion and Multi-view Stereo methods. They trained three different networks using scaleinvariant reconstruction and gradient matching loss and ordinal depth loss which achieves good generalization performance.

Depth Effect in Images
Early work in Depth-of-field generation in images uses automatic portrait segmentation. Shen et al. (2016) used a FCN (Long et al., 2015) network to segment the foreground in the images and use the foreground for different kind of background editing such as background replacement, BW one color and depth-of-field effect. Wadhwa et al. (2018) uses a neural network to segment person and its accessories in an image to generate a foreground mask in the case of person images. Then they generate dense depth maps using a sensor with dual-pixel auto-focus hardware and use this depth map along with foreground mask (if available) for depth-dependent rendering of shallow depth-of-field images. Xu et al. (2018) too uses both learning based and traditional techniques to synthesize depthof-field effect. They use off-the-shelf networks for portrait segmentation and single image depth estimation and use SPN (Liu et al., 2017) to refine the estimated segmentation and depth maps. They split the background into different layers of depth using CRF (Zheng et al., 2015) and render the bokeh image. For fast rendering, they also learn a spatially varying RNN filter (Liu et al., 2016)) with CRF-rendered bokeh result as ground truth. Purohit et al. (2019) proposed an end-to-end network for Bokeh Image Synthesis. First, they compute depth map and saliency map using pretrained monocular depth estimation and saliency detection networks respectively. These maps and the input image are fed to a densely-connected hourglass architecture with pyramid pooling module. This network generates spatially-aware dynamic filters to synthesize corresponding bokeh image.
In this paper, an end-to-end deep learning approach is proposed for Bokeh effect rendering. In this approach, the corresponding bokeh image is thought to be a weighted sum of different smoothed versions of the input image. The spatially varying weights are learned with the help of a pretrained monocular depth estimation network by modifying the last convolutional layer. Compared to some other methods in the literature, this  approach doesn't rely on foreground or salient region segmentation, which makes the proposed algorithm lightweight. The proposed method generates good bokeh effect rendering both qualitatively and quantitatively. This approach came 2nd in Perceptual track of AIM 2019 challenge on Bokeh effect synthesis.

Depth Estimation Network
Megadepth (Li and Snavely, 2018) is used as Monocular Depth estimation network in this work. The authors use an hourglass architecture which was originally proposed in Chen et al. (2016). The architecture is shown in Figure-2. The encoder part of this network consists of a series of convolutional modules (which is a variant of inception module) and downsampling. In the decoder part, there is a series of convolutional modules and upsampling with skip connections that add back features from higher resolution in between.
Megadepth was trained using a large dataset collected from internet using structure-from-motion and multi-view stereo methods. Megadepth works quite well in a wide variety of datasets such as KITTI (Geiger et al., 2012), Make3D (Saxena et al., 2006) and DIW (Chen et al., 2016). This generalization ability of Megadepth makes it suitable for bokeh effect rendering for diverse scenes. Figure-3 shows predicted depth maps by Megadepth and other state-of-the-art models. We can see that Megadepth produces better quality depth estimation compared to other models.

Bokeh effect rendering
The depth-of-field image can be formulated as a weighted sum of the original image and differently smoothed versions of original image. These different smoothed images can be generated using different size of blur kernels. Concretely, intensity value of each pixel position in the bokeh image can be thought of as a weighted sum of intensity values from original image and smoothed images in that pixel position. Hence, the generated bokeh image is given by, where I org is the original image, B(I org , k i ) is image I org smoothed by blur kernel of size k i × k i and stands for elementwise multiplication, such that for each pixel position (x, y), In the proposed approach, the weights W i are predicted with the help of a neural network. This can be achieved by modifying the last layer of the pretrained depth estimation network. Specifically, in the last layer a convolutional layer with 3 × 3 kernel is used along with spatial softmax activation to learn the weights. The proposed approach is summarized in Figure-1.
In the experiments, n is chosen to be 3 i.e. three different smoothed images are used. The smoothed images were obtained by applying Gaussian Blur with Blur kernels of size 25 × 25, 45 × 45 and 75 × 75. The effect of using less number of kernels and kernels of different size are discussed in the following section.

System Configuration
The codes were written in Python and Pytorch( (Paszke et al., 2019)) is used as the deep learning framework. The models were trained on a machine with Intel Xeon 2.40 GHz processor, 64 GB RAM and NVIDIA GeForce TITAN X GPU card with approximately 12GB of GPU memory.

Dataset Description
We use ETH Zurich Bokeh dataset (Ignatov et al., 2019), which was used in AIM 2029 Bokeh Effect Synthesis Chal-lenge. This dataset contains 4893 pairs of bokeh and bokehfree images. Training set contains 4493 pairs whereas Validation and Testing datasets contain 200 pairs each. Validation and Testing Bokeh images are not publicly available at this moment. This dataset contains a variety of outdoor scenes. The normal images were captured using narrow aperture ( f /16) and a high aperture ( f /1.8) was used to generate the Bokeh effect in images. The images in this dataset have resolution of around 1024×1536 pixels. A snapshot of this dataset is given in Figure-

4.
The training set of ETH Zurich Bokeh Dataset is divided to a train set containing 4400 pairs and a validation set containing 294 pairs for experiments in this paper. This validation set is denoted as Val294.

Loss functions
Reconstruction loss: l 1 loss is used as reconstruction loss to guide the network to generate images with pixel values close to ground truth. Reconstruction loss is given by, Perceptual loss: Negative SSIM (Wang et al., 2004) loss is used to improve perceptual quality of the generated images. Perceptual loss is given by,

Training Strategy
The model is trained in 3 phases. In phase-1, the model is trained on smaller resolution images to save training time. Each image from training set is resized to a resolution of 384 × 512 and the network is trained using reconstruction loss. In phase-2, the network is trained using images of resolution 768 × 1024 with reconstruction loss. In phase-3, the model is further finetuned with perceptual loss l p .

Other training details
Adam optimizer( (Kingma and Ba, 2014)) is used to train the network. The values of β 1 and β 2 are chosen to be 0.9 and 0.999 respectively. The initial learning rate is set to be 1e-3 and gradually decreased to 1e-5. Each image in the training set is horizontally and vertically flipped to augment the dataset.

Testing Strategy
During testing, the input image is first resized to the original dimension in which the network was trained(384 × 512 for Phase-1 and 768 × 1024 for Phase-2 and Phase-3) and then passed to the network. The synthesized image is then scaled back to the input image resolution using bilinear interpolation.

Evaluation metrics
Both fidelity and perceptual metrics are used to evaluate the model's performance. Fidelity measures include Peak Signal-to-Noise Ratio(PSNR) and Structural Similarity Index(SSIM) (Wang et al., 2004). LPIPS  is used to measure perceptual quality of the generated results.

Quantitative and Qualitative Results
Baseline. Bokeh effect can be generated by simply segmenting the foreground in the image and blurring the rest of the image. One such system is used as baseline in this paper. A stateof-the-art saliency detection method, Stacked Cross Refinement Network(SCRN) (Wu et al., 2019) is used for segmenting the foreground in the input image. The background is blurred using 75 × 75 Gaussian kernel. As SCRN generates soft saliency maps, bokeh images are generated using Equ-1. Table-5. We can see that the proposed approach performs significantly better than the baseline on all three metrics. Figure-5 shows saliency maps generated by SCRN and corresponding bokeh effect rendering by baseline and the proposed model. We can see that the performance of the baseline is limited to accuracy of saliency detection which leads to artifacts near edges and incorrect blurring of foreground in rendered images.

Visualization
Examples of generated bokeh images and the corresponding predicted weight maps by our network are shown in Figure-6. We can see that, image regions that are closer have higher weights in W 0 and W 1 compared to W 2 and W 3 . This allows the model to generate blur effect that is varying with respect to depth i.e. closer objects look sharp and distant regions get more blurred with increasing depth in the rendered bokeh image.

Ablation Study
Less number of kernels vs. more number of kernels. The main intuition behind using a number of kernels is to try to synthesize bokeh images in a way that regions that are closer to the camera have no or less blur effect whereas distant objects have more blur effect. Using less number of kernels will guide network to learn lesser number of such blur levels in the image. Using more number of blur kernels helps the network two generate smoother background in synthesized bokeh images. Table-2 shows quantitative comparison between using just one blur kernel of size 75 and the proposed approach. We can observe in Figure-7 using more number of kernels produces more visually pleasing background.
Smaller kernels vs bigger kernels. To demonstrate the effect of using bigger kernels, two different sets of blur kernels were used. In one experiment, blur kernel sizes are respectively 5, 25 and 45 and in another experiment blur kernels of size 25, 45 and 75 are used. Table-3 shows that using bigger kernels yields better performance. Qualitative comparison be-  tween these two settings are shown in Figure-8. By analysing weight maps W 1 and W 0 in both settings, we observe that the value of W 0 is more in the in-focus region in case of bigger kernels than of smaller kernels. This helps network that uses bigger blur kernels to generate bokeh images that preserve more details in closer objects.
Effect of three-phase training. The model is trained in three phases. As we can see in Table-4, results from Phase-1 scores the highest PSNR among all the phases, whereas Phase-3 produces best results with respect to SSIM and LPIPS. We observe in Figure-9 that Phase-2 and Phase-3 results preserve more details in the closer object than Phase-1 as Phase-2 and 3 models are trained on higher resolution images. Also, through fine-tuning with SSIM loss in Phase-3, the model generates a smoother blur effect in the background, as shown in Figure-9.

AIM 2019 Challenge on Bokeh Effect Synthesis
The proposed solution participated in AIM 2019 Challenge on Bokeh Effect Synthesis which is a competition on example-

Efficiency
The proposed model is lightweight. It takes 28 MB to store the model on hard drive. The model can process an HD image of resolution 1024×1575 in 0.03 seconds (I/O time is excluded).

Conclusion
In this paper, an end-to-end deep learning approach for Bokeh effect synthesis is proposed. The synthesized bokeh image is rendered as a weighted sum of the input image and a number of differently smoothed images, where the corresponding weight maps are predicted by a depth-estimation network.
The proposed system is trained in three phases to synthesize realistic bokeh images. It is shown through experiments that using more number of blur kernels and bigger blur kernels produce better quality bokeh images. The proposed algorithm is lightweight and can post-process an HD image in 0.03 seconds.
The proposed approach has been ranked 2nd among the solutions proposed in Perceptual Track of AIM 2019 challenge on Bokeh effect synthesis. In future, the effect of using different kinds of blur kernel (e.g. average blur, disk blur) can be explored. It is also interesting to see how one can incorporate both monocular depth estimation and saliency detection to produce a lightweight system that generates high-quality bokeh effect.