{'fraction_non_alphanumeric': 0.03204797998381043, 'fraction_numerical': 0.013135624402089926, 'mean_word_length': 4.675297556901232, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning to Deblur using Light Field Generated and Real Defocus Images

Defocus deblurring is a challenging task due to the spatially varying nature of defocus blur. While deep learning approach shows great promise in solving image restoration problems, defocus deblurring demands accurate training data that consists of all-in-focus and defocus image pairs, which is difficult to collect. Naive two-shot capturing cannot achieve pixel-wise correspondence between the defocused and all-in-focus image pairs. Synthetic aperture of light fields is suggested to be a more reliable way to generate accurate image pairs. However, the defocus blur generated from light field data is different from that of the images captured with a traditional digital camera. In this paper, we propose a novel deep defocus deblurring network that leverages the strength and overcomes the shortcoming of light fields. We first train the network on a light field-generated dataset for its highly accurate image correspondence. Then, we fine-tune the network using feature loss on another dataset collected by the two-shot method to alleviate the differences between the defocus blur exists in the two domains. This strategy is proved to be highly effective and able to achieve the state-of-the-art performance both quantitatively and qualitatively on multiple test sets. Extensive ablation studies have been conducted to analyze the effect of each network module to the final performance.


Introduction
The use of large camera aperture can increase the luminous flux so that the image can be captured with a shorter exposure time. However, this also reduces the depth of field (DOF) -only points near the focal plane can be captured sharply, while a point far from the focal plane will cast to the camera sensor, instead of a single image point, a spot called the Circle of Confusion (COC) [25] and result in defocus blur. Shallow DOF is sometimes an aesthetic effect pursued sedulously by the photographer [7,30], but it may also * denotes equal contribution and † denotes corresponding author.
degrade important visual information. Thus, restoring an all-in-focus image from its defocused version is highly demanded to reveal the latent information and benefit to artificial intelligence applications such as object detection [26] and text recognition [19]. Despite its great potential, defocus deblurring remains a challenging problem due to its spatially varying nature -every point has its own diameter of COC depending on the depth of the corresponding scene point. Besides, the shape of COC varies with respect to the relative position from the optical axis. To address defocus blur, the most intuitive way is to first estimate the blur kernel for each pixel, then apply non-blind deconvolution [5,12,15,17,24,32]. However, both steps have limitations. First, blur kernel estimation is not accurate and is often based on simple Gaussian [12,15,24] or disk kernel [5] assumption. Second, deconvolution tends to introduce ringing artifact on edges due to the Gibbs phenomenon [41] even if an accurate blur kernel is given. Recently, researchers have adopted end-to-end deep neural networks to directly restore sharp images from defocus blur [2,16,36], which largely outperform the conventional two-step approaches in terms of performance and efficiency. These networks are all trained on a dataset called Dual-Pixel Defocus Deblurring (DPDD) [2] which is captured sequentially with different aperture sizes to attain defocused and all-in-focus image pairs. However, it is hardly possible to capture defocused and all-in-focus pairs with accurate correspondence in two shots, especially for outdoor scenes due to moving objects (e.g., plants, cars) and illuminance variation. To this end, another dataset LFDOF [29] is built utilizing the benefit of light field refocusing and synthetic aper-ture to generate a large number of defocused images with a variety of DOFs and focal distances from a single light field sample. To examine the consistency among the image pairs, we select similar scenes from the two datasets and calculate the cross correlation between defocused and allin-focus pairs. As shown in Fig 1, LFDOF has strong cross correlation in the sharp regions, whereas DPDD does not hold consistence even at the sharp regions (the tree trunk is in focus in both defocused and all-in-focus images). However, despite good pixel-wise consistency of LFDOF, the defocus blur generated from light field data is not the same as that captured with a conventional digital camera (see Sec. 3). In this paper, we intend to make full use of the advantages of LFDOF and DPDD datasets to train a deep network for defocus deblurring. In summary, the contributions of this paper are as follows: • We analyze the characteristics of two defocus blur datasets LFDOF and DPDD and develop a novel training strategy for single image defocus deblurring. We also estimate and compare the Point Spread Function (PSF) of light field generated defocus blur against that captured with a conventional digital camera.
• We propose an end-to-end network architecture equipped with a novel dynamic residual block to reconstruct the sharp image in a coarse-to-fine manner.
• We conduct extensive experiments to evaluate the effect of each network module and demonstrate the stateof-the-art performance quantitatively and qualitatively on multiple test sets.

Related Work
Conventional Methods Conventional defocus deblurring methods typically follow a two-step approach consisting of defocus map (blur level for each pixel) estimation [12,37,46] followed by non-blind deconvolution [6,14,17]. Much effort has been made to improve the accuracy of defocus map as it influences the deblurring performance significantly [12,24,32,32]. However, this approach usually requires intensive computation while ending up with limited performance due to the defective intermediate defocus map. Defocus Blur Dataset There are several publicly available datasets for problems related to defocus blur. Defocus blur datasets comprising real defocused RGB images and binary masks are built by Shi et al. [32] and Zhao et al. [43]. But they can only work for blur detection due to the absence of all-in-focus images. Abuolaim and Brown [2] built the DPDD dataset using a dual-pixel camera and capturing the defocus and all-in-focus pairs in two successive shots. Ruan et al. [29] proposed the light field based defocus deblurring dataset LFDOF leveraging the synthetic aperture and refocusing features of light field technology [22], where the im-age pairs were acquired in single shots. Lee et al. [16] provided a benchmark test set consisting of 50 scenes captured with a dual-camera system with beam splitter. Adopting existing RGBD dataset [28,33] to generate defocus images based on the depth maps, Lee et al. [15] have simulated single defocused images while Abuolaim et al. [3] and Pan et al. [23] have simulated dual defocused pairs. However, the images in these datasets are synthetic, thus lacking realism.
In this paper, we use the DPDD and LFDOF datasets that are captured in real-world scenes. CNN-based Methods Abuolaim and Brown [2] adopted an U-Net-like architecture to restore the sharp image in an end-to-end manner and their follow-up work [1] incorporated a single-encoder multi-decoder architecture to further improve the performance. Lee et al. [16] proposed a network equipped with iterative filter adaptive module to tackle the spatially varying defocus blur and auxiliary reblurring module to enhance the restoration performance. Son et al. [36] proposed an effective way to simulate inverse kernels via kernel sharing parallel atrous convolution block. The aforementioned networks are all trained on DPDD dataset [2]. Ruan et al. [29] addressed single image defocus deblurring based on the conventional two-step strategy that takes the intermediate defocus map as the guidance for the deblurring step. Some other works used dual views to address the defocus deblurring problem [23,38]. However, we focus on single image defocus deblurring in this paper. Dynamic Filtering Dynamic filtering, also called filter adaptive convolution, has been successfully adopted in various low level vision tasks, such as denoising [21], video deblurring [44], super-resolution [10,13,40] and defocus deblurring [16], etc., since first introduced by Jia et al. [9]. It aims to learn per-pixel kernels instead of a single kernel over the entire image and thus is capable of handling nonuniform or spatially varying degradation. Our proposed dynamic residual block is partially inspired by dynamic filtering, but instead of directly applying convolution with the learned per-pixel kernels to get the output, we treat them as the learned dynamic residuals along with the direct residual for better performance. Also, being different from the method of Lee et al. [16] which transforms the features with the learned kernels to remove the spatially varying blur in the feature space, we learn the dynamic residual in the image domain in a coarse-to-fine manner to restore the sharp image. Section 5.3 demonstrates the superior performance of our proposed method against existing works in addressing single image defocus deblurring.

Conventional Digital Camera vs. Light Field Camera
We begin with describing the defocus blur discrepancy between the images captured with conventional digital cam-  eras (Canon EOS R5) and light field cameras (Lytro Illum) respectively as shown in Fig. 2. For a conventional digital camera (Fig. 2a), the rays emitted from a scene point on the focal plane converge to a single pixel of the image sensor by the main lens, while a point away from the focal plane projects to a patch of pixels on the sensor in a circular shape (COC), causing defocus blur. For a light field camera, a micro lens array is placed in front of the sensor, thus the rays coming from the main lens are re-distributed to the pixels under micro lenses, which means that each pixel does not only record the integrated illuminance but also the directional information of the rays. Each sub-aperture view of a light field only records a small part ( 1 14×14 for Lytro Illum) of the full aperture. After capturing, the aperture size can be further synthesized by integrating an appropriate subset of samples from multiple subaperture views. Similarly, refocusing to different depth can be achieved by integrating pixels along different directions on the epipolar plane image (EPI) [4] as shown in Fig. 2b.
To understand the differences between the defocus blur produced by these two types of cameras, we estimate and visualize their PSFs using the algorithm proposed by Mannan and Langer [20]. The PSFs of three typical cases: front focus, in focus, and back focus are estimated and shown alongside the imaging optical path diagram in Fig. 2. Generally, the PSFs produced by the digital camera follows the diffraction pattern of single Airy disk, while the PSFs produced by the light field camera resemble the patterns of multiple Airy disks, which can be explained by the synthetic nature of light field generated defocus blur. More PSF estimations can be found in the supplementary materials.

Methodology
Single image defocus deblurring aims to recover the latent sharp imageŷ from an observed input x that is distorted by defocus blur. A deep network can be trained as a mapping function F parameterized by θ: A loss function should be tailored to optimize θ in order to minimize the distance betweenŷ and y: where (x i , y i ) are defocused and all-in-focus image pairs. As shown in Fig. 3, we design our network F in an encoderdecoder [27] structure. The encoder (E) extracts the multiscale pyramidal features, which are then added to the corresponding scale of decoder (D) by skip connections to stabilize the network training. Two residual blocks [8] are added to each scale of decoder. We will show in Sec. 5.3 that simple encoder-decoder structure cannot handle defocus deblurring well. Dynamic Residual Block (DRB) Inspired by the dynamic filtering approach [9,10,16,34,40], we design a residual version of dynamic filtering block R to better handle the spatially varying defocus blur. We connect the dynamic residual block to each scale of decoder in a cascaded fashion to restore the latent sharp image progressively. Each dynamic residual block can be formulated as: wherex ↓n represents one of the input of R, which is also the ×2 up-sampled version of the output from the previous dynamic residual block:x ↓n = ↑ 2 (ŷ ↓ n 2 ). Note that the input of the first dynamic residual block is x ↓8 , which is a direct down-sample from the input image x. Specifically, as illustrated in the green inset in Fig. 3, the inputx ↓n is passed to three convolution layers and concatenated with the equivalent size feature map d ↓n from decoder D, then sent to two paths: one is to estimate the dynamic kernel volume K and another is to estimate the residual ∆r ↓n . The estimated dynamic kernel volume K is then convolved with the inputx ↓n to obtain the dynamic residual ∆x ↓n : The dynamic filtering procedure is depicted within the gray inset in Fig. 3. Finally the output of each dynamic residual  Figure 3. Our network architecture is mainly composed of encoder E, decoder D and four dynamic residual modules R. The LFDOF dataset is used for main training and the DPDD dataset is used for network fine-tuning.
block can be calculated by: We visualize a small patch in each step of our four dynamic residual blocks in Fig. 4, which clearly shows how the latent sharp image is reconstructed step by step from its defocused version. Specifically, the dynamic residual ∆x ↓n extracts the high frequency features like edges and corner points, which are lost during the defocus blur formation process, while the residual ∆r ↓n focus on the low frequency features that represent the essential content covered by defocus blur. The dynamic residual ∆x ↓n , residual ∆r ↓n and the input imagex ↓n jointly contribute to recovering the latent sharp image. Section 5.3 demonstrates the effectiveness of the proposed DRB.x ↓nx↓nŷ↓n ↓n ↓n Figure 4. Visualization of each step in DRB on all scales. From left to right: original input x and ground truth y, input of DRB x ↓n , dynamic kernel on two pixels k ↓n , dynamic residual ∆x ↓n , residual ∆r↓n, output of DRBŷ ↓n .
Training Strategy and Loss As demonstrated in Sec. 3, the defocus blur produced by a conventional digital camera and a light field camera are different. To remedy this gap, we propose a training strategy that leverages the strength and overcomes the shortcoming of light field data. Specifically, we apply the light field generated dataset LFDOF in the main training round for its highly accurate image correspondence. Then, we fine-tune the network using DPDD dataset to alleviate the differences between the two domains. Different losses are used in each stage. In the main training stage, ℓ 1 norm is used: During the fine-tuning stage, pixel-wise loss should be avoided because of the misalignment (induced from twoshot) between the defocused and all-in-focus pairs in DPDD dataset. We apply the VGG-based feature loss [11] in this step to transfer the learned knowledge to the target domain in the feature space, thus avoiding exact matching in the image space. We extract the feature maps from the 2nd, 7th, and 14th layer of the pre-trained VGG-19 network [35], denoted as ϕ.
Please note that the losses are applied to all scales. We will present in Sec. 5.3 that our training strategy contributes significantly to the final restoration performance.
elDP have no all-in-focus ground truth, as the former one is collected from the Internet while the latter is due to the fixed aperture of smartphone. We utilize LFDOF and DPDD for training and the remaining datasets for evaluation. Implementation We implement and evaluate our models using PyTorch with Tesla V100-32GB. We use the Rectified-Adam optimizer [18] with β 1 = 0.9 and β 2 = 0.99. The initial learning rate is set to 10 −4 when trained on LFDOF for 200 epochs and 10 −5 trained on DPDD for 100 epochs. The updating strategy of learning rate is similar to that of Zhu et al. [45], where the same learning rate is used for the first 100 epochs and then decayed linearly to zero over the rest 100. We set the batch size to 8 and patch size to 320 × 320 augmented with Gaussian noise, gray-scale image conversion and scaling.

Comparison to the State-of-the-Art Methods
Evaluation We compare our proposed method with the four latest learning-based single image defocus deblurring works: DPDNet S [2], AIFNet [29], IFANet [16], and KPAC [36]. All these networks are trained on the DPDD dataset [2] except AIFNet which is trained on LFDOF. Specifically, IFANet needs to incorporate dual views in their network training to estimate the disparity map, while others do not. We use the codes and pre-trained weights released by the authors for comparison, then further evaluate their performance using RealDOF dataset [16]. In Tab. 2, we report the quantitative result using three standard evaluation metrics, including Peak Signalto-Noise Ratio (PSNR), Structural Similarity (SSIM) [39] and Learned Perceptual Image Patch Similarity (LPIPS) [42]. We also list the parameter numbers for readers' reference. Our proposed method performs the best among all by a significant margin especially on the RealDOF test set. For instance, our network outperforms DPDNet S , AIFNet, IFANet and KPAC by 12.6% (2.88dB), 11.5% (2.65dB), 4.2% (1.04dB) and 7.3% (1.76dB) in terms of PSNR respectively. Please note that RealDOF is not used for training, thus it is a fairer benchmark test set for comparing network performance. Figure 5 shows the corresponding qualitative comparison. Although DPDNet S can reduce the defocus blur to some extent, it produces artifacts as shown in the second row in Fig. 5b ). AIFNet gives sharp details in some cases, for instance, the leaves in third row in Fig. 5c and the characters in fourth row, but it fails to remove defocus blur in other cases. This can be explained by their two-step network architecture, in which the final performance is partially determined by the intermediate defocus map. Thus, AIFNet fails to restore high quality details when the estimated defocus map is inaccurate. KPAC performs slightly better than DPDNet S , while in general it gives limited performance due to its small model capacity. Despite that IFAN shows competitive restoration performance, our proposed method performs better in restoring text (second and fourth row in Fig. 5f), texture (first row in Fig. 5f) and object boundary (third row in Fig. 5f). It is worth noting that IFAN needs dual views for network training while ours only needs single view. More results regarding the visual comparison, model complexity, computational costs, etc. are presented in the supplementary material. Generalization Ability To inspect the generalization ability of our network, we further compare the visual performance of the networks using CUHK [31] and PixelDP [2] datasets. CUHK targets for blur detection with relatively small spatial resolution, and all images are collected from the Internet thus no all-in-focus ground truth are provided. PixelDP is collected with Google Pixel 4 smartphone which has a fixed aperture size and the image data is limited to one of the Green channels in the ray-Bayer frame. Figure 6 reports the visual comparison among the five networks and a similar conclusion can be drawn: our method gives the best generalization ability and can successfully restore the fine details regardless of the camera type. More results are provided in the supplementary material.

Analysis and Discussion
In this section, we conduct comprehensive ablation studies and analysis. Why LFDOF? To understand the necessity of LFDOF for defocus deblurring, we train our network on LFDOF only, DPDD only and both. Here, we use 'LFDOF & DPDD' to represent the network trained on LFDOF then fine-tuned on DPDD for convenience. Table 3 and Fig. 7 show the quantitative and qualitative results evaluated using RealDOF test set. It is observed that our network trained on DPDD gives better scores in terms of all metrics than LFDOF. This can be explained by the domain difference between the light field-generated and real defocused images. However, the visual performance is not in line with the quantitative performance. The network trained on LFDOF produces sharper content and details than the one trained on DPDD does owing to the accurate pixel correspondence of LFDOF. However, at the same time, it also introduces artifacts (see wall in Fig. 7) due to the defocus blur discrepancy between light field generated and real data. Our strategy to train the network on LFDOF then fine-tune on DPDD largely outperforms the networks trained on either dataset alone and generates the best quantitative (increase up to 11 Figure 5. Qualitative evaluation on DPDD and RealDOF datasets among DPDNetS [2], AIFNet [29], IFANet [16], KPAC [36] and ours. 4.23% in terms of PSNR) and qualitative results.  Performance Gain To further validate the proposed training strategy, we have retrained two state-of-the-art networks, DPDNet S [2] and KPAC [36], to see whether our training scheme could improve the performance of these two networks. AIFNet and IFANet are not listed here because the former requires defocus map as the ground truth and the latter requires dual views for training. For KPAC, we choose the 3-level model with 2 KPAC blocks for comparison. Except the loss and learning rate which are set to be the same as ours in the two training stages, other parameter settings for DPDNet S and KPAC are the same as their original configurations. The network performance are evaluated on 76 test samples in DPDD dataset. Table 4 shows that DPDNet S , KPAC and our proposed architecture have gained 0.511dB, 0.249dB, and 0.253dB in terms of PSNR when networks are trained on LFDOF & DPDD. The visual quality has also been largely improved as shown in Fig. 8.
Loss Our proposed method trains the network on LFDOF with L 1 loss, then on DPDD with feature loss. One may wonder if it is possible to (1) directly mix the datasets for training instead of using a two-stage training strategy, and (2) adopt the same L 1 loss in the fine-tuning stage. To an-  Figure 6. Qualitative comparison among DPDNetS [2], AIFNet [29], IFAN [16], KPAC [36] and ours. Image in the first row is from CUHK [31] and that in the second row is from PixelDP [2]. No all-in-focus ground truth is provided in these two datasets.    swer these questions, we ablate the use of loss, then evaluate their performance on DPDD and RealDOF datasets as shown in Tab. 5 and Fig. 9. To balance the ratio of the two datasets, we augment DPDD dataset (350 images) for 32 times, then mix with LFDOF (11261 images). The network performs similarly when using L 1 loss alone and L 1 combined with L VGG loss on the mixed dataset, while performing slightly worse than the version trained on DPDD only (see Tab. 3 and Tab. 4). When adopting the same L 1 loss in two stages, the quantitative result in terms of PSNR and SSIM is comparable with our final one, while the perceptual score LPIPS is relatively worse on DPDD dataset. It is because the per-pixel loss on DPDD may not lead to an optimal performance, even the misalignment is not obvious for human perception (e.g., Fig. 1) but sensitive to the network. In addition, its performance on RealDOF is less favorable with 0.62dB lower than our final one in terms of PSNR. In Fig. 9, the results in red frames are produced by our final model, which yield the most realistic and fine details. Both quantitative and qualitative performance further validate and support the proposed training strategy.  Table 5. Quantitative comparison of the training strategy with respect to dataset and loss. S1 and S2 represent the main training on LFDOF and fine-tune on DPDD respectively. There is only one stage when training on the mixed dataset and λ is set to 10 −5 .

DRB Configurations
To validate the effect of each component in DRB, we conduct an ablation study and report the results in Tab. 6  puts the restored result without DRB. Those variants are all with multi-scale architecture. Another variation with one full DRB (remove R ↓ {8,4,2} thus leaving R only) is also added for comparison. Both quantitative and qualitative result show the final model is able to restore the finest details as shown in Tab. 6 and Fig. 10. For visual quality, Figure  10 demonstrates that only the DRB with all the components is capable of restoring realistic details (see last two in the second row).   Figure 10. Qualitative results of the ablation study of DRB configurations evaluated on RealDOF.
Dynamic Kernel Size We analyze the effect of dynamic kernel size in our restoration performance. As illustrated in Tab. 7, we experimentally identify the best performing kernel size for DPDD dataset to be 9 while that for RealDOF to be 7. Considering the overall performance and parameter number, kernel size 7 is chosen.  Table 7. Quantitative evaluation on DPDD and RealDOF datasets with respect to the kernel size in DRB block.
AIFNet vs. Ours Both AIFNet and ours are trained on LFDOF dataset. However, AIFNet employs defocus map estimation network followed by deblurring network while our network is an end-to-end architecture without explicit defocus map estimation. In order to compare these two networks, we train and test their performance only on LFDOF. Table 8 shows that our network architecture outperforms AIFNet by 0.726dB in terms of PSNR without the help of defocus map. Accurate defocus map may not be able to boost the restoration performance, whereas inaccurate ones will limit or impede the performance. This further validates the effectiveness of our proposed network architecture.

Conclusion
We have proposed a novel method drawing on the synthetic aperture and refocusing features of light fields along with real captured defocus blur dataset to address the single image defocus deblurring problem. Our end-to-end neural network equipped with dynamic residual block is proven to be effective for removing spatially varying defocus blur. We train our network on light field generated dataset with MAE loss for the superior pixel-wise correspondence, then on real defocus dataset with feature loss to fully utilize the advantages of two types of data. We have proved this training strategy can be applied to improve the performance of several existing learning methods. Extensive comparison and ablation studies have demonstrated the effectiveness of our method, which outperforms others by a significant margin on multiple test sets. Limitations Despite showing competitive performance, our proposed method shares some similar limitations with Lee et al. [16] and Son et al. [36] in handling blur with irregular shapes and defocus blur mixed with object motion. We include these failure cases in the supplementary material. Our future work will take these challenging cases into consideration.