{'fraction_non_alphanumeric': 0.03934519370760981, 'fraction_numerical': 0.012179965200099428, 'mean_word_length': 4.574425969912906, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Gaussian Kernel Mixture Network for Single Image Defocus Deblurring

Defocus blur is one kind of blur effects often seen in images, which is challenging to remove due to its spatially variant amount. This paper presents an end-to-end deep learning approach for removing defocus blur from a single image, so as to have an all-in-focus image for consequent vision tasks. First, a pixel-wise Gaussian kernel mixture (GKM) model is proposed for representing spatially variant defocus blur kernels in an efficient linear parametric form, with higher accuracy than existing models. Then, a deep neural network called GKMNet is developed by unrolling a fixed-point iteration of the GKM-based deblurring. The GKMNet is built on a lightweight scale-recurrent architecture, with a scale-recurrent attention module for estimating the mixing coefficients in GKM for defocus deblurring. Extensive experiments show that the GKMNet not only noticeably outperforms existing defocus deblurring methods, but also has its advantages in terms of model complexity and computational efficiency.


Introduction
The appearance sharpness of an object in an image taken by a camera is determined by the scene distance of the object to the focal plane of the camera. An object will have the sharpest appearance when it is on the focal plane, i.e., the object is in focus. The area around the focal plane where objects appear to be in focus is called the depth of field (DoF). When an object is away from the DoF, it will appear blurry. The further is an object away from the DoF, the more blurry it appears. Such a phenomenon is called defocus blur or out-of-focus blur. Defocus blur effects will be prominent in an image with a shallow DoF, e.g. images captured with a large aperture. This paper concerns the problem of single image defocus deblurring (SIDD) which is about reconstructing an all-in-focus image from a defocused image (i.e. an image with defocused regions). SIDD is of practical values to many applications in machine vision, e.g., photo refocusing, object recognition, and many others [1].
Consider a defocused image y, which relates to its all-in-focus counterpart x by where denotes the measurement noise, and B is a linear operator defined by Each pixel at location [m, n] is associated with a defocus kernel b m,n , also referred to as point spread function (PSF), which is determined by the distance to the focal plane. Often, these pixel-wise PSFs are approximated by Gaussian kernels [2,3,4,5,6] or disk kernels [7,8]. Without supplementary information on the scene depth, these pixel-wise PSFs are unknown. Therefore, SIDD is a challenging nonlinear inverse problem which needs to estimate both B and x from (1).
Deep learning has become one prominent tool for solving a wide range of image restoration problems. In comparison to the rapid progress of DNNs for spatially-varying motion deblurring (e.g. [12,13,14,15,16,17]), there have been few works on studying DNN-based approaches to defocus deblurring. One might directly adapt an existing motion deblurring DNN for SIDD. However, the kernels of defocus blur are very different from those of motion blur, e.g., roughly isotropic support vs highly curvy support. Also, the spatial variation in defocus blur differs much from that in motion blur, e.g., transparency effects of moving objects in motion blur do not exist for out-of-focus objects in defocus blur. As a result, it is sub-optimal to directly call a motion deblurring method for SIDD.
Another straightforward implementation of introducing deep learning to SIDD is to replace the defocus map estimator in a traditional two-stage approach by a DNN-based method (e.g. [6]). Such an implementation still suffers from the issues existing in traditional methods, i.e. inaccurate estimation of B from a non-perfect defocus map and high computational cost for deblurring with a spatially-variant blurring operator. To fully exploit the potential of deep learning for SIDD, one needs to specifically design an end-to-end DNN that directly predicts the all-in-focus image from the defocused one. Recently, Abuolaim and Brown [18] developed an end-to-end DNN for constructing an all-in-focus image from a pair of images containing two sub-aperture views of the same scene. They also adapted their DNN to SIDD, but saw a significant performance decrease.

Main Idea
This paper aims at developing an end-to-end DNN for SIDD with better performance than existing methods, which is based on the following two derivations.
GKM-based model for defocus blurring Since defocus PSFs show strong isotropy and smoothness, we propose to model the kernels {b m,n } m,n by Gaussian kernel mixture (GKM): where g(σ) denotes the 2D Gaussian kernel of variance σ 2 , and β k denotes the matrix of mixing coefficients for the k-th Gaussian kernel in the GKM. As the GKM can fit well most isotropic kernels, Eq. (3) is a more accurate model for real defocus PSFs than the often-used single Gaussian/disk form; see supplementary materials for a demonstration. Remark 1. The GKM degenerates to the single Gaussian form when only one mixing coefficient is 1 and the others are 0 for every location [m, n]. In general cases, the weighted summation of Gaussian kernels can represent non-Gaussian kernels, and thus the GKM can express a wider family of defocus PSFs than the single Gaussian form. There is also another work [9] that models defocus PSFs beyond single Gaussian kernels. It uses the generalized Gaussian function [9] where the parameters to be estimated are wrapped in a complex nonlinear function. In comparison, our GKM model is linear with pre-defined {σ k } k , which facilitates the estimation on its parameter {β k } k .
We can rewrite (1) as where δ m,n denotes the Dirac delta centered at location [m, n], and ⊗, denote the operations of 2D convolution and entry-wise multiplication, respectively. Then we have the GKM-based model for defocus blurring: Fixed-point iteration unrolling Recall that the blurring operator B is about keeping the low-frequency components and attenuating high-frequency ones of an image. Let I denote the identity mapping. The mapping I − B is then about attenuating the low-frequency components and keeping the high-frequency ones. Neglecting the noise and rewriting (1) by we have then a fixed-point iteration for solving defocus deblurring, which is given by Note that the fixed-point iteration above will be convergent if I − B is a contractive mapping, or equivalently the eigenvalues of B fall in (0, 1), which holds true when the defocus blurring is uniform with a normalized Gaussian kernel, i.e., the scene depths are constant in the view.
Define σ 1 = 0 and g(σ 1 ) = δ, so that clear regions can be modeled by setting β 1 = 1 and zeroing β k for k > 1. Let γ 1 = 1 − β 1 and γ k = −β k for k > 1. The iteration (6) can be expressed as In short, based on the GKM model of defocus PSFs, we can unroll a fixed-point iteration to solve (1) with learnable coefficient matrices γ 1 , · · · , γ K . The motivation of unrolling a fixed-point iteration, instead of other iterative schemes such as gradient descent [19] and half quadratic splitting (HQS) [20], is to involve the forward operator B only, without introducing the transpose B and the pseudo-inverse B † . Remark 2. Our approach is sort of in the category of optimization unrolling, a widely-used methodology of designing DNNs for solving inverse problems. The key is to choose an appropriate iteration scheme that fits the problem well.
Most existing optimization unrolling based image deblurring methods (e.g. [21,22,20,23,24,25,26,27]) consider uniform blurring, where the matrix B can be represented by a convolution. The iterative schemes they adopt such as HQS, usually involve an inversion process for B, which can be efficiently computed using FFT when B is a convolution operator. In our case, B is a spatially-varying blurring operator which does not have a computationally efficient inversion process. The proposed fixed-point iteration unrolling enables us to avoid such an inversion process in the DNN and use the forward operator only.
The matrices γ 1 , · · · , γ K of mixing coefficients can be intezpreted as the attention maps associated to the feature maps generated by different Gaussian kernels. Thus, we construct a DNN with attention modules and long skip connections to utilize (7) for SIDD. In addition, we take a multi-scale scheme to implement the unrolling: at each iteration the DNN predicts the all-in-focus image at current scale and up-samples it for the calculation of the next iteration, with weight sharing used across scales. This leads to a scale-recurrent attentive DNN with a lightweight implementation.

Main Contributions
In comparison to existing two-stage or dual-view-based methods, this paper is among the first ones to present an end-to-end DNN for SIDD. See below for the summary of our technical contributions: • A new and efficient parametric model based on GKM for defocus blur kernels, which fits real-world data better than existing models and thus leads to better performance in SIDD; • A new formulation of the deblurring process derived from a fixed-point iteration so as to have a simple and efficient parameterization of defocus deblurring, which inspires an effective DNN for SIDD with low model complexity and high computational efficiency; • A scale-recurrent attention mechanism which combines the coarse-to-fine progressive estimation and the unrolled deblurring process for better performance.
The experiments show that the proposed DNN brings noticeable improvement over existing approaches to SIDD, in terms of recovery quality, model complexity and computational efficiency.

Related Work
Two-stage SIDD Most studies of two-stage methods for SIDD are concentrated on the first stage, i.e. defocus map estimation, while the second stage is often done by calling existing non-blind deconvolution methods (e.g. [28,29,11]).
Defocus map estimation itself is a challenging task. There are several non-learning-based methods [2,3,5,8] available for sparse defocus map estimation on edge points or regions. The dense defocus map is then constructed via some propagation method (e.g. Poisson matting [30]). Recently, deep learning has been extensively studied for defocus map estimation; see e.g. [7,4,6,31]. As discussed in Section 1.1, the two-stage methods suffer from the sensitivity to estimation errors and high computational costs. In comparison, the proposed end-to-end DNN does not involve the defocus map estimation and thus does not suffer from these issues.
End-to-end learning for defocus deblurring There are few works on learning an end-to-end DNN for defocus deblurring. Abuolaim and Brown [18] proposed to train an end-to-end U-Net for predicting an all-in-focus image from two view images captured by a dual-pixel sensor, and contributed a dataset of quadruples: a defocus blurred image, its all-in-focus counterpart, and two dual-pixel view images. Such a dual-pixel-based DNN showed impressive performance. However, its performance significantly decreases when being used for SIDD where only a single image is available for input. The prerequisite on dual-view inputs also limits the wider applicability of this method. In contrast, the proposed DNN is grounded by the defocus blurring model and only assumes a single image as input, thus being applicable to commodity cameras. Very recently, one parallel work to ours on end-to-end learning of SIDD was done by Lee et al. [32]. They proposed a DNN that predicts the pixel-wise filters for deblurring the deep defocused features of an image. In comparison, our DNN predicts the blurring filters for deblurring the image in an unrolled fixed-point iteration framework.
DNNs for spatially-varying motion deblurring There have been many studies on deep learning for spatially-variant motion deblurring, especially on dynamic scenes with moving objects; see e.g. [12,14,13,16,15,33,34,35,36,37,17]. The methods using optical flow (e.g. [36]) or temporal cues of moving objects (e.g. [37]) for training are not applicable to SIDD. Many of these methods also adopt multi-scale structures (e.g. [13]) or attention mechanisms (e.g. [17]), but with generic designs which cannot effectively exploit the inherent characteristics of defocus blurring, e.g. strong isotropy and high correlation of the shapes of pixel-wise defocus PSFs. In comparison, our DNN is specifically designed for defocus deblurring and thus enjoys better performance in SIDD.
Unrolling-based deep learning for image deblurring Unrolling-based DNNs have been extensively studied for non-blind image deblurring where the PSF is given as an input; see e.g. [20,23,26]. There are also some works [21,22,24,27] on unrolling-based blind image deblurring where the PSF is unknown. However, these methods are restricted to the case of uniform blurring. The proposed unrolling-based DNN is the first one that can handle spatially-varying defocus blurring without given the PSFs, thanks to the proposed GKM-based defocus blurring model.

Network Architecture
The DNN we construct for SIDD, named as GKMNet (Gaussian Kernel Mixture Network), is based on the fix-point iteration (7) with a multi-scale recurrent fashion. See Figure 1 for the outline.
Given an input image y, we generate its multi-scale versions y 1 , · · · , y T via bi-linear downsampling with factors 2 T −1 , · · · , 2 0 , respectively. Let x 1 = y 1 and let ↑ 2 denote the upsampling operation by factor 2. We consider the following multi-scale extension of (7): The GKMNet employs T recurrent blocks to implement (8). The t-th block takes y t and x t ↑ 2 as input and outputs x t+1 . There are two modules in each block: a Gaussian convolution module (GCM) and a scale-recurrent attention module (SRAM). The GCM performs the Gaussian filtering in (8) to provide K feature maps, denoted by {z t,1 , . . . , z t,K }, for the t-th block. The SRAM generates the corresponding coefficient maps {γ t,1 , . . . , γ t,K } from y t . The output of the t-th block is given by where the weights {ω t,k } t,k are for scaling the mixing coefficients {γ t,k } t,k predicted by the SRAM within a certain range. The weighted summation with {ω t,k } t,k is implemented by 1 × 1 convolution. The output of the T -th block, i.e. x T , is used as the final deblurring result. Note that the GKMNet relates the scales not only by passing the output from one scale to the next, but also by the recurrence mechanism built in the SRAM. Let x gt t denotes the downsampled ground truth of the same size as x t . The training loss is defined by the supervision at all scales: C(x t , x gt t ), for some cost function C(·, ·), where we assign equal weights to the losses at different scales for simplicity.  Note that in addition to x t , we also input y t in parallel for feeding more information to the next stage.

Scale-Recurrent Attention Module
The SRAM maps an input image y t to the coefficient maps {γ t,k } K k=1 . The coefficient maps can be viewed as the spatial-channel attention maps associated to the feature maps generated by different Gaussian kernels. We thus draw inspirations from existing attention modules (e.g. [13,38,17]) to have a lightweight design on SRAM. As illustrated in Figure 2, the SRAM consists of (i) an attentive encoder-decoder backbone [39] for feature extraction; and (ii) an attention prediction unit (APU) based on Conv-LSTM (convolutional long short-term memory) [40].
The attentive encoder-decoder backbone sequentially connects a convolutional layer, two encoder blocks, and two decoder blocks. Each encoder/decoder block contains one convolutional layer with downsampling/upsampling, two convolutional layers with residual connections, and a triplet attention block [41]. The triplet attention is used to improve the NN's spatial and channel adaptivity for better prediction, which can be viewed as an attention-in-attention mechanism in the SRAM. Given a feature tensor X ∈ R C×H×W defined in the channel-height-width space, the triplet attention block generates three parallel attention maps a W ∈ R C×H , a H ∈ R C×W , a C ∈ R H×W from the channel-height, channel-width and height-width slices of X respectively, and then applies them for re-calibrating X so as to encode spatial-channel dependencies into the features.
The APU contains two paths. The first path mainly contains a Conv-LSTM and applies downsampling/upsampling before/after it for inducing local smoothness on the predicted coefficient maps. The second path contains two convolutional layers, but without downsampling/upsampling for preserving detailed information for the prediction. Let γ 1 ,γ 2 denote the predictions from these two paths. Then the final coefficient map is predicted by γ = tanh(γ 1 γ 2 + γ 1 ). Such a design is motivated from the Squeeze-and-Attention [38]. The Conv-LSTM at the first path in the APU is used for capturing the dependencies among the blur amount at different scales. The hidden states in the Conv-LSTM can capture useful information from different scales and benefit the restoration across scales. This enables the SRAM to progressively improve the estimation on the coefficient maps. Remark 3. Unlike many existing unrolling-based DNNs (e.g. [22,20,23,24,25,26,27]) for image deblurring, the GKMNet does not have any explicit artifact removal block. However, it still performs well empirically, as shown in the experiments. The reasons may be as follows. First, the inversion process implemented by the GCM and SRAM is not the classic non-learnable estimator in existing unrolling-based DNNs, but the one containing learnable blocks. Thus, the GKMNet can be viewed as some forms of the inversion process with implicit built-in artifact removal. For instance, in the GKM model, the learned combination of Gaussian kernels not only can express the defocus kernels, but also may encode a component to suppress the artifacts. Second, there is probably a regularization effect due to the fact that the fixed point iterations do not necessarily converge to a fixed point (similar to early stopping), as well as a regularization effect due to the relaxation of the fixed-point iteration to the multi-scale iteration. While the exact nature of the regularizations and built-in artifact removal is unknown, it does not necessarily prevent the GKMNet from working, since the parameters are end-to-end learned in a supervised manner.

Experiments
There are few datasets available for benchmarking defocus deblurring. The most well-known one is the DPD dataset [18]. It provides 500 pairs of images with defocus blur and their corresponding all-in-focus images, as well as the two associated sub-aperture views called dual-pixel images, all in 16-bit color. The training/validation/test splits in the dataset consist of 350/74/76 samples. Our GKMNet is trained and tested on the DPD dataset without using the dual-pixel images. In addition, we test GKMNet on the RTF test set [7] as well as the images from CUHK-BD dataset [42].
Three image quality metrics are used for quantitative evaluation, including two standard metrics: PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measure) [43], and the LPIPS (Learned Perceptual Image Patch Similarity) [44] for perceptual quality (also used in [18]). The GKMNet is also compared to other DNNs in terms of model complexity and computational efficiency. The model complexity is measured by three metrics: number of parameters, number of FLOPs (floating-point operations per second), and model size. The computational efficiency is measured by the average inference time on an image of 1680 × 1120 pixels, tested on an Intel i5-9600KF CPU and on an NVIDIA GTX 1080Ti GPU, respectively. For non-DNN-based methods, only the inference time on CPU is reported.
Through all experiments, the maximum size of Gaussian kernels in GCM is set to M = 21. The number of scales is set to T = 3. The learnable parameters in SRAM are initialized by Xavier [45]. The Adam optimizer [46] is used for training with 3000 epochs and batch size 4. The learning rate is fixed at 10 −4 in the first 2000 epochs and decayed to 10 −5 in the last 1000 epochs. The cost function in (10) is set to the squared 2 loss in the first 1000 iterations, and alternatively set to the SSIM loss and squared 2 loss every 500 epochs afterwards. Data augmentation is done by random cropping to 256 × 256 pixels. The code of GKMNet is available at https://github.com/csZcWu/GKMNet.

Evaluation on DPD Dataset
The methods for comparison include JNB [2], EBDB [5], DMENet [6] and DPDNet [18]. The DPDNet [18] has two versions: DPDNet-D taking two dual-pixel images as input and DPDNet-S only taking a single image as input. Both versions are included for comparison whose results are reproduced by the pre-trained models from their authors. The JNB, EDBD and DMENet are all two-stage methods focusing on defocus map estimation, with results quoted from [18]. In addition, we also include two DNNs of dynamic scene deblurring: SRN [13] and AttNet [17]. These two DNNs also adopt a multi-scale architecture. Both DNNs are retrained using the same data as ours.
See Table 1 for the quantitative comparison on deblurring performance. Our GKMNet outperforms all other compared methods. The performance of EBDB, DMENet and JNB is much worse than that of GKMNet. This is probably because the errors in their defocus maps are magnified in the consequent deconvolution process. It indicates the advantage of the end-to-end learning of GKMNet which does not require defocus map estimation. The GKMNet also outperforms DPDNet-S by a large margin. This is not surprising as DPDNet-S is originally designed for defocus deblurring on two view images, not a single one. Also unsurprisingly, GKMNet outperforms SRN and AttNet, two DNNs designed for dynamic scene deblurring rather than SIDD. Surprisingly, our GKMNet with a single-image input even outperforms DPDNet-D which takes dual-pixel images as input. All these results have clearly demonstrated the effectiveness of the proposed GKMNet on SIDD. See supplementary materials for the visualization of coefficient maps generated by the SRAM. Table 1 also lists the results on model complexity and computational efficiency. In terms of all three metrics, GKMNet's complexity is much lower than that of other deep models. For instance, around 1/23 of DPDNet and 1/7 of SRN for number of parameters, and around 1/62 of DPDNet for model size. Such low complexity comes from the compact architecture of GKMNet. Regarding the running time, GKMNet is also much faster than the JNB, EBDB and DEMNet, as these methods need to call an iterative deconvolution post-process which is slow. GKMNet is about ten time faster than DPDNet, and its speed is comparable to SRN and AttNet. It is noted that GKMNet has a much less training time due to its low complexity. Its training on the DPD dataset takes less than 32 hours, while SRN and AttNet take nearly three days, on an NVDIA GTX 1080Ti GPU. As mentioned in Remark 3, without any explicit artifact/noise removal block in the GKMNet, the robustness of GKMNet is one concern. Thus, we evaluate the robustness to image noise using the DPD dataset with additional corruption by additive Gaussian white noise with standard deviationσ = 1, 3, 5 respectively. The DPDNet-D, SRN and AttNet are selected for comparison. Both the models trained on the original DPD training set and those retrained on the noisier DPD training set are included. The later ones are marked with a "+". See Table 2 for the results. In the presence of additional noise, our GKMNet still outperforms other models for both versions of training data.

Evaluation on RTF Dataset and CUHK-BD Sample Images
The RTF dataset [7] contains 22 pairs of defocused and all-in-focus images. Following [7], in addition to the original images, two noisier versions are generated by adding Gaussian white noise with standard deviationσ = 1 and 2.55 respectively. The DPDNet-S, SRN and AttNet are selected for comparison. Note that the training set of RTF is not available. Thus, we directly apply the models that are respectively trained on the original DPD dataset and the noisier DPD dataset. See Table 3 for the results. The GKMNet again outperforms other DNNs. Since the models art trained and tested with different datasets, such results also demonstrate the generalizability of our GKMNet.  The CUHK-BD dataset [42] contains 704 defocused images without ground truths. We select some of its images for test. See Figure 3 for the visual inspection on the results. Our GKMNet can successfully restore fine details with less visual artifacts, in comparison to DPDNet-S, SRN and AttNet. More visual comparison can be found in supplementary materials. It is worth mentioning that the defocus PSFs of the image synthesized from dual-pixel images are slightly different from the ones in real images [47,48,49]. However, our GKMNet still generalizes well.

Ablation Study
To evaluate the performance contribution of each component in GKMNet, we implement the following baselines from GKMNet for comparison, which are trained in the same way as the original GKMNet. The kernels in GCM are randomly initialized by Xavier [45] and learned with the DNN. Note that weight sharing is adopted for GCM across scales in (d) and (e) because it leads to performance improvement. See Table 4 for the results of the baselines, from which we have the following observations. (a) The multi-scale estimation scheme in SRAM leads to significant improvement over the single-scale one, with only a few additional model parameters introduced. (b) The improvement from the Conv-LSTM over simple cross-scale weight-sharing convolutions is significant, which implies that the Conv-LSTM can effectively exploit the features learned from different scales to guide the deblurring process. (c) Directly using the SRAM as an end-to-end DNN for SIDD leads to much worse results, which justifies the effectiveness of our specific DNN design. (d) Learning the GCM with Gaussian kernel initialization leads to a very minor improvement over using fixed Gaussian kernels in GCM. (e) Learning the GCM with a random initialization even yields slightly worse results. Both (d) and (e) indicate the effectiveness of our GKM model for defocus PSFs. See also Figure 4 for the kernels w/o and w/ learning in GCM. The kernels learned with (d) are very close to original predefined ones. Recall that the fixed point iteration (7) is applied to multiple iterations at the same scale, while our actual architecture also simultaneously increases the scale. We could further write our architecture as S scales and R iterations per scale, where we use S = 3, R = 1 in the GKMNet. To further verify the effectiveness of the multi-scale estimation in GKMNet, Table 5 lists the results using different values of S and R. It shows that using single or fewer scales with multiple iterations noticeably decreases the performance and increases the inference time. Using an additional iteration with S = 3 only brings a minor improvement but nearly doubles the complexity and time. Thus, our coarse-to-fine estimation scheme is a better implementation for high performance and low computational cost.

Conclusion
Defocus blur often occurs in images and has its own characteristics from motion blur. This paper proposed a DNN for SIDD with strong motivations from unrolling a fixed-point iteration derived from a GKM-based model of defocus blurring process. Together with a scale-recurrent implementation, we developed a lightweight DNN with state-of-the-art performance. In future, we will study the extension to other image deblurring problems such as dynamic scene deblurring.
While our approach achieved state-of-the-art results on existing datasets, its implicit regularizations and built-in artifact removal mechanisms are not very clear. In addition, its performance on severely blurred regions is not that satisfactory, with much room for improvement. These issues will also be studied in our further work.