{'fraction_non_alphanumeric': 0.03862726212804785, 'fraction_numerical': 0.022490326287152973, 'mean_word_length': 4.908156134571229, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Image Deblurring: A Survey

Image deblurring is a classic problem in low-level computer vision with the aim to recover a sharp image from a blurred input image. Advances in deep learning have led to significant progress in solving this problem, and a large number of deblurring networks have been proposed. This paper presents a comprehensive and timely survey of recently published deep-learning based image deblurring approaches, aiming to serve the community as a useful literature review. We start by discussing common causes of image blur, introduce benchmark datasets and performance metrics, and summarize different problem formulations. Next, we present a taxonomy of methods using convolutional neural networks (CNN) based on architecture, loss function, and application, offering a detailed review and comparison. In addition, we discuss some domain-specific deblurring applications including face images, text, and stereo image pairs. We conclude by discussing key challenges and future research directions.


INTRODUCTION
Image deblurring is a classic task in low-level computer vision, which has attracted the attention from the image processing and computer vision community. The objective of image deblurring is to recover a sharp image from a blurred input image, where the blur can be caused by various factors such as lack of focus, camera shake, or fast target motion [1,15,51,125]. Some examples are given in Figure 1.
Non-deep learning image deblurring methods often formulate the task as an inverse filtering problem, where a blurred image is modeled as the result of the convolution with blur kernels, either spatially invariant or spatially varying. Some early approaches assume that the blur kernel is known, and adopt classical image deconvolution algorithms such as Lucy-Richardson, or Wiener deconvolution, with or without Tikhonov regularization, to restore sharp images [107,129,144]. On the other hand, blind image deblurring methods assume the blur kernel is unknown and aim to simultaneously recover both the sharp image and the blur kernel itself. Since this task is ill-posed, the solution is regularized using various additional constraints [6,18,26,142]. While these non-deep learning methods show good performance in certain cases, they typically do not perform well in more complicated yet common scenarios such as strong motion blur.
Recent advances of deep learning techniques have revolutionized the field of computer vision; significant progress has been made in numerous domains, including image classification [36,120] and object detection [35,44,101,167]. Image deblurring is no exception: a large number of deep learning methods have been developed for single image and video deblurring, and have advanced the state of the art. However, the introduction of new methods with different network designs makes it challenging to obtain a rapid overview of the field. This paper aims to fill this gap by providing a survey of recent advances, and to serve as a reference point for new researchers. Specifically, we will focus the discussion on recently published deep learning based image and video deblurring methods. The aims of this paper are: -To review the preliminaries for image deblurring, including problem definitions, causes of blur, deblurring approaches, quality assessment metrics, and benchmark datasets for performance evaluation. -To discuss new developments of deep learning models for single image and video deblurring and provide a taxonomy for categorizing the existing methods (see Figure 2). -To analyze the challenges of image deblurring and discuss research opportunities.
The paper is organized as follows. In Section 2, we discuss the problem formulation, the causes of blur, the types of deblurring, and image quality metrics. Sections 3 and 4 introduce the CNN-based non-blind and blind image deblurring methods, respectively. Loss functions applied in deep deblurring methods are discussed in Section 5. We introduce public benchmark datasets and evaluations in Sections 6 and 7, respectively. In Section 8, we review three deblurring methods for specific domains, for face, text, and stereo images. Finally, we discuss the challenges and future opportunities in this research area.

Problem Formulation
Image blur can be caused by various factors during image capture: camera shake, in-scene motion, or out-offocus blur. We denote a blurred image I b as where Φ is the image blur function, and θ η is a parameter vector. I s is the latent sharp version of the blurred image I b . Deblurring methods can be categorized into non-blind and blind methods, depending on whether or not the blur function is known (see Sections 3 and 4). The goal of image deblurring is to recover a sharp image, i.e., finding the inverse of the blur function, as where Φ −1 is the deblurring model, and I db denotes the deblurred image, which is the estimate of the latent sharp image I s .

Motion
Blur. An image is captured by measuring photons over the time period of camera exposure. Under bright illumination the exposure time is sufficiently short for the image to capture an instantaneous moment. However, a longer exposure time may result in motion blur. Numerous methods directly model the degradation process as a convolution process by assuming that the blur is uniform across the entire image: where K is the blur kernel and θ µ represents additive Gaussian noise. In such an image, any object moving with respect to the camera will look blurred along the direction of relative motion. When we use Eq. 3 to represent the blur process Eq. 1, θ η corresponds to a blur kernel and Gaussian noise, while Φ corresponds to convolution and sum operator. For camera shake, motion blur occurs in the static background, while, in the absence of camera shake, fast moving objects will cause these objects to be blurred while the background remains sharp. A blurred image can naturally contain blur caused by both factors. Early methods model blur using shift-invariant kernels [26,143], while more recent studies address the case of non-uniform blur [28,59,60,86,131]. Out-of-focus Blur. Aside from motion blur, image sharpness is also affected by the distance between the scene and the camera's focal plane. Points on the focal plane are in true focus, and points close to it appear in focus, defining the depth of field. If the scene contains objects outside this region, parts of the scene will appear blurry. The Point Spread Function (PSF) for out-of-focus blur [73] is often modeled as: where (k, l) is the center of the PSF and r the radius of the blur. Out-of-focus deblurring has applications in saliency detection [45], defocus magnification [5] and image refocusing [160]. To address the problem of outof-focus blur, classic methods remove blurry artifacts via blur detection [117] or coded apertures [77]. Deep neural networks have been used to detect blur regions [130,162] and predict depth [4] to guide the deblurring process.
Gaussian Blur. Gaussian convolution is a common simple blur model used in image processing, defined as where x and y are the distance from the origin in the horizontal and vertical axis, respectively, σ is the standard deviation. Several methods have been developed to remove the Gaussian blur [13,41,132].
Mixed Blur. In many real-world scenes multiple factors contribute to blur, such as camera shake, object motion, and depth variation. For example, when a fastmoving object is captured at an out-of-focus distance, the image may include both motion blur and out-offocus blur as shown in Figure 1(d). To synthesize this type of blurry image, one option is to firstly transform sharp images to their motion-blurred versions (e.g., by averaging neighboring sharp frames taken in sequence) and then apply an out-of-focus blur kernel based on Eq. 4. Alternatively, one can train a blurring network to directly generate realistically blurred images [14,154]. In addition to these main types of blur there can be other causes, such as channel-dependent blur resulting from chromatic aberration [121,128].

Image Quality Assessment
Methods for image quality assessment (IQA) can be classified into subjective and objective metrics. Subjective approaches are based on human judgment, which may not require a reference image. One representative metric is the Mean Opinion Score (MOS) [39], where people rate the quality of images on a scale of 1-5. As MOS values depend on the population sample, methods typically take the statistics of opinion scores into account. For image deblurring, most existing methods are evaluated on objective assessment scores, which can be further split into two categories: full-reference and no-reference IQA metrics.
No-Reference Metrics. While the full-reference metrics require a ground-truth image for evaluation, noreference metrics use only the deblurred images to measure the quality. To evaluate the performance of deblurring methods on real-world images, several no-reference metrics have been used, such as BIQI [82], BLINDS2 [106], BRISQUE [80], CORNIA [149], DIIVINE [83], NIQE [81], and SSEQ [70]. Further, a number of metrics have been developed to evaluate the performance of image deblurring algorithms by measuring the effect on the accuracy of different vision tasks, such as object detection and recognition [66,148].

Non-Blind Deblurring
The goal of image deblurring is to recover the latent image I s from a given blurry one I b . If the blur kernel is given, the problem is also known as non-blind deblurring. Even if the blur kernel is available, the task is challenging due to sensor noise and the loss of highfrequency information.
Some non-deep methods employ natural image priors, e.g., global [57] and local [169] image priors, either in the spatial domain [102] or in the frequency domain [58] to reconstruct sharp images. To overcome undesired ringing artifacts, Xu et al. [143] and Ren et al. [104] combine spatial deconvolution and deep neural networks. In addition, several approaches have been proposed to handle saturated regions [19,138] and to remove unwanted artifacts caused by image noise [54,89].
We summarize existing deep learning based nonblind methods in Table 1. These approaches can be broadly categorized into two groups: the first group uses deconvolution followed by denoising, while the second group directly employs deep networks.
Deconvolution with Denoising. Representative algorithms in this category include [104,108,143,151,156]. Schuler et al. [108] develop a multi-layer perceptron (MLP) to deconvolve images. This approach first recovers a sharp image through a regularized inverse in the Fourier domain, and then uses a neural network to remove artifacts produced in the deconvolution process. Xu et al. [143] use a deep CNN to deblur images containing outliers. This algorithm applies singular value decomposition to a blur kernel and draws a connection between traditional optimization-based schemes and CNNs. However, the model needs to be retrained for different blur kernels. Using the low-rank property of the pseudo-inverse kernel in [143], Ren et al. [104] propose a generalized deep CNN to handle arbitrary blur kernels in a unified framework without re-training for each kernel. However, low-rank decompositions of blur kernels can lead to a drop in performance. Both methods [104,143] concatenate a deconvolution CNN and a denoising CNN to remove blur and noise, respectively. However, such denoising networks are designed to remove additive white Gaussian noise and cannot handle outliers or saturated pixels in blurry images. In addition, these non-blind deblurring networks need to be trained for a fixed noise level to achieve good performance, which limits their use in the general case. Kruse et al. [58] propose a Fourier Deconvolution Network (FDN) by unrolling an iterative scheme, where each stage contains an FFT-based deconvolution module and a CNN-based denoiser. Data with multiple noise levels is synthesized for training, achieving better deblurring and denoising performance.
The above methods learn denoising modules for non-blind image deblurring. Learning denoising modules can be seen as learning priors, which will be discussed in the following.
Learning priors for deconvolution. Bigdeli et al. [7] learn a mean-shift vector field representing a smoothed version of the natural image distribution, and use gradient descent to minimize the Bayes risk for non-blind deblurring. Jin et al. [48] use a Bayesian estimator for simultaneously estimating the noise level and removing blur. They also propose a network (GradNet) to speed up the deblurring process. In contrast to learning a fixed image prior, GradNet can be integrated with different priors and improves existing MAP-based deblurring algorithms. Zhang et al. [156] train a set of discriminative denoisers and integrate them into a model-based optimization framework to solve the non-blind deblurring problem. Without outlier handling, non-blind deblurring approaches tend to generate ringing artifacts, even in cases when the estimated kernel is accurate. Note that some super-resolution methods (with a scale factor of 1) can be adopted for the task of non-blind deblurring as the formulation as image reconstruction task is very similar, e.g., USRNet [155].

Blind Deblurring
In this section, we discuss recent blind deblurring methods. For blind deblurring, both the latent image and the blur kernel are unknown. Early blind deblurring methods focus on removing uniform blur [18,26,78,109,142]. However, real-world images typically contain nonuniform blur, where different regions in the same image are generated by different blur kernels [105]. Numerous approaches have been developed to address nonuniform blur by modeling the blur kernel from 3D camera motion [37,139]. Although these approaches can model out-of-plane camera shake, they cannot handle dynamic scenes, which motivated the use of blur fields of moving objects [12, 29,42]. Motion discontinuities and occlusions make the accurate estimation of blur kernels challenging. Recently, several deep learning based methods have been proposed for the deblurring of dynamic scenes [28,86,131]. Tables 2 and 3 summarize representative single image and video deblurring methods, respectively. To analyse these methods, we first introduce frame aggregation methods for network input. We then review the basic layers and blocks used in existing deblurring networks. Finally, we discuss the architectures, as well as the advantages and limitations of current methods.

Network inputs and frame aggregation
Single image deblurring networks take a single blurry image as input and generate the corresponding deblurred result. Video deblurring methods take multiple frames as input and aggregate the frame information in either the image or feature domain. Image-level aggregation algorithms, e.g., Su et al. [122], stack multiple frames as input and estimate the deblurred result for the central frame. On the other hand, feature-level aggregation approaches, e.g., Zhou et al. [165] and Kim et al. [43], first extract features from the input frames and then fuse the features for predicting the deblurred results.

Basic Layers and Blocks
This section briefly reviews the most common network layers and blocks used for image deblurring.
Recurrent layer. For single image deblurring, recurrent layers can extract features across images at multiple scales in a coarse-to-fine manner. Two representative methods are SRN [131] and PSS-SRN [28]. SRN is a coarse-to-fine architecture to remove motion blur via a shared-weight deep autoencoder, while PSS-SRN includes a selective parameter sharing scheme, which leads to improved performance over SRN.
Recurrent layers can also be used to extract temporal information from neighboring frames in videos [43,74,87,96,164,165]. The main difference from single image deblurring is that the recurrent layers in videobased methods extract features from neighboring images, rather than transferring information across only one input image at different scales. These methods can be categorized into two groups. The first group transfers the feature maps from the last step to the current network to obtain finer deblurred frames [43,87]. The second type of methods generates sharp frames via directly inputting deblurred frames from the last step [165].
Residual layer. To avoid vanishing or exploding gradients during training, the global residual layers are used to directly connect low-level and high-level layers in the area of image deblurring [59,86,161]. One representative method using this architecture is DeblurGAN [59] where the output of the deblurring network is added to the input image to estimate the sharp image. In this case, the architecture is equivalent to learning global residuals between blurry and sharp images. The local ResBlock uses local residual layers, similar to the residual layers in ResNet [36], and these are widely used in image deblurring networks [28,43,59,86,91,131]. Both kinds of residual layers, local and global, are often combined to achieve better performance.
Dense layer. Using dense connections can facilitate addressing the gradient vanishing problem, improving feature propagation, and reducing the number of parameters. Purohit et al. [98] propose a region-adaptive dense network composed of region adaptive modules to learn the spatially varying shifts in a blurry image. These region adaptive modules are incorporated into a densely connected auto-encoder architecture. Zhang et al. [154] and Gao et al. [28] also apply dense layers to  build their deblurring networks in which DenseBlocks are used to replace the CNN layers or ResBlocks.
Attention layer. The attention layer can help deep networks focus on the most important image regions for deblurring. Shen et al. [116] propose an attentionbased deep deblurring method consisting of three separate branches to remove blur from the foreground, the background, and globally, respectively. Since image regions containing people are often of the most interest, the attention module detects the location of people to deblur images using the guidance of a human-aware map. Other methods employ attention to extract better feature maps, e.g., using self-attention to generate a non-locally enhanced feature map [98].

Network Architectures
We categorize the most widely used network architectures for image deblurring into five sets: Deep auto-encoders (DAE), generative adversarial networks (GAN), cascaded networks, multi-scale networks, and reblurring networks. We discuss these methods in the following sections.
Deep auto-encoders (DAE). A deep auto-encoder first extracts image features and a decoder reconstructs the image from these features. For single image deblurring, many approaches use the U-Net architecture with Table 2 Overview of deep single image blind deblurring methods. In the 'Dataset' column 'averaging' refers to averaging over temporally consecutive sharp frames to synthesize training data.

Method
Category Blur type Dataset

Key idea
Learning-to-Deblur [109] Motion Cascade The first stage uses a CNN to estimate blur kernels and latent images. The second stage operates on the blurry images and latent image for kernel estimation. Averaging GAN An extension of DeblurGAN using a feature pyramid network and wide range of backbone networks for better speed and accuracy.
ASNet [52] Convolution DAE DAE framework, which first estimates the blur kernel in order to recover sharp images.
EBMD [46] Averaging DAE An event-based motion deblurring network, introducing a new dataset, DAVIS240C. RNN-DAE R ecurrent architecture which includes a dynamic temporal blending mechanism for information propagation.
PICNN [2] DAE Permutation invariant CNN which consists of several U-Nets taking a sequence of burst images as input. DBLRGAN [153] GAN GAN-based video deblurring method, using a 3D CNN to extract spatio-temporal information.
Reblur2deblur [14] Nonuniform Motion Averaging Reblur Three consecutive blurry images are fed into the reblur2deblur framework to recover sharp images, which are used to compute the optical flow and estimate a blur kernel for reconstructing the input.
IFIRNN [87] RNN-DAE RNN-based video deblurring network, where a hidden state is transferred from past frames to the current frame.

EDVR [134] DAE
Pyramid, Cascading and Deformable (PCD) module for frame alignment and a Temporal and Spatial Attention (TSA) fusion module, followed by a reconstruction module to restore sharp videos.

STFAN [165] DAE
STFAN takes the current blurred frame, the preceding blurred and restored frames as input and recovers a sharp version of the current frame.

CDVD-TSP [92] Cascade
Cascaded deep video deblurring which first calculates the sharpness prior, then feeds both the blurry images and the prior into an DAE.
a residual learning technique. [28,91,114,118,131]. In some cases additional networks help exploiting additional information for guiding the U-Net. For example, Shen et al. [114] propose a face parsing/segmentation network to predict face labels as priors, and use both, blurry images and predicted semantic labels, as the input to a U-Net. Other methods apply multiple U-Nets to obtain better performance. Tao et al. [131] analyze different U-Nets as well as DAE, and propose a scalerecurrent network to process blurry images. The first U-net obtains coarse deblurred images, which are fed into another U-Net to obtain the final result. The work in [115] combines the two ideas, using a deblurring network to obtain coarse deblurred images, then feeding them into a face parsing network to generate semantic labels. Finally, both, coarse deblurred images and labels, are fed into a U-Net to obtain the final deblurred images.
Video deblurring methods can be split into two groups based on their input. The first group takes a stack of neighboring blurry frames as input to extract spatio-temporal information. Su et al. [122] and Wang et al. [134] design DAE architectures to remove blur from videos by feeding several consecutive frames into the encoder, and the decoder recovers the sharp central frame. Features extracted from different layers  of the encoder are element-wise added to the corresponding decoder layers as shown in Figure 4, which accelerates convergence and generates sharper images. The second approach is to feed a single blurry frame  into an encoder to extract features. Various modules have been developed to extract features from neighboring frames, which are jointly fed into the decoder to recover the deblurred frame [165]. The features from neighboring frames can also be fed into the encoder for feature extraction [43]. Numerous deep video deblurring algorithms based on this architecture have been developed [87,118,134]. The main differences are in the module for fusing temporal information from neighboring frames, e.g., STFAN [165] in Figure 5 and an RNN module [87] in Figure 6.
Generative adversarial networks (GAN). GANs have been widely used in image deblurring [59,60,86,114] in recent years. Most GAN-based deblurring models share the same strategy: the generator (Figure 7) generates sharp images such that the discriminator cannot distinguish them from real sharp images. Kupyn et al. [59] proposed DeblurGAN, an end-to-end conditional GAN for motion deblurring. The generator of DeblurGAN contains two-strided convolution blocks, nine residual blocks, and two transposed convolution blocks to transform a blurry image to its corresponding sharp version. This method is further extended to DeblurGAN-v2 [60], which adopts a relativistic conditional GAN and a double-scale discriminator, which consists of local and global branches as in [44]. The core block of the generator is a feature pyramid network, which improves efficiency and performance. An adversarial loss is employed by Nah et al. [86] and Shen et al. [114] to generate better deblurred images.
GANs are also used in video deblurring networks. The main difference to single image deblurring is the generator, which also considers temporal information from neighboring frames. A 3DCNN model was developed by Zhang et al. [153] to exploit both spatial and temporal information to restore sharp details using adversarial learning, see Figure 8. Kupyn et al. [60] proposed DeblurGAN-v2 to restore sharp videos via modifying the single image deblurring method of Deblur-GAN.
Cascaded networks. A cascaded network contains several modules, which are sequentially concatenated to construct a deeper structure. Cascaded networks can be divided into two groups. The first one uses different architectures in each cascade. For example, Schuler et al. [109] propose a two-stage cascaded network, see Figure 9. The input to the first stage is a blurry image, and the deblurred output is fed into the second stage to predict blur kernels. The second group re-trains the same architecture in each cascade to generate deblurred images. Deblurred images from preceding stages are fed into the same type of network to generate finer deblurred results. This cascading scheme can be used in almost all deblurring networks. Although this strategy achieves better performance, the number of CNN parameters increases significantly. To reduce these, recent approaches share parameters in each cascade [92].
Multi-scale networks. Different scales of the input image describe complementary information [22,24,141]. The strategy of multi-scale deblurring networks is to first recover low-resolution deblurred images and then progressively generate high-resolution sharp results. Nah et al. [86] propose a multi-scale deblurring   network to remove motion blur from a single image ( Figure 10). In a coarse-to-fine scheme, the proposed network first generates images at 1/4 and 1/2 resolutions before estimating the deblurred image at the original scale. Numerous deep deblurring methods use a multi-scale architecture [28,99,131], improving deblurring at different scales, e.g., using nested skip connections in [28], or increasing the connection of networks across different scales, e.g., recurrent layers in [131].
Reblurring networks. Reblurring networks can synthesize additional blurry training images [6,154]. Zhang et al. [154] propose a framework which includes a learning-to-blur GAN (BGAN) and a learning-todeblur GAN (DBGAN). This approach learns to transform a sharp image into a blurry version and recover a sharp image from the blurry version, respectively.
Chen et al. [14] introduce a reblur2deblur framework for video deblurring. Three consecutive blurry images are fed into the framework to recover sharp images, which are further used to compute the optical flow and estimate the blur kernel for reconstructing the blurry input.
Advantages and drawbacks of various models. The U-Net architecture has shown to be effective for image deblurring [114,131] and low-level vision problems. Alternative backbone architectures for effective image deblurring include a cascade of Resblocks [59] or Denseblocks [154]. After selecting the backbone, deep models can be improved in several ways. A multi-scale network [86] removes blur at a different scales in a coarse-to-fine fashion, but at increased computational cost [158]. Similarly, cascaded networks [115] recover higher-quality deblurred images via multiple deblurring stages. Deblurred images are forwarded to another network to further improve the quality. The main difference is that the deblurred images in multi-scale networks are intermediate results, whereas the output of each deblurring network in the cascaded architecture can be individually regarded as final deblurred output. Multi-scale architectures and cascaded networks can also be combined by treating the multi-scale networks as a single stage in a cascaded network.
The primary goal of deblurring is to improve the image quality, which may be measured by reconstruction metrics such as PSNR and SSIM. However, these metrics are not always consistent with human visual perception. GANs can be trained to generate deblurred images, which are deemed realistic according to a discriminator network [59,60]. For inference, only the generator is necessary. GAN-based models typically perform poorer in terms of distortion metrics such as PSNR or SSIM.
When the number of training samples is insufficient, a reblurring network may be used to generate more data [14,154]. This architecture consists of a learn-to-blur and learn-to-deblur module. Any of the above-discussed deep models can be used to synthesize more training samples, creating training pairs of original sharp images and the output from the learn-to-blur model. Although this network can synthesize an unlimited number of training samples, it only models those blur effects that exist in the training samples.
The issue of pixel misalignment is challenging for image deblurring with multi-frame input. Correspondences are constructed by computing pixel associations in consecutive frames using optical flow or geometric transformations. For example, a pair of noisy and blurry images can be used for image deblurring and patch correspondences via optical flow [33]. When applying deep networks, alignment and deblurring can be handled jointly by providing multiple images as input and processing them via 3D convolutions [153].

Loss Functions
Various loss functions have been proposed to train deep deblurring networks. The pixel-wise content loss function has been widely used in the early deep deblurring networks to measure the reconstruction error. However, pixel loss cannot accurately measure the quality of deblurred images. This inspired the development of other loss functions like task-specific loss and adversarial loss for reconstructing more realistic results. In this section, we will review these loss functions.

Pixel Loss
The pixel loss function computes the pixel-wise difference between the deblurred image and the ground truth. The two main variants are the mean absolute error (L1 loss) and the mean square error (L2 loss), defined as: where I s(x,y) and I db(x,y) are the values of the sharp image and the deblurred image at location (x, y), respectively. The pixel loss guides deep deblurring networks to generate sharp images close to the groundtruth pixel values. Most existing deep deblurring networks [11,86,131,147,152,153] apply the L2 loss since it leads to a high PSNR value. Some models are trained to optimize the L1 loss [91,146,154]. However, since the pixel loss function ignores long-range image structure, models trained with this loss function tend to generate over-smoothed results [60].

Perceptual Loss
Perceptual loss functions [49] have also been used to calculate the difference between images. Different from the pixel loss function, the perceptual loss compares the difference in high-level feature spaces such as the features of deep networks trained for classification, e.g., VGG19 [120]. The loss is defined as: where Φ l x,y,c (·) are the output features of the classifier network from the l-th layer. C is the number of channels in the l-th layer. The perceptual loss function compares network features between sharp images and their deblurred versions, rather than directly matching values of each pixel, yielding visually pleasing results [59,60,75,153,154].

Adversarial Loss
For GAN-based deblurring networks, a generator network G and a discriminator network D are trained jointly such that samples generated by G can fool D.
The process can be modeled as a min-max optimization problem with value function V (G, D): where I and I b represent the sharp and blurry images, respectively. To guide G to generate photo-realistic sharp images, the adversarial loss function is used: where D(G(I b )) is the probability that the deblurred image is real. GAN-based deep deblurring methods have been applied to single image and video deblurring [59,60,86,114,153]. In contrast to the pixel and perceptual loss functions, the adversarial loss directly predicts whether the deblurred images are similar to real images and leads to photo-realistic sharp images.

Relativistic Loss
The relativistic loss is related to the adversarial loss, which can be formulated as: where D(·) is the probability that the input is a real image, C(·) is the feature captured via a discriminator and σ(·) is the activation function. During the training stage, only the second part of Eq. 11 updates parameters of the generator G, while the first part only updates the discriminator.
To train a better generator, the relativistic loss was proposed to calculate whether a generated image is more realistic than the synthesized images [50]. Based on this loss function, Zhang et al. [154] replace the standard adversarial loss with the relativistic loss: where E(·) denotes the averaging operation over images in one batch. C(·) is the feature captured via a discriminator and σ(·) is the activation function. The generator is updated by the relativistic loss function where I r denotes a real image. The method provided in [60] also applies this loss function to improve the performance of image deblurring.

Optical Flow Loss
Since the optical flow is able to represent the motion information between two neighboring frames, several studies remove motion blur via estimating optical flow. Gong et al. [31] build a CNN to first estimate the motion flow from the blurred images and then recover the deblurred images based on the estimated flow field. To obtain pairs of training samples, they simulate motion flows to generate blurred images. Chen et al. [14] introduce a reblur2deblur framework, where three consecutive blurry images are input to the deblur sub-net. Then optical flow between the three deblurred images is used to estimate the blur kernel and reconstruct the input.
Advantages and drawbacks of different loss functions. Generally, all the above loss functions contribute to the progress of image deblurring. However, their characteristics and goals differ. The pixel loss generates deblurred images which are close to the sharp ones in terms of pixel-wise measurements. Unfortunately, this typically causes over-smoothing. The perceptual loss is more consistent with human perception, while the results still exhibit obvious gaps with respect to real sharp images. The adversarial loss and optical flow loss functions aim to generate realistic deblurred images and model the motion blur, respectively. However, they cannot effectively improve the values of PSNR/SSIM or only work on motion blurred images. Multiple loss functions can also be used as a weighted sum, trading off their different properties.

Benchmark Datasets for Image Deblurring
In this section, we introduce public datasets for image deblurring. High-quality datasets should reflect all the different types of blur in real-world scenarios. We also introduce domain-specific datasets, e.g., face and text images, applicable for domain-specific deblurring methods. Table 4 presents an overview of these datasets.

Image Deblurring Datasets
Levin et al. dataset. To construct an image deblurring dataset, Levin et al. [64] mount the camera on a tripod to capture blur of actual camera shake by locking the Z-axis rotation handle and allowing motion in X and Y-directions. A dataset containing 4 sharp images of size 255 × 255 and 8 uniform blur kernels is captured using this set-up.
Sun et al. dataset. Sun et al. [126] extend the dataset of Levin et al. [64] by using 80 high-resolution natural images from [127]. Applying the 8 blur kernels from [64], this results in 640 blurred images. Similar to Levin et al. [64], this dataset contains only uniformly blurred images and is insufficient for training robust CNN models.
Köhler et al. dataset. To simulate non-uniform blur, Köhler et al. [56] use a Stewart platform (i.e., a robotic arm) to record the 6D camera motion and capture a printed picture. There are 4 latent sharp images and 12 camera trajectories, resulting in a total of 48 nonuniformly blurred images.
Lai et al. dataset. Lai et al. [61] provide a dataset which includes 100 real and 200 synthetic blurry images generated using both uniform blur kernels and 6D camera trajectories. The images in this dataset cover various scenarios, e.g., outdoor, face, text, and low-light images, and thus can be used to evaluate deblurring methods in a variety of settings.
GoPro dataset. Nah et al. [86] created a large-scale dataset to simulate real-world blur by frame averaging. A motion blurred image can be generated by integrating multiple instant and sharp images over a time interval: where g(·) is the Camera Response Function (CRF), and T denotes the period of camera exposure. Instead of modeling the convolution kernel, M consecutive sharp frames are averaged to generate a blurry image: REDS dataset. In order to capture the motion of fast moving objects, Nah et al.
[85] captured 300 video clips at 120 fps and 1080 × 1920 resolution using a GoPro Hero 6 Black camera, and increased the frame rate to 1920 fps by recursively applying frame interpolation [90]. By generating 24 fps blurry videos from the 1920 fps videos, spike and step artifacts present in the DVD dataset [122] are reduced. Both the synthesized blurry frames and the sharp frames are downscaled to 720 × 1280 to suppress noise and compression artifacts. This dataset has also been used for evaluating single image deblurring methods [88].

Performance Evaluation
In this section, we discuss the performance evaluation of representative deep deblurring methods.

Single Image Deblurring
We summarize representative methods in Table 5, which compares the performance on three popular single image deblurring datasets, the GoPro [86] dataset, and the datasets from Köhler et al. [56] and Shen et al. [116]. Note that all results are obtained from the respective papers. For the results of single image deblurring on the GoPro dataset and the results of video deblurring on the DVD dataset, we additionally provide bar graphs for easier comparison, as shown in Fig. 13 and Fig. 16. The methods developed by Sun et al. [125], Gong et al. [31] and Zhang et al. [152] are three early deep deblurring networks based on CNN and RNN, showing that deep learning based methods achieve competitive results. Using coarse-to-fine schemes, Nah et al. [86], Tao et al. [131], and Gao et al. [28] designed multi-scale networks and achieved better performance compared to single-scale deblurring networks as the coarse deblurring networks provide a better prior for higher resolutions. In addition to the multi-scale strategy, GANs have been employed to produce more realistic deblurred images [59,60,154]. However, GAN-based models achieve poorer results in terms of PSNR and SSIM metrics on the GoPro dataset [86] as shown in Table 5. Purohit et al. [98] applied an attention mechanism to the deblurring network and achieved state-ofthe-art performance on the GoPro dataset. The results on Shen et al.'s dataset [116] demonstrate the effectiveness. Networks using ResBlocks [86] and DenseBlocks [28,154] also achieve better performance than previous networks which directly stack convolutional layers [31,125]. For non-UHD motion deblurring, the multi-patch architecture has the advantage of restoring images better than the multi-scale and GAN-based architectures. For other architectures, such as multi-scale networks, the small-scale version of images provides less details, and thus these methods cannot improve the performance significantly. Fig. 12 shows sample results of single image deblurring from the GoPro dataset. We compare two multiscale methods [86,131] and two GAN-based methods [60,154] in the experiments. Despite the differences in model architectures, all these methods perform well on this dataset. Meanwhile, methods using similar architectures can yield different results, e.g., Tao et al. and Nah et al. [86], which are based on multi-scale architectures. Tao et al. use recurrent operations between different scales and achieve sharper deblurring results.
We also analyze the performance on the GoPro dataset using the LPIPS metric. Results of representative methods are shown in Table 6. Under this metric, Tao et al. [131] performs worse than Nah et al. [86] and Kupyn et al. [59]. This is because the LPIPS measures the perceptual similarity rather than the pixel-wise similarity measures like L1. Specifically, Tao et al. [131] outperforms the Nah et al. [86] and Kupyn et al. [59] in terms of PSNR and SSIM on the GoPro dataset. The results show that we may draw different conclusions when various metrics are used. Therefore, it is critical to evaluate deblurring methods using different quantitative metrics. As evaluation metrics optimize different criteria, the design choice is task-dependent. There is a perception vs. distortion tradeoff in image reconstruction tasks, and it has been shown that these two measures are at odds with each other, see [8]. To evaluate methods in fairly, one may design a combined cost function including measures such as PSNR, SSIM, FID, LPIPS, NIQE, and carry out user studies.
To analyze the effectiveness of different loss functions, we design a common backbone of several Res-Blocks (provided by DeblurGAN-v2). Different loss functions, including L1, L2, perceptual loss, GAN loss, Fig. 12 Evaluation results of the state-of-the-art deblurring methods on the GoPro dataset [86]. From left to right: blurry images, results of Nah et al. [86], Tao et al. [131], DBGAN [154] and DeblurGAN-v2 [60]. [86] and [131] are two multi-scale based image deblurring networks. [154] and [60] are two GAN based image deblurring networks.  Table 6 Performance of representative single deblurring methods using the LPIPS metric on the GoPro dataset.
Method LPIPS Nah et al. [86] 0.1819 Kupyn et al. [60] 0.2528 Tao et al. [131] 0.7879 Gao et al. [28] 0.0359 Zhang et al. [154] 0.1097 and RaGAN loss [154], and their various combinations are used for training. Similar to the settings in DeblurGAN-v2 [60], we train each model for 200 epochs on the GoPro dataset. The learning rate scheduler corresponds to that in DeblurGAN-v2. Experimental results are reported in Table 7. In general, the evaluated models perform better when combining reconstruction and perceptual loss functions. However, using GANbased loss functions does not necessarily improve PSNR or SSIM, which is consistent with the findings in prior reviews [63]. In addition, the same model using GAN loss or RaGAN loss achieves similar deblurring performance.  Fig. 13 Comparison among state-of-the-art image deblurring methods in terms of PSNR on the GoPro dataset. Fig. 14 Comparison among state-of-the-art non-blind and blind image deblurring methods. The blur kernels are predicted by the method in [145]. From left to right: blurry images, results of FDN [58], RGDN [32], SRN [131] and DBGAN [154]. FDN [58] and RGDN [32] are non-blind deblurring methods, whereas SRN [131] and DBGAN [154] are blind deblurring methods.
In order to evaluate the performance difference between non-blind and blind single image deblurring methods, we conduct a study comparing representative non-blind (FDN [58] and RGDN [32]) and blind single image deblurring methods [28,60,86,131,154] on the RWBI dataset. Considering that the RWBI dataset does not provide the ground-truth blur kernels, we use [145] to estimate blur kernels for non-blind image deblurring. Results are presented in Table 8 using the NIQUE and BRIQUE metrics. Overall, the blind deblurring methods outperform the non-blind methods since non-blind approaches require explicitly estimating the blur kernel. In practice, estimating the blur kernel is still a challenging task. If a kernel is not well estimated, it will negatively affect the image restauration task. Fig. 14 shows sample results from the RWBI dataset.
Method NIQUE BRISQUE FDN [58] 12.4276 57.7418 RGDN [32] 13.0038 51.5109 Nah et al. [86] 12.2365 49.9521 Kupyn et al. [60] 11.8186 40.4656 Tao et al. [131] 12.4606 51.1515 Gao et al. [28] 12.3987 50.5300 Zhang et al. [154] 11.5048 45.5496 from the GoPro dataset [86]. The generation method is 1 Kupyn et al. [60] 0.48 Suin et al. [123] 0.34 based on Eq. 3 using Gaussian noise. The blur kernels and blurry images are the input to non-blind image deblurring methods (DWDN [23] and USRNet [155]), while the input to blind image deblurring methods (MSCNN [86] and SRN [131]) are only blurry images. Table 9 and Fig. 15 show quantitative and qualitative results, respectively. Overall, non-blind image deblurring methods can achieve better performance than blind image deblurring approaches if the ground-truth blur kernels are provided. For a better understanding of the role of different training datasets, we train the SRN model [131] on three different public datasets (RealBlur-J [105], Go-Pro [86], and BSD-B [76,105]) separately. In addition, we also train the SRN model on the combinations of "RealBlur-J + GoPro", "RealBlur-J + BSD-B", and "RealBlur-J + BSD-B + GoPro". Results are reported in Table 13. The models using diverse data perform better than those only using one type of training data. In particular, the model using the combination of three datasets achieves the highest PSNR and SSIM values on the RealBlur-J testing dataset.
Considering that modern mobile devices allow capturing ultra-high-definition (UHD) images, we synthesize a new dataset to study the performance of different architectures on UHD image deblurring. We use a Sony RX10 camera to capture 500 and 100 sharp images with 4K resolution as training and testing sets, respectively. Next, we use 3D camera trajectories to generate blur kernels and synthesize corresponding blurry images by convolving sharp images with large blur kernels (sizes 111 × 111, 131 × 131, 151 × 151, 171 × 171, 191 × 191). We use two multi-scale networks (MSCNN, SRN), two GAN-based networks (DeblurGAN, DeblurGAN-v2) and one multi-patch networks (DMPHN) for experiments. Table 14 shows the results of these representative deblurring methods. The results show that UHD image deblurring is a more challenging task and multiscale architectures achieve better performance in terms of PSNR and SSIM. For UHD motion deblurring, multiscale architectures achieve better performance. As UHD images have higher resolution, downsampled versions at 1/4 scale still contain sufficient detail. Multi-scale architectures take UHD blurry images at 1/4 of the original resolution as input and generate the corresponding sharp versions. During the training stage, the images with 1/4 resolution can provide additional information for training and thus improve the performance of deblurring networks.
In order to analyze the performance of different architectures on defocus deblurring, we create another dataset and conduct numerous experiments. Specifically, we use a Sony RX10 camera to capture 500 pairs of sharp and blurry images for training, and 100 pairs of sharp and blurry images for testing. The image resolution is 900×600 pixels. Similarly, two multi-scale networks (MSCNN, SRN), two GAN-based networks (De-blurGAN, DeblurGAN-v2) and one multi-patch network (DMPHN) are evaluated on this dataset. Table 15 shows the results of the above methods. Compared with averaging-based motion deblurring, GAN-based architectures can achieve better performance on defocus deblurring. Multi-scale and multi-patch architectures do not show significant improvements in terms of PSNR and SSIM. For defocus deblurring, results show that these two networks (with and without GAN framework) do not show significant differences in terms of PSNR and SSIM. However, for motion deblurring, GAN-based architectures yield lower PSNR and SSIM values. This may be attributed to the fact that GAN-based architectures paying attention to whole images using the adversarial loss function. In comparison, methods without GANs consider the pixel-level error (L1, L2), and thus ignore the whole image. We provide a run-time comparison of representative methods in Table 10.

Video Deblurring
In this section, we compare recent video deblurring approaches on two widely used video deblurring datasets, the DVD [122] and REDS [85] datasets, see Table 11. Deep auto-encoders [122] are the most commonly used architecture for deep video deblurring methods. Similar to single image deblurring methods, the convolutional layer and ResBlocks [36] are the most important components. The recurrent structure [55] is used to extract  1.72 Kim et al. [43] 9.24 Nah et al. [87] 28.8 Table 13 Performance evaluation of different training sets [76,86,105]. The image deblurring model is Tao et al. [131], and the test set is from RealBlur-J dataset [105]. The values are reported in [105]. temporal information from neighboring blurry images, which is the main difference to single image deblurring networks. Pixel-wise loss functions are employed in most video deblurring methods, but, similar to single image deblurring, perceptual loss [166] and adversarial loss [153] have also been used.
While single image deblurring methods can be applied to videos, they are outperformed by videobased methods that exploit temporal information. Su et Zhou et al. [165] proposed the STFAN module, which better incorporates information from preceding frames into the deblurring process of the current frame. In addition, a filter adaptive convolutional (FAC) layer is employed for aligning the deblurred features from these frames. Pan et al. [92] achieve state-of-the-art performance on the DVD dataset by using a cascaded network and PWC-Net to estimate optical flow for calculating sharpness priors [124]. A DAE network uses the priors and blurry images to estimate sharp images. Run times of representative methods are shown in Table 12.

Domain-specific Deblurring
While most deep learning based methods are designed for deblurring generic images, i.e. common natural and man-made scenes, some methods have studied the deblurring problem in specific domains or settings, e.g., faces and texts. Fig. 17 Face deblurring examples. From left to right: input images, results obtained by the methods in [145], [93], [86] and [114], respectively.

Face Image Deblurring
Early face image deblurring methods used key structures of facial images [34,93]. Recently, deep learning solutions have dominated the development of face deblurring by exploiting specific facial characteristics [47,103,114,147]. Shen et al. [114] use semantic information to guide the process of face deblurring. Pixel-wise semantic labels are extracted via a parsing network and serve as priors to the deblurring network to restore sharp faces. Jin et al. [47] develop an end-to-end network with a resampling convolution operation to widen the receptive field. Chrysos et al. [20] propose a twostep architecture, which first restores the low frequencies and then restores the high frequencies to ensure the outputs lie on the natural image manifold. To address the task of face video deblurring, Ren et al. [103] explore 3D facial priors. A deep 3D face reconstruction network generates a textured 3D face for the blurry input, and a face deblurring branch recovers the sharp face under the guidance of the posed-aligned face. Figure 17 shows deblurring results from two non-deep methods [93,145] and two deep learning based methods [86,114].

Text Image Deblurring
Blurred text images impact the performance of optical character recognition (OCR), e.g., when reading documents, displays, or street signs. Generic image deblurring methods are not well suited for text images. In early work Panci et al. [95] model the text image as a random field to remove blur via blind deconvolution algorithms [27]. Pan et al. [94] deblur text images via an L 0 -regularized prior based on intensity and gradient. More recently, deep learning methods, e.g. the method by Hradiš et al. [40] have shown to effectively remove out-of-focus and motion blur. Xu et al. [147] adopt a GAN-based model to learn a category-specific prior for the task, designing a multi-class GAN model and a novel loss function for both face and text images. Figure 18 shows the deblurring results from sev-  [145], [18], [163], [16], [17], [94] and [40], respectively.

Fig. 19
Stereo image deblurring example [166]. Two stereo blurry images are fed into deep deblurring networks to generate their corresponding sharp images.

Stereo Image Deblurring
Stereo vision has been widely used to achieve depth perception [30] and 3D scene understanding [25]. When mounting a stereo camera on a moving platform, vibration will lead to blur, negatively affecting the subsequent stereo computation [110]. To alleviate this issue, Zhou et al. [166] proposed a deep stereo deblurring method to make use of depth information, and varying information in corresponding pixels across two stereo images. Depth information helps estimate the spatiallyvarying blur as it provides prior knowledge that nearer points are more blurry than farther points, in the case of translational motion, which is a common real-world scenario. Further, the blur of corresponding pixels in two stereo images is different, which allows using the sharper pixel in the final deblurred images. In addition to the above tasks, other domain-specific deblurring tasks include extracting a video sequence from a single blurred image [47,99], synthesizing high-framerate sharp frames [113], and joint deblurring and superresolution [161].

Challenges and Opportunities
Despite the significant progress of image deblurring algorithms on benchmark datasets, recovering clear images from a real-world blurry input remains challenging [86,122]. In this section, we summarize key limitations and discuss possible approaches and research opportunities.
Real-world data. There are three main reasons that deblurring algorithms do not perform well on real-world images.
First, most deep learning based methods require paired blurry and sharp images for training, where the blurry inputs are artificially synthesized. However, there is still a gap between these synthetic images and real-world blurry images as the blur models (e.g., Eq. 3 and 15) are oversimplified [10,14,154]. Models trained on synthetic blur achieve excellent performance on synthetic test samples, but tend to perform worse on realworld images. One feasible approach to obtain better training samples is to build better imaging systems [105], e.g., by capturing the scenes using different exposure times. Another option is to develop more realistic blur models that can synthesize more realistic training data.
Second, real-world images are not only corrupted due to blur artifacts, but also due to quantization, sensor noise, and other factors like low-resolution [114,161]. One way to address this problem is to develop a unified image restoration model to recover high-quality images from the inputs corrupted by various nuisance factors.
Third, deblurring models trained on general images may perform poorly on images from domains that have different characteristics. Specifically, it is challenging for general methods to recover sharp images of faces or text while maintaining the identity of a particular person or characters in the text, respectively.
Loss functions. While numerous loss functions have been developed in the literature, it is not clear how to use the right formulation for a specific scenario. For example, as Table 7 shows that an image deblurring model trained with L1 loss may achieve better performance than models trained with L2 loss on the GoPro dataset. However, the same network trained with L1 and perceptual loss functions is worse than the models trained with L2 and perceptual loss functions.
Evaluation metrics. The most widely used evaluation metrics for image deblurring are PSNR and SSIM. However, the PSNR metric is closely related to the MSE loss, which favors over-smoothed predictions. Therefore, these metrics cannot accurately reflect perceptual quality. Images with lower PSNR and SSIM values can have better visual quality [63]. The Mean Opinion Score (MOS) is an effective measure of perceptual quality. However, this metric is not universal and cannot be easily reproduced as it requires a user study. Therefore, it is still challenging to derive evaluation metrics that are consistent with the human visual response.
Data and models. Both data and models play important roles in obtaining favorable deblurring results. In the training stage, high-quality data is important to construct a strong image deblurring model. However, it is difficult to collect large-scale high-quality datasets with ground-truth. Usually, it requires using to use two cameras (with proper configurations) to capture real pairs of training samples, and thus the amount of these high-quality data is small with small diversity. While it is easier to generate a dataset with synthesized images, deblurring models developed based on such data usually perform worse than those built upon high-quality realworld samples. Therefore, constructing a large number of high-quality datasets is an important and challenging task. In addition, high-quality datasets should contain diverse scenarios, in terms of types of objects, motion, scenes, and image resolution. Deblurring network models have been mainly designed based on empirical knowledge. Recent neural architecture search methods, such as AutoML [69,97,168], may also be applied to the deblurring task. In addition, transformers [133] have achieved great success in various computer vision tasks. How to design more powerful backbones using Transformer may be an opportunity.
Computational cost. Since many current mobile devices support capturing 4K UHD images and videos, we test several state-of-the-art deep learning based deblurring methods. However, we found that most these deep learning based deblurring methods cannot handle 4K resolution images at high speed. For example, on a single NVIDIA Tesla V100 GPU, Tao et al. [131], Nah et al. [86] and Zhang et al. [154] take approximately 26.76, 28.41, 31.62 seconds, respectively, to generate a 4K deblurred image. Therefore, efficiently restoring highquality UHD images remains an open research topic. Note that most existing deblurring networks are evaluated on desktops or servers equipped with high-end GPUs. However, it requires significant effort to develop efficient deblurring algorithms directly on mobile devices. One recent example is the work by Kupyn et al. [60], which proposes a MobileNet-based model for more efficient deblurring.
Unpaired learning. Current deep deblurring methods rely on pairs of sharp images and their blurry counterparts. However, synthetically blurred images do not represent the range of real-world blur. To make use of unpaired example images, Lu et al. [72] and Madam et al. [75] recently proposed two unsupervised domain-specific deblurring models. Further improving semi-supervised or unsupervised methods to learn deblurring models appears to be a promising research direction.