{'fraction_non_alphanumeric': 0.03338330660344467, 'fraction_numerical': 0.0171884805801548, 'mean_word_length': 4.660647325833827, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Face Deblurring using Dual Camera Fusion on Mobile Phones

Motion blur of fast-moving subjects is a longstanding problem in photography and very common on mobile phones due to limited light collection efficiency, particularly in low-light conditions. While we have witnessed great progress in image deblurring in recent years, most methods require significant computational power and have limitations in processing high-resolution photos with severe local motions. To this end, we develop a novel face deblurring system based on the dual camera fusion technique for mobile phones. The system detects subject motion to dynamically enable a reference camera, e.g., ultrawide angle camera commonly available on recent premium phones, and captures an auxiliary photo with faster shutter settings. While the main shot is low noise but blurry, the reference shot is sharp but noisy. We learn ML models to align and fuse these two shots and output a clear photo without motion blur. Our algorithm runs efficiently on Google Pixel 6, which takes 463 ms overhead per shot. Our experiments demonstrate the advantage and robustness of our system against alternative single-image, multi-frame, face-specific, and video deblurring algorithms as well as commercial products. To the best of our knowledge, our work is the first mobile solution for face motion deblurring that works reliably and robustly over thousands of images in diverse motion and lighting conditions.

. We present a robust and efficient system that leverages synchronized dual capture, which is commonly available on mobile phones, to deblur faces at an interactive rate on mobile devices. Input (a): In this setting where the subject performs ordinary exercise (jumping on a trampoline), the commercial-grade auto-exposure system equipped with high-sensitivity sensor from a modern premium phone still produces objectionable motion blur on the face region. Reference (b): Our system detects the subject motion and uses the ultrawide camera to capture a short-exposure shot as reference simultaneously. While the image appears noisy, has low-resolution, and has wrong color, it preserves the subject's facial details. Output (c): Our system recovers the sharp and authentic facial details by aligning and fusing the input with the reference.
Motion blur of fast-moving subjects is a longstanding problem in photography and very common on mobile phones due to limited light collection efficiency, particularly in low-light conditions. While we have witnessed great progress in image deblurring in recent years, most methods require significant computational power and have limitations in processing highresolution photos with severe local motions. To this end, we develop a novel face deblurring system based on the dual camera fusion technique for mobile phones. The system detects subject motion to dynamically enable a reference camera, e.g., ultrawide angle camera commonly available on recent premium phones, and captures an auxiliary photo with faster shutter settings. While the main shot is low noise but blurry (Figure 1(a)), the reference shot is sharp but noisy (Figure 1(b)). We learn ML models to align and fuse these two shots and output a clear photo without motion blur (Figure 1(c)). Our algorithm runs efficiently on Google Pixel 6, which takes 463 ms overhead per shot. Our experiments demonstrate the advantage and robustness of our system against alternative single-image, multi-frame, face-specific, and video deblurring algorithms as well as commercial products. To the best of our knowledge, our work is the first mobile solution for face motion

INTRODUCTION
Taking photos of moving human subjects such as kids running or jumping has been a popular photography topic yet notorious challenge due to its well-known motion blur artifacts. While motion blur on limbs or body conveys the dynamic nature of human actions, it can distort or even totally remove the subject's facial identity or expression which are critical to the photo. To capture crisp and clean facial details from the subject, sophisticated skills are required to predict types and strengths of motion, carefully estimate the lighting When a subject is illuminated by artificial lighting, e.g., 60Hz, increasing shutter speed faster than 1/120s (a) not only leads to excessive noise but also objectionable banding artifacts as shown in (b) due to the rolling shutter effects in CMOS sensors on mobile phones.
conditions, and set the perfect exposure time. Such skills take welltrained photographers years to master and would be otherwise difficult for casual users. Given the difficulty in precise motion estimation, premium mobile cameras today employ large sensors and aggressively reduce the exposure time when motion is detected [Liba et al. 2019]. Unfortunately, increasing the shutter speed introduces stronger noise [Plotz and Roth 2017], and brings objectionable flickering and banding artifacts under common AC-supplied lighting, due to the rolling shutter read of CMOS sensors [Dietz and Eberhart 2019]. For example, the 60Hz AC supply in the U.S. prevents the exposure time from being faster than 1/120s, as illustrated in Figure 2.
Recovering sharp details from blurry faces through post-capture processing is fundamentally challenging. Blind deconvolution methods often model motion blur as a global or slowly-varying local kernels [Ren et al. 2016;Yang and Ji 2019], and cannot handle face motions that may contain non-rigid and out-of-plane movements, saturation at eye highlights, camera ISP non-linearity, lens aberration, and self-occlusion between hairs and facial landmarks. Deep learning methods using a single image [Cho et al. 2021;Nah et al. 2017], a multi-frame burst [Aittala and Durand 2018], or a video sequence [Son et al. 2021;] have made significant progresses over the years. However, high-quality solutions on highresolution data require expensive computation that is prohibitive on mobile phones with limited memory and processing budget. This paper describes a system to deblur faces of moving subjects captured by handheld mobile phones. We tailor the system to faces given its importance in mobile photography. Our system, which launched to the public as Face Unblur on Google Pixel 6 smartphones, aims to achieve the following goals: -Reliably produce high-quality, high-resolution (2.3MP), realisticlooking, and artifact-free deblur results on moving subjects in the wild. We handle any form of face motion in real-world, where the motion blur size can reach hundreds of pixels. -Require no additional users inputs other than pressing the shutter button. The system detects motion blur, identifies the blurry areas, and applies deblurring automatically. -Run on mobile phone at interactive rate. In other words, users see the result immediately after the photo is taken. -Operate within the power, memory, and computation budget provided by the SoC on Google Pixel 6.
-Support zero-shutter lag capture for motion subject. The deblurring is applied to the moment that the user intends to capture, not nearby frames. Our solution is built on (1) the heterogeneous dual camera system adopted by the majority of phone vendors, i.e., a wide-angle main camera and a secondary ultrawide-angle camera, denoted as W and UW throughout this paper, and (2) the ability to synchronize timestamp, auto-exposure, and auto white-balancing between W and UW cameras offered by most smartphones. When the user captures a moving subject using the main W camera, our system captures a synchronized reference raw using the UW camera at a faster shutter speed to acquire sharp but noisy facial details. The main W and reference UW shots exhibit different viewpoint and sensor characteristics, e.g., noise, gain, color, etc. We train deep CNNs to align and fuse the low noise but blurry linear RAW from W, and noisy but sharp linear RAW from UW, generating a clean and sharp linear raw of the face region from the W 's point of view (Section 3). The fused linear RAW is then fed to a standard image processing pipeline to output the tone-mapped result. Compared to the alternative that takes long and short exposure frames from the same camera, our dual camera approach leverages temporal synchronization to greatly improve the quality of image alignment between W and UW.
However, dual W and UW streaming consumes prohibitively high power and memory footprint. To reduce the amortized power, we propose an adaptive stream system to dynamically enable and disable the UW sensor based on real-time motion analysis. We integrate the algorithm into the camera driver to avoid excessive activation delay (Section 4) . Finally, we describe the requirements of the camera image signal processing pipeline to enable our system (Section 5).
Section 6 evaluates our deblur results over 1783 images exhibiting motions in daily tasks, such as jumping, walking, and exercising, containing subjects of diverse skin color, gender, and face poses. The inputs are challenging and consist of local, non-rigid, and large motion, where blur kernel may reach up to hundreds of pixels. We compare our results against state-of-the-art methods in single-image, multi-frame, face-specific, and video-based deblurring papers, as well as commercial products such as iPhone13 Pro, Shake Reduction filter in Adobe PhotoShop, and Samsung Gallery's Remaster function. Our method achieves higher scores on no-reference image quality metrics and generates results that look visually more pleasing. Our images used for comparisons in Section 6 are available at the project website 1 . To the best of our knowledge, our system is the first practical face deblur solution that runs on mobile phones.

RELATED WORK
Our system addresses a long-standing computer vision problem: motion deblurring. The full review of this highly active field is outside the scope of this paper, and we refer users to an overview paper [Koh et al. 2021]. The state-of-the-art methods using variants of UNet [Cho et al. 2021] require a large and realistic dataset for training [Nah et al. 2017;Rim et al. 2020]. However, collecting large-scale real-world blur images paired with ground truth is still  3. System overview. Our system takes raw bursts from W and UW. We control the shutter speed of UW to be 4× faster than W. UW image is sharper but appears noisier and may contain flickering, i.e., the vertical bands in the image, due to rolling shutter under artificial lighting. We merge the burst raw into a linear RAW, and crop the face region for deblurring using ML-based alignment (Section 3.2) and fusion (Section 3.3). Detailed steps are described in Figure 4. We then blend the deblurred face back to W (Section 3.4), and outputs the final image through tone mapping and post-processing (Section 3.5). Note that the reference image size is 2× smaller than the source. We enlarge the reference image for better visualization.
impractical. In contrast, we show that our system based on dual camera fusion only requires training on synthetic blur images to generate high-quality results. We cover the most relevant papers related to our work in this section.

Single-image motion deblurring
Conventional methods typically rely on the MAP optimization framework [Cho and Lee 2009;Fergus et al. 2006;Shan et al. 2008;Xu and Jia 2010] and natural image priors [Krishnan et al. 2011;Lai et al. 2015;Pan et al. 2016b;Sun et al. 2013;Xu et al. 2013] for single-image blind deblurring. However, using complicated image priors requires iterative optimization steps, which lead to high computational cost, and thus makes it difficult to be deployed to mobile devices with limited computing resources.
With the advance of deep learning techniques in recent years, several learning-based approaches adopt deep CNNs for motion deblurring. Early methods train CNNs to predict local blur kernels [Gong et al. 2017;Sun et al. 2015] or learn image priors [Li et al. 2018], but still involve conventional optimization-based framework to deblur images. Recent approaches train end-to-end CNNs for single-image debluring by exploring different architecture designs [Cho et al. 2021;Gao et al. 2019;Kaufman and Fattal 2020;Li et al. 2020;Nah et al. 2017;Suin et al. 2020;Tao et al. 2018;Zamir et al. 2021; or GAN-based training [Kupyn et al. 2018[Kupyn et al. , 2019. While state-of-the-art methods achieve great performance on benchmark datasets (e.g., GoPro [Nah et al. 2017] and RealBlur [Rim et al. 2020]), single-image deblurring methods typically are not robust enough to be generalized well to real images in the wild, e.g., either fail to remove large motion blur or leave unpleasing visual artifacts (see Figure 12). In contrast, we develop an effective reference-based deblurring approach that achieves high-quality results on mobile devices, which enables processing 2.3MP images, while most singleimage deblur methods are limited to 720p [Hu et al. 2021;Nah et al. 2017] due to computational constraints. Furthermore, unlike GANbased approaches [Kupyn et al. 2019;Xu et al. 2017] that have risks of generating a different face, our system takes a reference image from the same subject captured at the same time, which can preserve the subject identity well.
On the other hand, when deblurring low-light images, saturated light streaks are commonly visible on eyeglasses, eyeballs, or reflections. Optimization-based approaches handle saturation by excluding saturated pixels Pan et al. 2016a;Whyte et al. 2014] or estimating blur kernels from light streaks . We found that our model trained on synthetic data may not be able to remove saturated light streaks and leave undesired ringing artifacts (see Figure 17). Therefore, we add synthetic light streaks to our training data and find this simple data augmentation sufficient to remove saturated blur streaks.

Multi-frame and video deblurring
As single-image deblurring is a challenging ill-posed problem, several methods have exploited additional input images for deblurring. Multi-frame deblurring approaches merge a burst of images via Fourier burst accumulation [Delbracio and Sapiro 2015a] [Cho et al. 2012;Delbracio and Sapiro 2015b]. Recent learning-based approaches learn to aggregate information from neighboring frames using optical flow ], recurrent networks Nah et al. 2019;Son et al. 2021;Zhou et al. 2019], spatial-temporal transformers [Kim et al. 2018], deformable convolutions , and multi-patch architectures [Deng et al. 2021]. Our system captures W and UW raw bursts consisting of 7 − 9 frames and uses the robust local alignment in [Hasinoff et al. 2016] to merge raw frames to a single linear image before deblurring. The robust local alignment step avoids additional blur in the raw-merging step. Compared to deblurring on raw burst frames, deblurring on the merged linear image avoids the challenging temporal alignment and excessive memory cost.

Reference-based deblurring
Reference-based approaches take a pair of blurry and sharp, shortexposed images as inputs Gu et al. 2020;Yuan et al. 2007;Zhuo et al. 2010]. Early approaches adopt conventional Richardson-Lucy deconvolution [Yuan et al. 2007] or MAP framework [Zhuo et al. 2010] to estimate a blur kernel for deblurring. These methods assume the input and reference images are wellaligned, and motion blur is spatially-invariant i.e., a global blur kernel exists. The assumption does not apply to dynamic subject motions in our problem. Gu et al. [2020] use optical flow to align the noisy reference image and build GMM priors to model the sharp latent image. This work requires expensive iterative optimization (10+ minutes as reported in [Gu et al. 2020]), which makes it difficult to run efficiently on mobile devices. Chang et al. [2021] learns CNNs to align and fuse long and short exposure frames for deblurring. They capture frames sequentially, which makes alignment challenging for fast-moving subjects that often involve non-rigid motion. On the contrary, our system captures a pair of blurry-noisy inputs simultaneously. Our alignment only needs to handle the parallax from the camera baseline, which is typically a rigid transform and is easier to produce satisfying alignment results.

Face Deblurring
Compared to natural images, face images are highly structural and consist of rich semantic information such as facial contours and landmarks. Conventional optimization approaches [Hacohen et al. 2013;Pan et al. 2014] search for a reference face to guide the image deblurring. These methods often require careful alignment between the blurry and reference images to estimate accurate blur kernels. Recent CNN-based approaches learn semantic labels [Shen et al. , 2020Yasarla et al. 2020], facial sketches , or 3D facial reconstructions  for deblurring. However, such methods based on facial priors could fail when a subject wears a mask (e.g., the third example in Figure 12). Instead of learning face priors or searching for a reference face from an external database, we propose an end-to-end system that concurrently captures a sharp reference face and learns ML models to fuse the blurry and reference images. and sharp yet noisy faces on UW, we produce the sharp and clean face by matching the color of UW to W, densely aligning the images using PWC-Net optical flow, and feed the W and aligned UW to a UNet-based FusionNet. We use face mask and occlusion mask to reduce the warping artifacts in the fused outputs. Note that the reference image is 2× smaller than the source. We enlarge the reference image for better visualization.

FUSION DEBLURRING
We consider the image acquisition system consisting of a main wide camera W, and a reference fixed-focus ultrawide camera UW. The field-of-view (FOV) are 80°and 107°, respectively, throughout the results in the paper. W and UW are roughly fronto-parallel such that UW's FOV covers W's. The sensor resolutions of both W and UW are 12MP. UW is noisier than W, since modern phones often optimize the sensitivity for the main W camera. Note that the dual W+UW setup is common for all mid-tier and premium smartphones.
Our fusion algorithm takes two images as input, where we treat W as the source and UW as the reference image. When subject motion is detected, we enable dual camera streaming to concurrently capture W and UW raw bursts [Hasinoff et al. 2016]. The exposure time of W is determined by the auto-exposure algorithm for the optimal SNR and sharpness performance in averaged cases, but is usually too long for fast-moving subjects, and leads to blurry faces. The UW image is deliberately captured at a higher shutter speed than W to minimize motion blur, at the cost of more noise and artifacts. We lock the UW's exposure time to be 1 of W's exposure time. We test = 2 and = 4 in this paper. Note that our system design is independent of the choice of raw burst capture. We adopt the raw capture in our pipeline since many phones employ raw burst capture in the post-capture processing pipeline for high-dynamicrange enhancements. Figure 3 shows an overview of our dual camera system pipeline. First, we merge the raw bursts into linear raw images [Hasinoff et al. 2016]. Then, we apply face detection and subject segmentation to determine a face mask and ROI crop that will be deblurred. The cropped W and UW images are then fused through our fusion ML models to generate the result in the linear raw format, as illustrated in Figure 4. Finally, we blend the cropped fusion result with the full-resolution linear raw image and apply post-processing, which includes sharpness enhancement, global, and local tone-mapping,  to output a final 8-bit RGB image. In this section, we will discuss the key components of the fusion algorithm.

Pre-processing
Fusion ROI and face mask. As we focus on faces, we restrict the processing area to a rectangular region-of-interest (ROI) around the largest face. We determine an ROI for fusion by detecting the largest face box from the source image (red box in Figure 5(a)), extends it by a margin to cover face, hairs, ears, and chin (blue box in Figure 5(a)), and round up to a maximum 1536 × 1536 for efficient ML inference (green box in Figure 5). We generate a face mask through a UNetbased ML subject segmenter [Wadhwa et al. 2018] over the fusion ROI to identify the facial pixels ( Figure 5). Finally, we smooth the mask boundaries to avoid sharp transition after blending.
Color matching. While we synchronize the total exposure, i.e., exposure time × total gain, and auto white-balance between W and UW cameras, the source and reference images can still show color mismatches given different sensor characteristics and calibration tolerance. The color mismatch leads to alignment error and undesired color shift in the fusion output. We match the color of reference ref to source by matching the image statistics [Reinhard et al. 2001] in linear space. We first normalize the color space using the following: where is the 3 × 3 color-conversion matrix obtained from camera calibration in image metadata. As we operate in the linear image space, RGB color is calibrated to the same color space after applying . Eq. (1) normalizes the color space of reference image to the source image such that src · ref would look visually similar to src · src . Then, we match the global mean of source and reference images by:ˆr where src and ref are the global color mean of the source image src and normalized reference image ref , respectively. We apply (2) to each RGB channel separately. We choose division instead of subtraction in (2) since the color mismatch between source and reference comes from different sensor sensitivities, which can be approximated through the intensity ratio. Subtracting the RGB means leads to negative intensity values, and clamping-to-zero loses pixel information and de-saturates images.

Image Alignment
As the W and UW cameras do not have the same optical center, the parallax introduces misalignment and occlusion between the source and reference images. To align the reference to source for image fusion, we use PWC-Net [Sun et al. 2018] to estimate optical flow, since it shows similar performance as recent state-of-the-art methods, e.g., RAFT [Teed and Deng 2020], while running faster with less memory consumption [Sun et al. 2021]. As the source image is blurry and the reference image is typically noisy, directly applying PWC-Net to full-resolution images may lead to inaccurate flow estimation. Furthermore, optical flow networks are typically trained on image sizes at 720p, i.e., 1280 × 720, or smaller 2 , where the maximal motion magnitude is typically less than 100 pixels (c.f., [Sun et al. 2021] Figure 8). Our full-resolution image size is 4080 × 3072, where the average motion magnitude between the source and reference images is larger than 300 pixels, and cropping to 1536 × 1536 does not change the motion magnitude that we need to handle. Directly aligning the source and reference images lead to severe distortion as shown in Figure 6. To account for this, we downsample both the source and reference images by 4× and upsample the estimated flow to warp the reference image: where (·) ↓ and (·) ↑ represent the bilinear downsampling and upsampling operator, respectively. The warped reference image is generated via bilinearly resampling W: (4) Figure 6 shows that the flow predicted from 1 4 × resolution images is much cleaner, resulting in better alignment quality.
Optimizing PWC-Net for mobile devices. The original PWC-Net [Sun et al. 2018] uses 7-level pyramids and a DenseNet [Huang et al. 2017] at the prediction level. The on-device inference takes 113 ms on 384 × 512 input images, which is too slow for our use cases. We optimize the architecture of PWC-Net by (1) reducing the number of pyramid levels from 7 to 5, (2) removing the DenseNet structure, (3) limiting the search range of cost volume from 4 to 2, and (4) replacing each 3 × 3 conv layer with a 3 × 3 depth-wise conv layer followed by an 1 × 1 conv layer [Howard et al. 2017]. We re-train the optimized PWC-Net on the AutoFlow dataset [Sun et al. 2021]. Our optimization reduces the inference latency from 113 ms to 13 ms, the memory usage from 600 MB to 34 MB, and the model size from 40 MB to 1.27 MB, which are suitable for mobile platform. While the average EPE on the Sintel training final set increases from 2.91 to 3.73 after our optimization, we find that the warping quality on faces remains on par with the original PWC-Net. We also quantize the model weights from fp32 to fp16 using post-training quantization, which saves another 50% of model binary size without affecting the flow quality.
Occlusion mask. We compute the occlusion mask using the forwardbackward flow consistency check. We first generate the forward fwd and the backward flow bwd using PWC-Net. Then, we project the pixel coordinates from the source image domain to the reference using the forward flow and re-project them back to the source image domain using the backward flow. The occlusion mask occ is defined as the re-projected distance multiplied by a scaling factor as the following: where W is the warping operator, x is the 2D image coordinate on the source, and controls the strength of the occlusion mask.

Image Fusion
We learn a UNet variant network, called FusionNet, for referencebased deblurring. Our FusionNet takes inputs from the blurry source image src , warped reference image ′ ref , face mask face , and occlusion mask occ . Since the focal length of UW camera is nearly half of the W in our camera system, the size of cropped face on the reference image is roughly half of the source image. The face mask is generated from a thumbnail resolution (640 × 480). The optical flow and occlusion mask are computed at the 1 4 × source image resolution. Instead of resizing all input images to the same resolution, we resize the reference image, face mask, and occlusion mask to 1 2 × size of the source to save the memory usage. Model architecture. Our FusionNet illustrated in Figure 24 has N-level pyramids for the encoder and decoder. At each level, the encoder block uses 1 Conv layer, followed by another Conv that uses a stride of 2 to downsample the features by 2. The decoder block at each layer uses 2 Conv layers followed by a bilinear upsampling layer to upsample the features. Except the output layer, each Conv layer is followed by a ReLU activation layer. The features in each encoder block are fed to the decoder block at the same level through skip connections. In the first level, the encoder extracts features from the source image and downsamples them by 2×. In the second level, we concatenate the warped reference image, face mask, and occlusion mask as inputs and extract reference features. The features from source and reference are fused and become the input features to the second pyramid level. In the last decoder block, the upsampled feature is fused with the source feature (before downsampling). We apply another Conv layer to generate a 3-channel image as the output. We also adopt global residual learning such that the output image is a summation of the source image and the model prediction. The output image is represented by: Optimizing FusionNet for mobile devices. We start exploring the architecture configuration from a 7-level FusionNet. While the 7level FusionNet (the numbers of feature channels at each level are 16, 32, 64, 128, 256, 512, and 1024) can produce high-quality results, Fig. 8. Synthetic motion blur kernel. We randomly sample a 2D camera trajectory to generate motion blur kernels for generating synthetic motionblurred images.
the model size is too heavy (102 MB after quantizing to fp16) to be deployed on mobile devices. We first optimize our FusionNet by reducing the number of pyramid levels, and we find the 5-level FusionNet (16, 32, 64, 128, and 256 channels in each level) can generate similar quality results as a 7-level FusionNet. Further reducing the pyramid level or the number of feature channels (e.g., from 16 to 8 at the first level) will degrade deblurring quality significantly. We also explore a similar architecture change as PWC-Net to replace each 3 × 3 conv layer with depth-wise conv layers. However, we also observe quality drop despite the reduction in latency and memory usage. Finally, we use Add instead of Concat to fuse skip connections with the decoder features to avoid extra buffer. Concat requires an extra buffer of × × 32 in the first pyramid level, which takes about 288 MB memory for an input size of 1536 × 1536. Instead, we use Add to reuse the existing buffer for skip connection. This extra 288 MB memory is critical to system stability on mobile phones.
Training data. It is nearly impossible to collect paired blurry input, i.e., face with motion blur, and the ground-truth, i.e., static face of the same scene at the same moment, in a large scale. Therefore, we resort to synthetic data to train our FusionNet. We first capture a pair of W and UW images from static subjects and treat W and UW images as ground-truth GT and reference, respectively. We then generate motion blur kernels by randomly sampling a camera trajectory [Boracchi and Foi 2012] and convolving the ground-truth image with the blur kernel to generate synthetic blurred images as the source. Figure 8 shows examples of our synthetic motion blur kernels. The maximum kernel size is 150 × 150 with a non-linearity of 0.5 (non-linearity = 0 and 1 correspond to linear lines and highly curved trajectories, respectively). We restrict the motion blur to the area covered by the face mask and apply a ghosting effect around the subject boundary through alpha blending between the blurred image and ground-truth, as described below: where is blur kernel, ⊗ denotes 2D convolution, blur = face ⊗ , and is realistic Gaussian-Poisson noise [Foi et al. 2008].
Synthetic highlights. In real images, reflection highlight from the eyes or other accessories often leads to saturated light streaks due to motion blur. These light streaks often saturate and do not follow the linearity assumed by the convolution model in Eq. (7), resulting in poor performance when training our model on synthetic data. To tackle this problem, we add random light dots to the groundtruth image before applying motion blur. We sample the synthetic light dots through uniform distribution over the radius between [1, 3] pixels, and intensity between [2,5]. After blurring, we create realistic light streaks as shown in Figure 9. Training with synthetic light streaks effectively reduces motion blur on light streaks during test time (see Figure 17). In total, we create about 2594 pairs of W and UW images for training data.
Training losses. We adopt the following loss functions to train our FusionNet: -Content loss: We compute the L1 difference between the fusion result and ground-truth: -Perceptual loss: We also compute the weighted L1 difference between the features of the fusion result and ground-truth extracted from a pre-trained VGG19 model [Simonyan and Zisserman 2014]: We use features from conv1-2, conv2-2, conv3-2, conv4-2, and conv5-2, and = 1/2.6, 1/4.8, 1/3.7, 1/5.6 and 10/1.5, respectively 3 . -Color-consistency loss: Even with AE, AWB synchronization, and our color matching step, the reference image may still have a different color than the source image. We observe color-shifting artifacts on hair or forehead in our fusion results. To alleviate this issue, we introduce a color consistency loss by matching the local patch mean between the model output and the source image. Specifically, we apply a Gaussian filter with a large kernel to blur both the model output and source image, and then compute the L1 loss between the smoothed images: where G is Gaussian filter with a standard deviation . We set = 20 in our experiments. The total loss is a weighted sum of the above losses: where we set content = 1, vgg = 2, and color = 1 in our experiments. We train our model using Tensorflow and ADAM [Kingma and Ba 2014] optimizer with decayed learning rate. The model converges after 1M iterations. The PSNR reaches 37.97 dB on the validation set.

Blending
To focus the deblurring on faces and keep the background unchanged, we adopt an alpha blending to merge the FusionNet output and source image, as described below: where the blending mask blending is the face mask subtracted by the occlusion mask and reprojection error map: blending = min( face − · occ − reproj , 0).  We set = 5.0 and = 2.0. The reprojection error map is calculated from the difference between the source image and the warped reference image: We use the reprojection error map and occlusion mask to gracefully fallback error-prone pixels to the source pixels. With the soft blending, the results look natural and artifact-free.

Post-processing
After blending the fusion result with the source linear image, we apply chroma denoising, local and global tone-mapping, and dehazing described in [Hasinoff et al. 2016] to generate the final 8-bit gammacorrected image in the sRGB color space. While FusionNet removes motion blur, the results look slightly over-smooth due to the fact that UW has lower optical resolution than W. To remove mild blur and enhance sharpness, we resort to a single-image sharpening method based on Polyblur [Delbracio et al. 2021], which estimates a Gaussian blur kernel from the input image and approximates the deconvolution with a series of polynomial filters. We modify Polyblur by limiting the support of the kernel estimation step to the face area, so that the estimated kernel accurately represents the local blur on the face area.

Fallback Mechanism
The reference quality, i.e., UW, is critical to the deblur quality. When the reference image shows poor quality, e.g., is too noisy or fails to align to the source image, we observe objectionable artifacts. Our strategy in these cases is to automatically fall back to the source image based on image quality heuristics. We calculate face motion from the center of the face rectangle between adjacent frames in W raw burst. (1) If the motion is too small, we will skip fusion. (2) If the timestamp difference between Wand UW is larger than 20ms, we fall back since the alignment is too challenging for our optical flow. (3) When the sensor gain is higher than a threshold (=160 in our work), e.g., low-light conditions, excessive noise due to insufficient exposure time, we skip fusion.
Finally, we check (4) the difference between the gamma-corrected fusion output and source image over the face area based on a masked mean square error (MSE) described below: If the MSE is smaller than a threshold (0.25 in our work), we fall back to the source image because the fusion result improvement is insignificant.

Hardware acceleration
Our fusion pipeline contains 3 ML models: optical flow (PWC-Net), fusion deblur (FusionNet), and subject segmentation (UNet). To deploy them on the mobile phone, we convert the models into Ten-sorFlow lite format [Abadi et al. 2016] and delegate inference to the accelerators. The SoC in our experiment has a Mali G78 GPU and a neural processing unit (NPU) for efficient convolution [Yazdanbakhsh et al. 2021]. We choose to delegate segmentation to NPU and other operations to GPU. The latter performs more efficiently for the gather operators used by PWC-Net and the large memory footprint required by FusionNet.

ADAPTIVE STREAMING
Our algorithm requires a mobile phone to stream both W and UW cameras. However, continuously streaming both W and UW camera consumes high power (+400 mW) unsustainable for thermal status. Moreover, it requires a large additional memory footprint (+160 MB). To save amortized power, we present an adaptive stream system that dynamically enables and disables UW camera based on real-time motion analysis, as illustrated in Figure 10. Our adaptive streaming system determines whether the subject in the frame demonstrates excessive motion that leads to motion blur, and, if so, streams UW at + 1 to capture the short-exposed reference. The motion analysis and decision need to be made within the duration of 1-frame, i.e., 33ms when the camera streams at 30 FPS. To meet the real-time requirement, we propose a lightweight SVM model that takes feature vectors from camera metadata relevant to motion blur and outputs a binary flag indicating whether to turn on UW camera. We now describe our SVM model and implementation for mobile camera systems.

Real-time Motion Analysis using SVM
Modern ISPs in smartphones offer a temporal denoising block based on real-time multi-frame alignment and merge. We take the alignment map computed from the patch-based inverse Lucas-Kanade algorithm [Ehmann et al. 2018] between the adjacent W frames as the motion flow and use the (1) average flow strength in the face region, and (2) the maximal flow as the input vector for the SVM model. We augment the face motion feature by calculating the (3) average gradient strength in the face area. We include the (4) exposure time in the feature vector due to the high correlation to motion blur. Furthermore, we include (5) sensor gain in the feature vector since excessive noise is detrimental to the deblur quality.  Fig. 10. Adaptive Streaming Design. The adaptive streaming system resides in the camera driver, passing camera metadata to the ML classifier to identify if the human face is suffering from motion blur, and turns on the UW camera dynamically.
If motion blur is insignificant in W, the fusion result appears similar to W, and we prefer to turn off UW to save the power. We leverage this observation to label the ground truth of the output for the SVM model. We collect 1777 pairs of W, UWand the feature vectors consisting of (1)-(5) above at various motion strengths and lighting conditions, and generate the deblur results from the pairs. We then manually examine the deblur output and W, and label positive if the results are visually better than W. Our SVM model achieves 6% and 11% false positive and negative rate, respectively, on the validation set consisting of 500 captures.

Implementation
The Android camera framework (Camera2 API) [cam 2021] allows the camera app developer to stream each camera dynamically, but leads to an 11-frame delay between motion analysis results and the moment when UW starts to stream in our experiment. If the users capture a motion shot within these 11 frames, our system may fail to stream UW for fusion deblurring. Alternatively, we reduce the frame delay to 7 frames by integrating the adaptive stream system into the camera driver. In our experiment over 600K shots by our users, only 8.1% shots were captured during the 7 frames.

CAMERA IMAGE PROCESSING PIPELINE
We describe the requirements of the image signal processing (ISP) pipeline for integrating our fusion deblurring (Section 3) with photo capture. Before the shutter is pressed, fusion deblurring requires careful auto exposure, auto white-balance, and temporal synchronization between W and UW camera in a real-time processing pipeline.
Auto exposure. Auto-exposure in the camera ISP decides a target total exposure time (TET) that represents the scene brightness. TET is factorized into the exposure time and sensor gain , i.e., , TET = [Liba et al. 2019]. Our system determines the and for UW from W camera. To capture facial details, we lock the exposure time of UW times faster than W, i.e., , = 1 , . To match the total scene brightness, we determine the TET for UW by calibrating the sensor sensitivity using a synchronization ratio at ISO100 gain, i.e., = , where = 3.7 in our testing device. Finally, we determine the sensor gain of UW by dividing by the exposure time of UW, i.e., = . We test = 2 and = 4 and find both lead satisfying deblurred results.
Auto white-balance. In the camera ISPs, the linear colors are determined by the color correction matrix ( ) and lens-shading correction ( ) table. We apply Eq. 1 to match . In theory, the sensor-dependent varies by scene illumination. In practice, since faces shown in W often appears near the center in UW, and is nearly fixed at the camera center, we apply the factorycalibrated table to UW and ignore the scene-dependent effect.
Timestamp synchronization. Our fusion deblurring requires the maximal timestamp difference between W and UW camera to be less than 20ms when streaming at 30FPS. In practice, frames may be misaligned due to frame drop, camera timestamp drifts, and different sensor integration time margins. Most SoCs on premium phones support hardware sensor synchronization across multiple cameras and achieve tolerance of less than 1ms. However, we found the synchronization causes preview jank when UW runs in adaptive mode (Section 4), since our SoC only allows syncing UW to W but not the other way, and drops frames in W during camera preview. Therefore, we implemented a software synchronization by calculating the frame time difference between UW and W, and send the difference to UW camera sensor to adjust the frame rate dynamically.
Power-efficient UW camera ISP.. Even with the proposed adaptive streaming system (Section 4), streaming two cameras simultaneously still consumes significant power. While the W camera ISP needs to produce the tone-mapped frames for preview, the UW camera ISP only needs to produce the RAW frames. We take the optimization opportunity by customizing the UW camera ISP. We disable the ISP blocks after RAW, including camera 3A, face detection, demosaic, and RGB-to-YUV conversion.
Dual zero shutter lag burst. We adopt zero shutter lag (ZSL) capture to avoid missing the fleeting moment of motion due to shutter lag. ZSL employs a ring buffer that continuously captures the RAW frame produced by the sensor, then determines the frame for postcapture processing at the shutter time, referred to as the reference frame. We adopt ZSL for both W and UW captures. We pair the frames of the two ZSL ring buffers based on the nearest-timestamp criteria. At shutter time, we determine the reference frame from the W ZSL ring buffer using the algorithm described in [Hasinoff et al. 2016] and then use the reference frame timestamp from W to determine reference frame in UW ZSL ring buffer. Figure 11 shows the results of our fusion deblurring. Under challenging motions, lighting conditions, and face poses, our method successfully reconstructs facial details without objectionable artifacts. We can see that in these examples, the subjects wear glasses and masks (bottom row), or move so fast that the facial features are barely identifiable (left in the bottom row and right in the top row). Our method robustly outputs clear faces in these difficult conditions. While our training images are synthetic and spatiallyinvariant blurred images, our FusionNet still generalizes well to  Fig. 11. Our results. In each pair, we show blurry input at left, and our deblurred result at right. We include the cases of challenging and fast motion from day-to-day scenarios, over subjects of diverse skin colors, face poses, ages, and genders, and under indoor, outdoor, and low-light conditions. The insets clearly show that our method recovers authentic and sharp details, making the captures usable and memorable. Please refer to the supplemental material for a complete list of results.  We calculate the no-reference image quality metric using NIMA [Talebi and Milanfar 2018]. Our method outperforms others and achieves the best perceptual quality. (b) We compute the ΔNIMA by subtracting the NIMA score before and after the deblurring. Some methods lead to NIMA score regression (ΔNIMA< 0 within one standard deviation range), while our method robustly shows improvement.

RESULTS
dynamic motion blur. For example, in the left example in the first row of Figure 11, the light streaks on the eyes and teeth show different orientations due to the head rotation. Our method can reduce the motion blur by fusing the details from the reference image. The full-resolution outputs and intermediate results are provided in the supplementary material.

Dataset
To the best of our knowledge, there are no datasets that contain triplets of blurry, noisy, and sharp images. Therefore, we collect our own test dataset for evaluation. We implement our system on an Android-based mobile camera to capture moving subjects under diverse scenes and lighting conditions, including cloudy days, evening, and indoor scenes. Subject motions include walking, running, jumping, biking, skateboarding, working out, and playing various sports: basketball, badminton, table tennis, volleyball, etc. We also include subjects of diverse ages and ethnic groups. Among the 6080 diverse test images collected by us, 2021 images trigger our adaptive streaming (Section 4), and 1783 images pass our fallback checks (Section 3.6) and trigger our system to apply fusion deblurring.

Comparisons
Comparisons with academic approaches. We compare our results to the recent deblurring approaches listed below: Note that the existing approaches are typically trained on 8-bit gamma images, e.g., GoPro dataset [Nah et al. 2017], while our training images are 16-bit linear before any tone-mappings. It is nontrivial to re-train or fine-tune existing models with our training data. Therefore, we use the pre-trained model released by the authors of each method for fair comparisons.
We collect 100 representative images that cover diverse scenes and identities and generate 8-bit RGB images from W raw burst using [Hasinoff et al. 2016]. Note that several CNN-based methods in other works require excessive memory usage. To avoid out-ofmemory, we further crop test images to fusion ROI defined in Section 3.1 and resize the cropped images to 768 × 768 for inference. For multi-frame and video-based methods, we create a sequence of 8-bit RGB images by applying [Hasinoff et al. 2016] to each frame in the burst. The sequence length is between 5 to 9 frames. Figure 12 shows the visual comparisons. Existing methods typically fail to remove large motion blurs and produce overly smooth results or artifacts such as ringings or halo. In contrast, our method generates sharper and clearer faces with plausible facial details. We provide all the visual comparisons in the supplementary material.
For quantitative evaluation, as there are no ground-truth images in our testing dataset, we rely on visual comparisons and noreference quality metrics using NIMA [Talebi and Milanfar 2018], which learns a CNN to predict human perceptual quality 4 . Figure 13 1/120 s 1/120 s 1/120 s 1/60 s plots the average NIMA scores and average NIMA difference between deblurred and source images of these methods. Our result achieves the best perceptual scores among others in Figure 13(a). In Figure 13(b), existing approaches except PVDNet [Son et al. 2021] show negative NIMA difference on 34.4 percentile, i.e., within 1standard deviation. Our method is more robust and achieves consistent perceptual improvement. Table 1 reports the per-frame inference latency on 768 × 768 input images. We use NVidia Quadro P5000 GPU on a desktop. Note that the implementation libraries are different for these methods, and our model is optimized for mobile GPU inference. Since our source code and training data involve proprietary information, we are not able to release them. Instead, we release the test images used for comparisons at the project website 5 .
Comparisons with commercial products. We further compare our method against commercial products using post-capture editing tools from the Shake Reduction feature in Adobe Photoshop CC 2021 and Samsung Gallery's Remaster function, as well as straightout-of-camera capture from Apple iPhone 13 Pro. The first two methods are single-image deblurring, while ours and iPhone are processed at capture time. There is no software to connect iPhones and Android devices for synchronized photo capture. Therefore, we use our Android device and iPhone 13 Pro to capture the same moving subject side-by-side until we find the best-synchronized pairs. In Figure 14, the iPhone produces blurry results on the fast-moving subject. Photoshop Shake Reduction cannot reduce blur effectively and leaves ringing artifacts in the output (bottom row). Samsung Remaster shows promising deblurred results, but misses the thin edges near the glasses frame (top row) and cannot remove large motion blur (bottom row). Their results also look overly smooth and lose many facial details. It takes about 8 to 10 seconds for the Samsung Remaster function to process a 3072 × 4080 image. Compared 5 https://www.wslai.net/publications/fusion_deblur to other commercial products, our system generates artifact-free results with the best details and sharpness at low latency.

System Performance
Latency and memory. We implement our system on Google Pixel 6 released in 2021. The SoC consists of Mali-G78 GPU, Cortex multi-core, and an NPU. The latency overhead is 463ms, which consists of preprocessing, alignment, fusion, and the Polyblur in post-processing, as reported in Table 2. We exclude merging W burst raw and tone-mapping from the overhead since they are already required in the single camera processing without our fusion deblur. Merging UW burst raw is scheduled concurrently with merging W burst and does not take additional latency. Our system's peak memory usage happens at the FusionNet inference stage, which takes an extra 264 MB compared to the peak memory usage in single camera processing pipeline.
Power. When dual camera streaming at 30 FPS, the power overhead is 468mW, which splits between power rail from UW sensor (332mW) and SoC (136mW) for RAW data transaction between Source Reference Ours w/o L color Ours Fig. 15. Effectiveness of color-consistency loss. Without the colorconsistency loss L color , inconsistent color may be transferred from the reference image to the fusion result, e.g., green color on hair in both cases (zoom-in to see the details).
DRAM and the UW camera system. On average, our adaptive system streams UW 1.8% of the camera session time, which results in 8.4mW amortized power cost.

Ablation Studies
In this section, we validate the contributions of key design choices of our fusion algorithm, including the color-consistency loss, synthetic light streaks in training data generation, and compare our method with single-image deblurring. Additional ablation analysis are provided in the supplementary material, including analysis on the perceptual loss, mask smoothing, Polyblur, optical flow, and how our FusionNet utilizes each input for deblurring.
Color-consistency loss. As W and UW use different camera sensors, there will be inevitable color mismatch even after our color matching and AE synchronization. Without the color-consistency loss L color , inconsistent color may be transferred from the reference image to the fusion result, such as green color on the hair of the two examples in Figure 15. By training with the color-consistency loss, our FusionNet learns to keep the color of the source image after deblurring.
Single-image vs. reference-guided deblurring. To validate the importance of the reference image, we train our FusionNet by removing the reference image from the input. The configuration effectively becomes a single-image deblurring approach. As shown in the third column of Figure 16, this approach fails to remove large motion blur and produces artifacts on the output. The eyeglasses in the top row show obvious ringing artifacts, and the face in the bottom row is still significantly blurred. On the contrary, our dual camera fusion approach restores sharp faces with rich details. our model can reduce the light streaks on eyeballs or glasses reflection and suppress ringing artifacts, providing more visually pleasing deblurred results.

Limitations
The quality of our fusion results mainly depends on the quality of UW images. Our method may fail to restore a clear face under the following situations: 1) extreme low-light: As the UW image is captured at a faster exposure time of W, the UW image may appear excessively noisy under the extreme low-light scenes. Our FusionNet may transfer the noise from the reference image and lead to painterly artifacts, as shown in Figure 18. 2) Small face: When the face is too small, UW may not be able to capture facial details for restoration, as shown in Figure 19. Our method is not able to recover a sharp face under such a case. In the above cases, the fallback mechanism in Section 3.6 would trigger to avoid artifacts. Figure 18 and 19 are created by disabling the fallback mechanism. When the face mask includes part of the clothes, the blending boundary may be visible if the clothes have regular texture, as shown in the baby's apron in the first row of Figure 11 . This issue could be mitigated by applying a stronger mask boundary smoothing or a better face segmentation that contains face regions only. Finally, our system currently applies deblurring to a single face (the largest face) due to the device computing budget. A multiface deblurring solution will require more justifications on system stability and is considered as our future work.

CONCLUSION
In this work, we have presented a robust system to address motion blur on faces in portrait photography. Our system runs efficiently on Google Pixel 6 smartphones at the capturing time and produces visually more pleasing results than state-of-the-art single-image, multi-frame, video, and face-specific deblurring methods through extensive evaluations. By leveraging the dual camera system available on major premium phones, we have shown that combining optical flow for alignment, residual UNet for image fusion, and training on synthetic data can address the challenges that would be difficult to approach through a single camera setup. We have specified a list of system requirements including adaptive stream and camera ISP to make the dual camera fusion practical on mobile devices. We hope our work encourages further progress on reference-based image deblurring.

APPENDIX -ADDITIONAL ABLATION STUDIES
Perceptual loss. Without the perceptual loss L vgg , our deblurred results often look overly smooth. Since motion blur cannot be completely removed, the results contain ringing or blurriness around the eyeglasses frame, as shown in Figure 20(b). By training with the perceptual loss, our FusionNet generates sharper results with richer facial details, as shown in Figure 20(c).
Face mask smoothing. We apply an alpha blending to blend our deblurred face with the full-resolution source image. Without the mask boundary smoothing, a clear border is visible in the deblurred image, as shown in Figure 21. The smoothed mask for alpha blending helps to avoid such a sharp transition.
Polyblur. As our fusion results may look overly-smooth, we adopt the Polyblur [Delbracio et al. 2021]   out-of-focus, while our FusionNet can reduce large motion blur caused by subject motion. Figure 22 shows an example that Polyblur improves the detail sharpness around the eyes.
Optical flow. The original PWC-Net may generate noisy flow and cannot properly align the face due to the resolution mismatch between training and inference images (the second column of Figure 23). By estimating flow from 1 4 × resolution input images, we obtain a more robust flow to align face more accurately, as shown in the third column of Figure 23. We further optimize the architecture to reduce latency and memory footprint. As shown in the fourth column of Figure 23, the optimized PWC-Net generates similar flow and fusion results as the original PWC-Net.
Understanding FusionNet. To understand how our FusionNet utilizes source, reference, face mask, and occlusion mask, we remove one of the input tensors by setting the tensor to zeros, and show the results in Figure 24 and 25. We observe the following from this experiment: (1) Without the source image, the result looks all black without any facial details (Figure 24(e)). (4) The contribution of the occlusion mask is mainly at face boundaries or background. Without the occlusion mask, the face boundary may look sharper, e.g., the edge of face mask in Figure 25. However, some warping artifacts on the background could be fused to the output when the face mask falsely includes some background pixels, e.g., the straight lines in the wall of Figure 25. From our analysis, the FusionNet does not blindly copy everything from the reference image to the output. Instead, the FusionNet learns to apply fusion only when the source and reference have correspondences, e.g., the same face, within the face mask. While using occlusion mask could sacrifice the sharpness on face boundary, it is crucial to prevent any unnatural artifacts in the results. Minimizing artifacts is an important goal of our system, and we are willing to be conservative in the amount of sharpening. The original PWC-Net may generate inaccurate flow due to the resolution mismatch between training and inference images. By estimating flow from 1 4 × resolution input images, we obtain more robust flow to align face regions more accurately. We further optimize the architecture to reduce latency and memory footprint. Our optimized PWC-Net can generate similar flow and fusion results as the original PWC-Net.