{'fraction_non_alphanumeric': 0.029135686168960066, 'fraction_numerical': 0.013390303805134071, 'mean_word_length': 4.930566640063847, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and Interpolation

Existing works address the problem of generating high frame-rate sharp videos by separately learning the frame deblurring and frame interpolation modules. Most of these approaches have a strong prior assumption that all the input frames are blurry whereas in a real-world setting, the quality of frames varies. Moreover, such approaches are trained to perform either of the two tasks - deblurring or interpolation - in isolation, while many practical situations call for both. Different from these works, we address a more realistic problem of high frame-rate sharp video synthesis with no prior assumption that input is always blurry. We introduce a novel architecture, Adaptive Latent Attention Network (ALANET), which synthesizes sharp high frame-rate videos with no prior knowledge of input frames being blurry or not, thereby performing the task of both deblurring and interpolation. We hypothesize that information from the latent representation of the consecutive frames can be utilized to generate optimized representations for both frame deblurring and frame interpolation. Specifically, we employ combination of self-attention and cross-attention module between consecutive frames in the latent space to generate optimized representation for each frame. The optimized representation learnt using these attention modules help the model to generate and interpolate sharp frames. Extensive experiments on standard datasets demonstrate that our method performs favorably against various state-of-the-art approaches, even though we tackle a much more difficult problem.

consecutive frames can be utilized to generate optimized representations for both frame deblurring and frame interpolation. Specifically, we employ combination of self-attention and cross-attention module between consecutive frames in the latent space to generate optimized representation for each frame. The optimized representation learnt using these attention modules help the model to generate and interpolate sharp frames. Extensive experiments on standard datasets demonstrate that our method performs favorably against various state-of-the-art approaches, even though we tackle a much more difficult problem. The project page is available at https://agupt013.github.io/ALANET.html.

INTRODUCTION
Motion blur and low frame-rate are often commonplace in videos captured by mobile devices, whether hand-held or on a moving platform. The reasons vary, including low shutter frequency, long exposure times, and the movement of the device itself [10,28]. These factors limit the quality of videos captured. As vast majority of video media is captured using mobile cameras these days, it calls for improved quality of the videos captured by these devices. Enhancing video quality requires restoring the degradation caused by motion blur along with increase in the frame-rate at which video is captured for temporal smoothness.
Most existing approaches have addressed the problem of high frame-rate sharp video generation by frame deblurring and frame interpolation, separately. In [10], separate models are used to deblur input frames and to interpolate between frames. The phenomenon of motion blur and frame-rate at which video is captured are related. Thus, a joint formulation is needed when addressing the task of high frame-rate sharp video generation from a low frame-rate blurry video. Recently, [25] studied the problem of joint video deblurring and interpolation. Here, authors proposed to use pyramid deep models to deblur and interpolate along with a pyramid of convolutional Long-Short Term Memory (LSTM) to capture temporal smoothness. However, these methods assume that all the input frames are blurry, which is often unrealistic because the quality of a video usually varies non-uniformly over time.
In this paper, we introduce a novel architecture Adaptive Latent Attention NETwork (ALANET) which aims to jointly deblur and interpolate frames from a poor quality video input without an assumption that all input frames are blurry. Specifically, we construct a Adaptive Latent Attention module that leverages the latent space with attention mechanisms to generate high frame-rate sharp video. ALANET has a U-Net variant [24] as it's backbone, combined with the proposed attention module. Similar to U-Net, we utilize contracting path (encoder) of the network for latent space representation and expanding path (generator) for video generation. However unlike U-Net, we do not pass the bottleneck features extracted from the encoder directly to the generator. We introduce our proposed adaptive attention module to modulate and interpolate the latent features for deblurring and interpolating frames from the input video. Figure 1 illustrates the concept of proposed adaptive attention module. Given a set of input blurry and sharp frames, their projection in latent space can be modulated and interpolated using Adaptive Latent Attention module, to generate optimized representations for sharp frames. These modulated and interpolated latent representations are then used by the generator to synthesize the high frame-rate sharp video.

Approach Overview
An overview of our approach is illustrated in Figure 2. Given a low frame-rate poor quality input, our objective is to generate a high frame-rate sharp video. Our proposed architecture, ALANET, consists of three modules: the frame encoding network E, the Adaptive Latent Attention network M, and the high frame-rate sharp video generator G. We modulate and interpolate the frame features by applying self-attention and cross-attention on channels of the latent features of consecutive frames using our proposed adaptive attention module. Self-attention on the feature space helps the model to focus on important features of the same frame whereas cross-attention helps the model to retrieve information from neighbouring frames that can be useful for either deblurring or interpolation tasks. In turn, the Adaptive Latent Attention module will give less importance to the neighbouring frame feature if the input is a sharp frame, and utilize this information from the neighbours if input frame is blurry. Hence, our proposed approach is able to deblur and generate high quality interpolated frames using selfattention and cross-attention on frame representations. To the best of our knowledge, our approach is the first work to exploit the ability of learning optimized latent representation for generation of high frame-rate sharp video using self-attention and cross-attention.

Contributions
The key contributions of our proposed framework are summarized as follows.
• We introduce a novel framework ALANET, Adaptive Latent Attention Network, designed to jointly deblur and interpolate for high frame-rate visually sharp video generation.
• This is the first work to generate high frame-rate sharp video from low frame-rate poor quality video by applying attention in the latent space without any assumption on the uniformity of blurriness in different frames of the video.
• Our framework demonstrates consistently effective results on two datasets, the benchmark Adobe240 and crawled YouTube240 with better or at par performance with state-of-the-art in both deblurring and interpolation tasks.

RELATED WORK
Our work relates to research in video deblurring, video interpolation, attention model, and joint video deblurring and interpolation.
In this section, we discuss some representative methods closely related to our work (see Table 1).
Video Deblurring. Inversion of motion blur is an ill-posed problem [21,23]. Recent works have used deep learning based methods to solve this restoration problem either using a single frame [26,27] or multiple frames [7,10,13,18,26]. [5] attempts to deblur a video by exploring similarity between the frames of the video and exploiting sharp patches of neighbouring frames. DeBlurNet [26] proposes to use consecutive frames stacked as input to generate a single clean central frame. ESVR [30] tries to align the features of multiple frames using a temporal and spatial fusion module for feature fusion from different layer to deblur a video. [12] proposes an integrated model to jointly predict the defocus blur, optical flow and latent frames. [8] proposed a spatio-temporal recurrent neural network that enforces temporal consistency between neighbouring frames. [35] proposes a spatio-temporal recurrent architecture with dynamic temporal blending mechanism. In contrast, we do not estimate any extra information like optical flow (which can be noisy and computationally heavy) in our approach and rely on proposed attention model to generate high frame-rate sharp videos.
x L ] using encoder network E. Adaptive Latent Attention module M utilizes combination of self-attention and cross-attention on X V to generate optimized representations for deblurring ( Z S ) and interpolation ( Z S ). These optimized representations are used by the generative network G to synthesize deblurred frames (S 1 , S 3 , · · · , S N −1 ) from Z S and interpolated frames (S 2 , S 4 , · · · , S N ) from Z S , thereby generating a high frame-rate video S = [S 1 , S 2 , · · · , S N ].
Video Interpolation. Many of the existing approaches [3,4,9,15,17,36] for frame interpolation use optical flow estimation between input frames. Consequently, the quality of estimated optical flow governs the quality of frame interpolation. Recent learning based methods have demonstrated effectiveness in frame interpolation tasks. A direct application of convolutional neural networks (CNNs) for intermediate frame synthesis is presented in [16]. Some methods [19,20] apply CNNs to estimate space-varying and separable convolutional kernels for synthesis using neighbourhood pixels. [1] proposes to generate videos by learning optimized representation by a non-adversarial approach and then interpolating between the optimized latent representation of two frames to synthesize central frame. However, they average the latent representations of two frames for frame interpolation which often generates a blurry image. Unlike these methods, our approach utilizes adaptive attention in the latent space for interpolation.
Attention Model. Attention mechanism has garnered a lot of interest due to their learnable guidance ability. With pioneering work in language translation [29], variations of attention mechanism have shown promising results in object recognition [2], image generation [32] and image super-resolution [33]. Residual channel attention mechanism for super-resolution is introduced in [33]. Authors in [31] used different length sequences to deblur the center frame and attention is applied on different outputs to generate a single central frame. Recently, variations of attention models are proposed for video deblurring [31] and video interpolation [6]. In [6], attention is applied channel-wise on concatenated down-shuffled frames for video interpolation. In contrast to our work, where we apply attention in latent space, the existing methods employ attention for video deblurring and interpolation tasks in pixel space.
Joint Video Deblurring and Interpolation. Joint video deblurring and interpolation still remains a challenging problem. [10] proposed DeBlurNet, to deblur, and InterpNet, for interpolating input frames in a jointly optimized cascade scheme to generate sharp slow motion videos using blurry input. Blurry Video Frame Interpolation proposed in [25] uses pyramid structure to deblur and interpolate along with a pyramid convolutional LSTM to capture temporal information. However, both these methods strongly assume that all the input frames are blurry. We relax this assumption to address a more difficult problem where we do not know which input frames are blurry and where to interpolate. Hence, the proposed ALANET framework is self-sufficient to make decisions on which frames to deblur using information from neighbouring frames.
with L frames, we aim to generate a high frame-rate sharp video S = [S 1 , S 2 , · · · , S N ] with N frames, where N > L. Our objective is to deblur and increase the frame-rate of the given input video V.
We propose to generate a high frame-rate video by adaptive attention modeling (see Section 4.2) of the feature representations of input video frames in the latent space. Our hypothesis is that in latent space, information from neighbouring frames can help learn optimized representations for deblurring and interpolation. Thus, the proposed Adaptive Latent Attentive model transforms input blurry frame representation (X V ) to the optimized representations (Z S ∈ R H 1 ×W 1 ×C 1 ×N ) for deblurring and interpolation in the latent space given by where z 2i is the representation for a deblurred frame S 2i , and z 2i+1 is the representation for an interpolated frame between S 2i and S 2i+2 , i.e., S 2i+1 . We denote all latent representations for deblurred frames by Z S and for interpolated frames by Z S . These optimized representations Z S = Z S Z S are used to deblur and interpolate sharp frames to generate a high frame-rate video.

ALANET: ADAPTIVE LATENT ATTENTION NETWORK
In this section, we describe the proposed framework, ALANET, in detail. Our framework consists of three components: the encoder E, the Adaptive Latent Attention module M and, the generator G. We use the encoder module to extract latent representation for each input frame. The Adaptive Latent Attention module generates optimized representations for frames to reduce blur and to interpolate frames, simultaneously. Finally, the optimized representations are used by the generator to synthesize a high frame-rate sharp video. Our overall framework is shown in Figure 2.

Latent Representation of Frames
The encoder E is a trainable convolutional neural network which projects the input video into a latent representation for each frame.
Here, x i ∈ R H 1 ×W 1 ×C 1 is the latent representation corresponding to V i . The representations generated by the encoder E are used by the Adaptive Latent Attention module M to generate optimized representations for deblurring and interpolation.

Adaptive Latent Attention
The latent representation of a frame generated by the encoder may not be optimized as all the channels of the input representation are not equally important for generation task. Also, since frames of a video are temporally correlated, their latent representation can be leveraged to extract information from neighbouring frames to generate an optimized representation for deblurring and interpolation.
To extract important information from the latent representation of the given frame and utilize the information from the neighbouring frames, we propose an Adaptive Latent Attention module M. The proposed module M applies attention on the input latent representations to generate the optimized representations for deblurring and interpolation. This module takes two latent representations (x i , x j ∈ R H 1 ×W 1 ×C 1 ) as input, where H 1 ×W 1 is dimension of each feature in C 1 channels of the latent representation. A combination of self-attention M S and cross-attention M C is then used to generate latent representations to jointly deblur and interpolate between two consecutive frames in an adaptive manner.
The basic building block of the attention mechanism is the channel attention function F . It computes attention weights of each channel in the latent representation. As in [33], the channel-wise global spatial information is extracted using global average pooling to condense input features to a channel descriptor. Then, a gating mechanism is applied to learn non-linear interactions and correlation between multi-channel features such that F : Figure 3 shows the self-attention M S and cross-attention M C modules along with the basic building block F for computation of the channel attention.
Self-Attention (M S ) correlates different channels of the latent representation of a frame in order to generate an informative representation. This is achieved by computing attention weights for each of the channels of the input representation followed by elementwise multiplication of the channels with their attention weights. This self-attention on x i can then be expressed as in (3).
Cross-Attention (M C ) provides attention weights for each channel of the latent representation x j conditioned on another latent representation x i . Cross-attention leverages information from other frames to generate a conditional representation. The conditional representation provides insight on what information is useful from other frames. This cross-attention on x j given the input x i can then be computed as in (4).
Note that, ⊗ in (3) and (4) represents element-wise multiplication, Deblurred and Interpolated Representations. A combination of self-attention and cross-attention modules is employed to obtain optimized latent representations for deblurring and interpolation. Given a window W, the optimized latent representations Z V = [ z 1 , z 2 , z 3 , z 4 , · · · , z N ] for a high frame-rate video S is computed as follows: where Q denotes integer values in [ i − 0.5W, i ) ( i, i + 0.5W ], z 2i is the optimized representation for deblurred frame S 2i and z 2i+1 is the optimized representation for the interpolated frame between S 2i and S 2i+2 . As defined by (5), an optimized representation z 2i for sharp output S 2i is computed using self-attention on i t h input representation x i and cross-attention of all the remaining input latent representation x j in a neighbourhood of W frames. Cross-attention is computed in a temporal window of W frames as the significant information for deblurring and interpolation is available in neighbouring frames compared to temporally distant frames. Similarly, a latent representation z 2i+1 for interpolated frame S 2i+1 between S 2i and S 2i+2 is given by (6), where we consider self-attention on each latent representations x i and x i+1 , and cross-attention for each representation conditioned on the other.

High Frame-Rate Video Generation
To generate a high frame-rate video from blurry inputs, we employ a generative neural network G that transforms the optimized representations to a sequence of frames. The optimized representations generated by the adaptive attention module M are used by generator G to synthesize deblurred frames as well as interpolate between frames represented by S = G([ z 1 , z 2 , z 3 , z 4 , · · · , z N ]) where z 2i and z 2i+1 are optimized representation used to deblur and interpolate frames S 2i and S 2i+1 , respectively.

Network Architecture
In this section, we describe the network architecture used for different modules in the proposed ALANET framework.
Encoder-Generator Network. A variation of U-Net [9] is employed to design the backbone network for the proposed framework. The contracting path is used as the encoder network E and the expansive path is used as the generator network G. The encoderdecoder network also retains the skip-connections as in the original U-Net architecture [24]. However unlike the U-Net architecture, our proposed Adaptive Latent Attention module M is introduced after the bottleneck to optimize the latent representations before they are fed to the generator G.
Adaptive Latent Attention Network. In order to make the generator model, G, focus more on informative features, we exploit the inter-dependencies within frame feature (self-attention) and across frame features (cross-attention). The basic building block of self-attention and cross-attention is the attention weight computation module, F . We adopt the channel attention module as in [33] for F . This channel attention module first extracts the channelwise global spatial information into a channel descriptor using global average pooling. Then, a gating mechanism is applied to learn non-linear interactions and non-mutually-exclusive relationship between multi-channel features [33]. Unlike self-attention for super-resolution in [33], we also employ cross-attention between consecutive features to learn interactions between these features for deblurring and interpolation.

EXPERIMENTS
In this section, we first introduce the benchmark datasets, and evaluation metrics. Next, the model used for generation of blurry training data is described. Finally, extensive experiments are shown to demonstrate the effectiveness of our proposed approach in generating high frame-rate sharp videos.

Datasets and Metrics
We evaluate the performance of our approach using publicly available Adobe240 [26] dataset which has been used in many prior works and a dataset crawled from YouTube as in [25].
Adobe240 Dataset. This dataset contains 118 videos captured at 240 frames per second (fps) with the resolution of 1280 × 720 . We choose 110 videos for training and remaining 8 for evaluation following the split provided in [9] for fair comparison.
YouTube240 Dataset. We download 60 random video videos captured at 240fps from the YouTube website to construct an evaluation dataset similar to that used in [25]. The resolution of the downloaded video is 1280 × 720. For this dataset, we train the model in Adobe240 but test on YouTube240 without any fine-tuning.
Dataset Preparation. For Adobe240 [26] and crawled YouTube240 dataset, low frame-rate poor quality videos of 30fps are generated using process described in section 5.2. All the frames are resized to 640 × 352 for training and evaluation purposes.  Top row consists of the input blurry frames and the missing frames faded. We show two high frame-rate videos generated by our proposed method (bottom row) and compare it with the state-of-the-art BIN 4 (middle row). ALANET is able to generate superior quality high frame-rate video.

Implementation Details
Our framework is implemented in PyTorch [22]. All the experiments are trained for 200 epochs with a batch size of 2. We use ADAM [14] optimizer with initial learning rate of 0.0001 and weight decay 5 × 10 −4 . The learning rate is reduced by a factor of 10 after 100 and 150 epochs. The proposed framework takes a 30fps blurry video as an input and generates a 60fps sharp video.
Blurry Video Formation. Camera shutter frequency affects degradation due to motion blur in each frame of a captured video. A low shutter frequency may not be able to capture temporal smoothness and hence generate blurry frames. To simulate the motion blur, we approximate the blurry frame as a discrete averaging of sharp frames within an overlapping window as defined in [9,10,26]. Let 2τ + 1 be the number of sharp frames between two blurry frames and β be the rate at which frames are captured. Then, a blurry frame V i is approximated as: where, S k 's are the sharp frames in the given video. Since we do not assume that all the input frames are blurry, we average 11 consecutive frames randomly using (7) on a sharp video to generate a poor quality video with low frame-rate. Training and Testing Protocol. During training, random blurry frames are generated on-the-fly by averaging 11 frames as defined in (7). The 5 t h and 9 t h sharp frames are considered as the groundtruth for deblurring and interpolation, respectively. The framework is jointly optimized for deblurring and interpolation using Adaptive Latent Attention Network. During testing, a low frame-rate (30fps) poor quality video is used as an input to the trained model and a high frame-rate (60fps) sharp video is generated.
Objective Function. Our objective function consists of a ℓ 1 pixel reconstruction loss 1 and the perceptual loss [11] defined as follows.
Here, L r = i |G i − S i | 1 denotes ℓ 1 reconstruction loss with G i being the ground-truth frame corresponding to the generated frame S i . L p denotes the perceptual loss computed using a pre-trained VGG16 network [11], and λ is a hyper-parameter. We use λ = 0.2 for all our experiments. Figure 4 shows some examples of high frame-rate videos generated using the proposed method and state-of-the-art BIN 4 [25] given a low frame-rate video (top row). From Figure 4a, it can be seen that our approach is able to tackle the motion blur introduced due to the object motion (car in the bottom left corner for this particular example) along with the blur produced by averaging of consecutive sharp frames. As our approach is extracting information by applying attention on latent representation of input frame, our method is able to deblur and interpolate visually more appealing videos. In Figure 4b, the last two frames of middle and bottom row show that the proposed method is able to deblur and interpolate visually good quality frames whereas BIN 4 generates a blurry interpolated frame. As the BIN 4 utilizes the deblurred frame to interpolate, the error from deblurred frame may propagate during interpolation and hence produce a blurry interpolated frame as shown in Fig 4b (middle row, last frame). Our approach overcomes this by generating optimized representation using attention mechanisms, which extracts relevant information from neighbouring frames in the latent space for both deblurring and interpolation.

Quantitative Results
Our proposed method performs joint deblurring and interpolation.
There are several methods that only solve the tasks of either deblurring or interpolation. We compare our proposed approach with these state-of-the-art methods that either perform deblurring or interpolation [3,4,9] given an input blurry video. We also compare ALANET with two recent approaches where deblurring and interpolation is performed jointly [10,25]. Quantitative result comparison with these baselines are shown in Table 2.
Results on Adobe240 Dataset. For deblurring task on Adobe240 dataset, we report a relative improvement of 1.04dB in the average PSNR value and 2.09% improvement in SSIM metric when compared to [25]. Our method achieves 32.98dB average PSNR in interpolation task as opposed 32.51dB reported by state-of-the-art method BIN 4 [25]. Overall, for the joint task of deblurring and interpolation the proposed method achieves relative improvement of 2.3% in average PSNR and 1.04% in SSIM index against BIN 4 . It can be observed that BIN 4 and ALANET both jointly formulate the deblurring and interpolation tasks which helps to outperform [10]. We again highlight that our method does not know which frames are blurry or where to interpolate, unlike BIN 4 [25].
Results on YouTube240 Dataset. We evaluate the performance of our model trained on Adobe240 dataset for deblurring and interpolation on YouTube240 dataset. For this experiment we crawled 60 videos from YouTube to create this dataset following authors in [25]. However, we do not have the same set of videos as in [25] as the list of videos is not publicly available. From Table 2, it can be observed that network trained on Adobe240 performs at-par when evaluated on YouTube240 dataset with average PSNR of 35.89dB and SSIM index of 0.9504 for joint deblurring and interpolation.

Ablation Study
In this section, we investigate the contribution of self-attention and cross-attention in the proposed approach. First, we study the impact of self-attention on video deblurring and interpolation. We remove the cross-attention M C terms from (5) and (6) and train the network using only self-attention in the latent space. Secondly, we study the impact of cross-attention in absence of self-attention by removing M S terms from (5) and (6) for training the network. Figure 5 presents the qualitative results of the ablation study. It can be observed that the network trained using only self-attention produces inferior results as compared to that of using only crossattention. The network trained with only self-attention module assumes that all the information to deblur and interpolate resides in a single frame and discards the temporal information available in consecutive frames. This loss in information results in poor quality frame when using only self-attention. On the other hand, using only cross-attention produces better results than using only self-attention module as it exploits the available temporal information by applying cross-attention on latent representation of the consecutive frames.
The quantitative results of impact of different attention mechanisms are shown in Table 3. Network trained on only cross-attention achieves improvement of 0.38dB PSNR as compared to using only self-attention for deblurring. However, for interpolation there is improvement of 1.90dB when using only cross-attention as, unlike self-attention, it exploits the temporal information available from neighbouring frames. From Table 3, we can observe that ALANET performs best as it extracts quality information from the latent representation by exploiting combination of self-attention and crossattention for deblurring and interpolation.

CONCLUSION
We present an Adaptive Latent Attention Network (ALANET) for generating high frame-rate sharp videos with no knowledge that either an input frame is blurry or not. The proposed approach employs self-attention and cross-attention mechanism in the latent representations of input video frames for deblurring and interpolation. Specifically, the self-attention module extracts information local to the input frame and the cross-attention module exploits the temporal relationship from latent representations of neighbouring frame. Using combination of self-attention and cross-attention our approach is able to generate high frame-rate sharp video. Experiments on standard datasets show the efficacy of our proposed attention module in task of joint deblurring and interpolation over state-of-the-art methods.