{'fraction_non_alphanumeric': 0.03873498406472175, 'fraction_numerical': 0.01908147421753698, 'mean_word_length': 4.783317580340265, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation

Video interpolation increases the temporal resolution of a video sequence by synthesizing intermediate frames between two consecutive frames. We propose a novel deep-learning-based video interpolation algorithm based on bilateral motion estimation. First, we develop the bilateral motion network with the bilateral cost volume to estimate bilateral motions accurately. Then, we approximate bi-directional motions to predict a different kind of bilateral motions. We then warp the two input frames using the estimated bilateral motions. Next, we develop the dynamic filter generation network to yield dynamic blending filters. Finally, we combine the warped frames using the dynamic blending filters to generate intermediate frames. Experimental results show that the proposed algorithm outperforms the state-of-the-art video interpolation algorithms on several benchmark datasets.


Introduction
A low temporal resolution causes aliasing, yields abrupt motion artifacts, and degrades the video quality. In other words, the temporal resolution is an important factor affecting video quality. To enhance temporal resolutions, many video interpolation algorithms [2-4, 12, 14, 16-18, 20-22] have been proposed, which synthesize intermediate frames between two actual frames. These algorithms are widely used in applications, including visual quality enhancement [32], video compression [7], slow-motion video generation [14], and view synthesis [6]. However, video interpolation is challenging due to diverse factors, such as large and nonlinear motions, occlusions, and variations in lighting conditions. Especially, to generate a high-quality intermediate frame, it is important to estimate motions or optical flow vectors accurately.
Recently, with the advance of deep-learning-based optical flow methods [5,10,25,30], flow-based video interpolation algorithms [2,3,14] have been developed, yielding reliable interpolation results. Niklaus et al. [20] generated intermediate frames based on the forward warping. However, the forward warping may cause interpolation artifacts because of the hole and overlapped pixel problems. To overcome this, other approaches leverage the backward warping. To use the backward warping, intermediate motions should be obtained. Various video interpolation algorithms [2][3][4]14,16,20,32] based on the bilateral motion estimation approximate these intermediate motions from optical flows between two input frames. However, this approximation may degrade video interpolation results.
In this work, we propose a novel video interpolation network, which consists of the bilateral motion network and the dynamic filter generation network. First, we predict six bilateral motions: two from the bilateral motion network and the other four through optical flow approximation. In the bilateral motion network, we develop the bilateral cost volume to facilitate the matching process. Second, we extract context maps to exploit rich contextual information. We then warp the two input frames and the corresponding context maps using the six bilateral motions, resulting in six pairs of warped frame and context map. Next, these pairs are used to generate dynamic blending filters. Finally, the six warped frames are superposed by the blending filters to generate an intermediate frame.
Experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art video interpolation algorithms [2,3,18,22,32] meaningfully on various benchmark datasets.
This work has the following major contributions: -We develop a novel deep-learning-based video interpolation algorithm based on the bilateral motion estimation.
-We propose the bilateral motion network with the bilateral cost volume to estimate intermediate motions accurately.
-The proposed algorithm performs better than the state-of-the-art algorithms on various benchmark datasets.

Deep-learning-based video interpolation
The objective of video interpolation is to enhance a low temporal resolution by synthesizing intermediate frames between two actual frames. With the great success of CNNs in various image processing and computer vision tasks, many deep-learning-based video interpolation techniques have been developed. Long et al. [18] developed a CNN, which takes a pair of frames as input and then directly generates an intermediate frame. However, their algorithm yields severe blurring since it does not use a motion model. In [19], PhaseNet was proposed using the phase-based motion representation. Although it yields robust results to lightning changes or motion blur, it may fail to faithfully reconstruct detailed texture. In [21,22], Niklaus et al. proposed kernel-based methods that estimate an adaptive convolutional kernel for each pixel. The kernel-based methods produce reasonable results, but they cannot handle motions larger than a kernel's size.
To exploit motion information explicitly, flow-based algorithms have been developed. Niklaus and Liu [20] generated an intermediate frame from two consecutive frames using the forward warping. However, the forward warping suffers from holes and overlapped pixels. Therefore, most flow-based algorithms are based on backward warping.

Cost volume
A cost volume records similarity scores between two data. For example, in pixelwise matching between two images, the similarity is computed between each pixel pair: one in a reference image and the other in a target image. Then, for each reference pixel, the target pixel with the highest similarity score becomes the matched pixel. The cost volume facilitates this matching process. Thus, the optical flow estimation techniques in [5,10,30,31] are implemented using cost volumes. In [5,10,31], a cost volume is computed using various features of two video frames, and optical flow is estimated using the similarity information in the cost volume through a CNN. Sun et al. [30] proposed a partial cost volume to significantly reduce the memory requirement while improving the motion estimation accuracy based on a reduced search region. In this work, we develop a novel cost volume, called bilateral cost volume, which is different from the conventional volumes in that its reference is an intermediate frame to be interpolated, instead of one of the two input frames. Fig. 1 is an overview of the proposed algorithm that takes successive frames I 0 and I 1 as input and synthesizes an intermediate frame I t at t ∈ (0, 1) as output. First, we estimate two 'bilateral' motions V t→0 and V 0→t between the input frames. Second, we estimate 'bi-directional' motions V 0→1 and V 1→0 between I 0 and I 1 and then use these motions to approximate four further bilateral motions. Third, the pixel-wise context maps C 0 and C 1 are extracted from I 0 and I 1 . Then, the input frames and corresponding context maps are warped using the six bilateral motions. Note that, since the warped frames become multiple

Bilateral motion estimation
Given the two input frames I 0 and I 1 , the goal is to predict the intermediate frame I t using motion information. However, it is impossible to directly estimate the intermediate motion between the intermediate frame I t and one of the input frames I 0 or I 1 because there is no image information of I t . To address this issue, we assume linear motion between successive frames. Specifically, we attempt to estimate the backward and forward motion vectors V t→0 (x) and V t→1 (x) at x, respectively, where x is a pixel location in I t . Based on the linear assumption, . We develop a CNN to estimate bilateral motions V t→0 and V t→1 using I 0 and I 1 . To this end, we adopt an optical flow network, PWC-Net [30], and extend it for the bilateral motion estimation. Fig. 2 shows the key components of the modified PWC-Net. Let us describe each component subsequently.
Warping layer: The original PWC-Net uses the previous frame I 0 as a reference and the following frame I 1 as a target. On the other hand, the bilateral motion estimation uses the intermediate frame I t as a reference, and the input frames I 0 and I 1 as the target. Thus, whereas the original PWC-Net warps the feature c l 1 of I 1 toward the feature c l 0 of I 0 , we warp both features c l 0 and c l 1 toward the intermediate frame, leading to c l 0→t and c l 1→t , respectively. We employ the spatial transformer networks [11] to achieve the warping. Specifically, a target feature map c tgt is warped into a reference feature map c ref using a motion vector field by  where V ref→tgt is the motion vector field from the reference to the target.
Bilateral cost volume layer: A cost volume has been used to store the matching costs associating with a pixel in a reference frame with its corresponding pixels in a single target frame [5,9,30,31]. However, in the bilateral motion estimation, because the reference frame does not exist and should be predicted from two target frames, the conventional cost volume cannot be used. Thus, we develop a new cost volume for the bilateral motion estimation, which we refer to as the bilateral cost volume. Fig. 3 illustrates the proposed bilateral cost volume generation that takes the features c l 0 and c l 1 of the two input frames and the up-sampled bilateral motion fields V l t→0 and V l t→1 estimated at the (l − 1)th level. Let x denote a pixel location in the intermediate frame I l t . Then, we define the matching cost as the bilateral correlation between features c l 0 and c l 1 , indexed by the bilateral motion vector that passes through x, given by Note that we compute only |D| = D 2 bilateral correlations to construct a partial cost volume, where D = 2d + 1. In the L-level pyramid architecture, a one-pixel motion at the coarsest level corresponds to 2 L−1 pixels at the finest resolution. Thus, the search range D of the bilateral cost volume can be set to a small value to reduce the memory usage. The dimension of the bilateral cost volume at the lth level is D 2 × H l × W l , where H l and W l denote the height and width of the lth level features, respectively. Also, the up-sampled bilateral motions V l t→0 and V l t→1 are set to zero at the coarsest level. Most conventional video interpolation algorithms generate a single intermediate frame at the middle of two input frames, i.e. t = 0.5. Thus, they cannot yield output videos with arbitrary frame rates. A few recent algorithms [2,14] attempt to interpolate intermediate frames at arbitrary time instances t ∈ (0, 1). However, because their approaches are based on the approximation, as the time instance gets far from either of the input frames, the quality of the interpolated frame gets worse. On the other hand, the proposed algorithm takes into account the time instance t ∈ [0, 1] during the computation of the bilateral cost volume in (2). Also, after we train the bilateral motion network with the bilateral cost volume, we can use the shared weights to estimate the bilateral motions at an arbitrary t ∈ [0, 1]. In the extreme cases t = 0 or t = 1, the bilateral cost volume becomes identical to the conventional cost volume in [5,10,24,30,31], which is used to estimate the bi-directional motions V 0→1 and V 1→0 between input frames.

Motion approximation
Although the proposed bilateral motion network effectively estimates motion fields V t→0 and V t→1 from the intermediate frame at t to the previous and following frames, it may fail to find accurate motions, especially at occluded regions. For example, Fig. 4(d) and (e) show that the interpolated regions, reconstructed by the bilateral motion estimation, contain visual artifacts. To address this issue and improve the quality of an interpolated frame, in addition to the bilateral motion estimation, we develop an approximation scheme to predict a different kind of bilateral motions V t→0 and V t→1 using the bi-directional motions V 0→1 and V 1→0 between the two input frames.   5 illustrates this motion approximation, in which each column represents a frame at a time instance and a dot corresponds to a pixel in the frame. In particular, in Fig. 5(a), an occluded pixel in I 0 is depicted by a green dot. To complement the inaccuracy of the bilateral motion at pixel x in I t , we use two bi-directional motions V 0→1 (x) and V 1→0 (x), which are depicted by green and red lines, respectively. We approximate two forward bilateral motions V f w t→1 and V f w t→0 in Fig. 5(b) using V 0→1 . Specifically, for pixel x in I t , depicted by an orange dot, we approximate a motion vector V f w t→1 (x) by scaling V 0→1 (x) with a factor (1−t), assuming that the motion vector field is locally smooth. Since the bilateral motion estimation is based on the assumption that a motion trajectory between consecutive frames is linear, two approximate motions V f w t→1 (x) and V f w t→0 (x) should be symmetric with respect to x in I t . Thus, we obtain an additional approximate vector V f w t→0 (x) by reversing the direction of the vector V f w t→1 (x). In other words, we approximate the forward bilateral motions by Similarly, we approximate the backward bilateral motions by as illustrated in Fig. 5(c). Note that Jiang et al. [14] also used these equations (3)∼(6), but derived only two motion candidates: V t→1 (x) by combining (3) and (6) and V t→0 (x) by combining (4) and (5). Thus, if an approximated motion in (3)∼(6) is unreliable, the combined one is also degraded. In contrast, we use all four candidates in (3)∼(6) directly to choose reliable motions in Section 3.3. Fig. 4 shows that, whereas the bilateral motion estimation provides visual artifacts in (d), the motion approximation provides results without noticeable artifacts in (f). On the other hand, the bilateral motion estimation is more effective than the motion approximation in the cases of (e) and (g). Thus, the two schemes are complementary to each other.

Frame synthesis
We interpolate an intermediate frame by combining the six intermediate candidates, which are warped by the warping layers in Fig. 1. If we consider only color information, rich contextual information in the input frames may be lost during the synthesis [16,20,27], degrading the interpolation performance. Hence, as in [2,3,20], we further exploit contextual information in the input frames, called context maps. Specifically, we extract the output of the conv1 layer of ResNet-18 [8] as a context map, which is done by the context extractor in Fig. 1.
By warping the two input frames and the corresponding context maps, we obtain six pairs of a warped frame and its context map: two pairs are reconstructed using the bilateral motion estimation, and four pairs using the motion approximation. Fig. 1 shows these six pairs. Since these six warped pairs have different characteristics, they are used as complementary candidates of the intermediate frame. Recent video interpolation algorithms employ synthesis neural networks, which take warped frames as input and yield final interpolation results or residuals to refine pixel-wise blended results [2,3,20]. However, these synthesis networks may cause artifacts if motions are inaccurately estimated. To alleviate these artifacts, instead, we develop a dynamic filter network [13] that takes the aforementioned six pairs of candidates as input and outputs local blending filters, which are then used to process the warped frames to yield the intermediate frame. These local blending filters compensate for motion inaccuracies, by considering spatiotemporal neighboring pixels in the stack of warped frames. The frame synthesis layer performs this synthesis in Fig. 1. Fig. 6 shows the proposed synthesis network using dynamic local blending filters. The coefficients of the filters are learned from the images and contextual information through a dynamic blending filter network [13]. We employ the residual dense network [34] as the backbone for the filter generation. In Fig. 6

Dynamic local blending filters:
For each x, the sum of all coefficients in the six filters are normalized to 1. Then, the intermediate frame is synthesized via the dynamic local convolution. More specifically, the intermediate frame is obtained by filtering the intermediate candidates, given by

Training
The proposed algorithm includes two neural networks: the bilateral motion network and the dynamic filter generation network. We found that separate training of these two networks is more efficient than the end-to-end training in training time and memory space. Thus, we first train the bilateral motion network. Then, after fixing it, we train the dynamic filter generation network.
Bilateral motion network: To train the proposed bilateral motion network, we define the bilateral loss L b as where L p and L s are the photometric loss [26,33] and the smoothness loss [17]. For the photometric loss, we compute the sum of differences between a ground-truth frame I l t and two warped frames I l 0→t and I l 1→t using the bilateral motion fields V l t→0 and V l t→1 , respectively, at all pyramid levels, where ρ(x) = √ x 2 + 2 is the Charbonnier function [23]. The parameters α l and are set to 0.01 × 2 l and 10 −6 , respectively. Also, we compute the smoothness loss to constrain neighboring pixels to have similar motions, given by We use the Adam optimizer [15] with a learning rate of η = 10 −4 and shrink it via η ← 0.5η at every 0.5M iterations. We use a batch size of 4 for 2.5M iterations and augment the training dataset by randomly cropping 256 × 256 patches with random flipping and rotations.
Dynamic filter generation network: We define the dynamic filter loss L d as the Charbonnier loss between I t and its synthesized versionÎ t , given by Similarly to the bilateral motion network, we use the Adam optimizer with η = 10 −4 and shrink it via η ← 0.5η at 0.5M, 0.75M, and 1M iterations. We use a batch size of 4 for 1.25M iterations. Also, we use the same augmentation technique as that for the bilateral motion network.

Datasets:
We use the Vimeo90K dataset [32] to train the proposed networks. The training set in Vimeo90K is composed of 51,312 triplets with a resolution of 448 × 256. We train the bilateral motion network with t = 0.5 at the first 1M iterations and then with t ∈ {0, 0.5, 1} for fine-tuning. Next, we train the dynamic filter generation network with t = 0.5. However, notice that both networks are capable of handling any t ∈ (0, 1) using the bilateral cost volume in (2).

Experimental Results
We evaluate the performances of the proposed video interpolation algorithm on the Middlebury [1], Vimeo90K [32], UCF101 [28], and Adobe240-fps [29] datasets. We compare the proposed algorithm with state-of-the-art algorithms. Then, we conduct ablation studies to analyze the contributions of the proposed bilateral motion network and dynamic filter generation network.

Datasets
Middlebury: The Middlebury benchmark [1], the most commonly used benchmark for video interpolation, provides two sets: Other and Evaluation. 'Other' contains the ground-truth for fine-tuning, while 'Evaluation' provides two frames selected from each of 8 sequences for evaluation.
Vimeo90K: The test set in Vimeo90K [32] contains 3,782 triplets of spatial resolution 256 × 448. It is not used to train the model.
Adobe240-fps: Adobe240-fps [29] consists of high frame-rate videos. To assess the interpolation performance, we selected a test set of 254 sequences, each of which consists of nine frames.

Comparison with the state-of-the-arts
We assess the interpolation performances of the proposed algorithm in comparison with the conventional video interpolation algorithms: MIND [18], DVF [17], SpyNet [25], SepConv [22], CtxSyn [20], ToFlow [32], SuperSloMo [14], MEMC-Net [3], CyclicGen [16], and DAIN [2]. For SpyNet, we generated intermediate frames using the Baker et al.'s algorithm [1]. Table 1 shows the comparisons on the Middlebury Evaluation set [1], which are also available on the Middlebury website. We compare the average interpolation error (IE) and normalized interpolation error (NIE). A lower IE or NIE indicates better performance. The proposed algorithm outperforms all the state-of-the-art algorithms in terms of average IE and NIE scores. Fig. 7 visually compares interpolation results. SepConv-L 1 [22], ToFlow [32] SuperSlomo [14], Table 1. Quantitative comparisons on the Middlebury Evaluation set. For each metric, the numbers in red and blue denote the best and the second best results, respectively.  CtxSyn [20], MEMC-Net [3], and DAIN [2] yield blurring artifacts around the balls, losing texture details. On the contrary, the proposed algorithm reconstructs the clear shapes of the balls, preserving the details faithfully.
In Table 2, we provide quantitative comparisons on the UCF101 [28] and Vimeo90K [32] datasets. We compute the average PSNR and SSIM scores. The proposed algorithm outperforms the conventional algorithms by significant margins. Especially, the proposed algorithm provides 0.3 dB higher PSNR than DAIN [2] on Vimeo90K. Fig. 8 compares interpolation results qualitatively. Because the cable moves rapidly and the background branches make the motion estimation difficult, all the conventional algorithms fail to reconstruct the cable properly. In contrast, the proposed algorithm faithfully interpolates the intermediate frame, providing fine details.
The proposed algorithm can interpolate an intermediate frame at any time instance t ∈ (0, 1). To demonstrate this capability, we assess the ×2, ×4, and ×8 frame interpolation performance on the Adobe240-fps dataset [29]. Because the conventional algorithms in Table 3, except for DAIN [2], can generate only intermediate frames at t = 0.5, we recursively apply those algorithms to interpolate intermediate frames at other t's. Table 3 shows that the proposed algorithm  outperforms all the state-of-the-art algorithms. As the frame rate increases, the performance gain of the proposed algorithm against conventional algorithms gets larger. In Fig. 9, the proposed algorithm reconstructs the details in the rotating wheel more faithfully than the conventional algorithms.

Model analysis
We conduct ablation studies to analyze the contributions of the three key components in the   -BM+Appx2: In addition to BM, two more candidates obtained using approximated bilateral motions V f w t→0 in (4) and V bw t→1 in (6) are used. -BM+Appx4: Six intermediate candidates are used as well (proposed model). Table 4 compares these models quantitatively. First, Appx4 shows the worst performance, while it is still comparable to the state-of-the-art algorithms. Second, BM provides better performance than Appx4 as well as the state-of-the-art algorithms, which confirms the superiority of the proposed BM to the approximation. Third, we can achieve even higher interpolation performance with more intermediate candidates obtained through the motion approximation.
Dynamic blending filters We analyze the optimal kernel size and the input to the dynamic filter generation network. Table 5 compares the PSNR performances of different settings. First, the kernel size has insignificant impacts, although the computational complexity is proportional to the kernel size. Next, when additional information (input frames and context maps) is fed into the dynamic filter generation network, the interpolation performance is improved. More specifically, using input frames improves PSNRs by 0.10 and 0.15 dB on the UCF101 and Vimeo90K datasets. Also, using context maps further improves the performances by 0.07 and 0.05dB. This is because the input frames and context maps help restore geometric structure and exploit rich contextual information.

Conclusions
We developed a deep-learning-based video interpolation algorithm based on the bilateral motion estimation, which consists of the bilateral motion network and the dynamic filter generation network. In the bilateral motion network, we developed the bilateral cost volume to estimate accurate bilateral motions. In the dynamic filter generation network, we warped the two input frames using the estimated bilateral motions and fed them to learn filter coefficients. Finally, we synthesized the intermediate frame by superposing the warped frames with the generated blending filters. Experimental results showed that the proposed algorithm outperforms the state-of-the-art video interpolation algorithms on four benchmark datasets.