{'fraction_non_alphanumeric': 0.04006604292790314, 'fraction_numerical': 0.016877637130801686, 'mean_word_length': 4.680700291788245, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Prediction-assistant Frame Super-Resolution for Video Streaming

Video frame transmission delay is critical in real-time applications such as online video gaming, live show, etc. The receiving deadline of a new frame must catch up with the frame rendering time. Otherwise, the system will buffer a while, and the user will encounter a frozen screen, resulting in unsatisfactory user experiences. An effective approach is to transmit frames in lower-quality under poor bandwidth conditions, such as using scalable video coding. In this paper, we propose to enhance video quality using lossy frames in two situations. First, when current frames are too late to receive before rendering deadline (i.e., lost), we propose to use previously received high-resolution images to predict the future frames. Second, when the quality of the currently received frames is low~(i.e., lossy), we propose to use previously received high-resolution frames to enhance the low-quality current ones. For the first case, we propose a small yet effective video frame prediction network. For the second case, we improve the video prediction network to a video enhancement network to associate current frames as well as previous frames to restore high-quality images. Extensive experimental results demonstrate that our method performs favorably against state-of-the-art algorithms in the lossy video streaming environment.


Introduction
Video streaming is a key technology in broadcasting industries [19], such as online video gaming, live show, video on demand, etc. Due to the variability of the video transmission channel, it is difficult to ensure the user-perceived Quality of Experience (QoE) [2,27,11]. Two main factors affect video QoE. One is the quality of received frames, while the other is the transmission delay. Since the randomness of time-varying channel fluctuations degrades the transmission performance, the streaming system will encounter a buffering interruption. Then users may encounter a frozen screen, resulting in a dramatic decrease in QoE.
A plausible way is to transmit frames in lower-quality when encountering poor bandwidths, such as using scalable video coding [45]. In this paper, we consider one of the image quality degradations, namely spatial resolution degradation. A low-resolution image with fewer pixels saves more transmission bits, thus reducing the possibility of buffering interruption. However, transmitting low-resolution images with fewer pixels will also reduce the quality of received frames. Therefore, high image quality and low transmission delay are contradictory to each other in video streaming. Several works [55,48,37] propose video bitrate control algorithms to balance the trade-off between quality and delay. Those methods require meticulous modeling. Thus, those approaches are not adaptive to practical scenarios.
In contrast to the methods that rely on bitrate control [55,48,37], we propose to improve video QoEs through low-level image processing. We consider enhancing the quality of the received frames under two streaming patterns. First, when current frames are too late to receive before rendering deadline (i.e., lost), we propose to use previously received high-resolution images to predict future frames. Second, when the quality of the currently received frames is low (i.e., lossy), but the rendering deadline is approaching, we propose to use previously received high-resolution frames to enhance the low-quality current frames.
For the first case, similar to existing video prediction algorithms [35,14,34,15], we propose a small, yet effective video prediction network. The common approach for the convolutional neural network (CNN) based video prediction network is to first predict the optical flow between the current reference frame and the future frame. Then, warping the reference frame to the future frame based on the flow. Finally, the occlusion areas are filled using a CNN-based synthesis network. For this approach, the accuracy of the future optical flow prediction is significant in this framework. Different from those works using a CNN (e.g., U-Net) to directly predict the optical flow, we design a new flow predictor. The new flow predictor integrates previously received high-resolution frames as well as wrapped flow and propagated flow to synthesize the final future flow.
For the second case, we obtain previously received highresolution frames and current low-resolution frames. Our Figure 1. Illustration of lossy video streaming. We propose to restore high-resolution (HR) frames at the current time under two streaming patterns. First, when current frames are too late to receive before rendering deadline, we proposed to use previously received high-resolution images to predict the future frames. Second, when the resolution of the current frames is low (i.e., LR), we propose to use previous high-resolution frames to enhance the current low-quality frames.
goal is to synthesize a group of high-resolution current frames. To this end, the common practice is to perform video super-resolution [51,30,53,21,4]. State-of-the-art video super-resolution methods typically take multiple lowresolution frames as input. The outputs of those approaches are often blurry or too smooth, i.e., losing too much texture details. Recently, reference-based image super-resolution algorithms are widely used in multi-camera systems [9,58,41,54], which transfers high-resolution textures from the given reference image to produce visually pleasing results. However, if the resolution of the received frame is small enough, those algorithms can not estimate an accurate future optical flow, as described in Figure 5. To this end, we propose a new flow predictor that integrates the estimated flow as well as another two kinds of predicted flow, called propagated flow and wrapped flow (as described in Figure 2). Thus, even if the estimated flow is not accurate, the other two flows help the flow fusion network produce an accurate optical flow.
Our main contributions are summarized as follows: • We propose a prediction-assistant video frame superresolution method to enhance the quality of video streaming. The method explicitly exploits previous frames to help predict or restore current frames, which contributes to accurate optical flow estimation and provides rich texture details. • We propose to integrate the estimated flow, propagated flow, and predicted flow for accurate future flow es-timation. This mechanism contributes to frame synthesis especially when the current flow is lost or the resolution of the current frame is small. • We demonstrate that the proposed model is more effective, efficient, and compact than the state-of-the-art approaches.

Related Work
In this section, we review previous works of learningbased video super-resolution and video prediction. Video Super-Resolution. Super-resolution can be formulated as a dense image regression problem, which learns a derivable mapping function between low-resolution and high-resolution images. Super-resolution algorithms can be divided into two main categories, image super-resolution, and video super-resolution.
Image super-resolution algorithms upscale the resolution of a single image. Dong et al. [12] first propose SR-CNN that composes of three-layer CNNs to learn the pixel mapping function. Later, Dong et al. [13] redesign SR-CNN to speed up the computation. Kim et al. propose VDSR [23] and DRCN [24] using residual connections. The residual blocks [16] are used as common components in SRResNet [28] and EDSR [30]. Soon afterward, residual dense blocks [57,50,17], channel attention [56], non-local networks [33] are subsequently incorporated into image super-resolution algorithms. Compared with image superresolution, video super-resolution approaches are more focused on exploiting spatial and temporal information to restore high-quality images. The common approach of video super-resolution is first performing frame temporal alignment using optical flow [53,29,22], deformable convolution [49,51] or spatio-temporal network [7,32,44,25], then conducting image synthesis that resembles single frame super-resolution. Figure 2. Architecture of the proposed prediction-assistant video frame super-resolution model. Given three previously received HR frames I−2, I−1, I0, and one current LR frame I ↓s t , where s is the scale factor. We first use the three HR frames to predict two optical flows, including the warped flowF0→t, and the propagated flowF0→t. We also estimate a flowF0→t using HR frame I0 and the up-scaled current LR frame bic(I ↓s t ). Then, the three flows are refined by a flow fusion network. We then project the fused flow and back-warp the frame I0 based on the projected flow Ft→0. Finally, we apply a frame synthesis network to generate the output frame. When the current LR frame I ↓s t is not received before rendering deadline, the proposed model is the same except those green modules.
Recently, reference-based image super-resolution algorithms are widely used in the multi-camera imaging systems [9,58,41,54], which transfers high-resolution textures from the given reference image to produce visually satisfactory results. Paliwal et al. [41] propose to use two video streams as inputs, including an auxiliary video with high-frame-rate, low resolution, and a main video with low-frame rate, high resolution. They use CNNs to compute the optical flows between missing frames and the two existing frames of the main stream by utilizing the content of the auxiliary video frames. Similar to Paliwal et al. [41], Cheng et al. [9] propose a model for high spatio-temporal resolution video super-resolution. They also adopt an auxiliary stream and a main stream, but they just use one high-resolution reference frame of the main stream and the low-resolution frames of the auxiliary video to synthesize the high-resolution images. Besides, the optical flows are estimated only using consecutive auxiliary frames in Paliwal et al. [41]'s model, while Cheng et al. [9] use the up-scaled auxiliary frames as well as the reference image to perform motion estimation, similar to the setting of CrossNet [58]. However, those method does not consider the situation when the low-resolution is too small to support optical flow estimation, that will dramatically degrade the quality of synthesized frames.
Video Frame Prediction. Recent video frame prediction or extrapolation algorithms are using pixel-based [10,6,36], motion-based [35,34], and joint motion-pixel based schemes [14,15,42]. Pixel-based methods that generate pixels from scratch typically use implicit motion representations, which often leads to blurry effects. While the motionbased approaches produce sharp content, but the occlusion areas are often erroneous. The joint motion-pixel based algorithms such as DPG [14] gates the output of pixel pre-dictions from a flow predictor for non-occluded regions and from a context encoder for occluded regions, respectively. However, most motion-based video prediction methods predict the future optical flow by inputting multiple historical frames using a CNN (such as a U-Net) directly. Without explicit flow modeling, the error-prone predicted flow will degrade the quality of synthesized images.

Proposed Model
In this section, we introduce the proposed Prediction-ASSistant Network (PASS-Net). The architecture of the proposed model is shown in Figure 2.

Flow Prediction
We first consider the case where the future frame is lost (i.e., not received before the deadline). Given n + 1 previously received HR frames I −n , . . . , I −1 , I 0 , the goal of flow propagation is obtaining the optical flow from the current frame I 0 to the future frame I t . In this work, we set n equals to two. To predict the future optical flow, a straightforward approach is flow warping. The flow warping operation works effectively in optical flow estimation algorithms [43]. We first compute the optical flow fields between frame I −1 and I 0 , denoted by F −1→0 and F 0→−1 . Then we backward warp F −1→0 with F 0→−1 to approximate the flow from time 0 to 1.
where W(x; f ) refers to the result of back-warping the input x based on the optical flow field f . Existing models [20,3,40] usually assume a uniform motion between consecutive frames, the optical flow from 0 to t can be derived as: (a) overlayed inputs However, the linear assumption may not hold true for the complex non-uniform motion, especially when the future frame I t is lost. In contrast, we take inspirations from the work by Xu et al. [52] where higher-order information is considered for more accurate motion modeling. We extend their method from flow interpolation into flow propagation.
Given three previously received frames I −2 , I −1 and I 0 , we first compute two optical flow fields F 0→−1 and F 0→−2 . Under the assumption of the uniform acceleration [52], these optical flow fields can be modeled using , v 0 and a denote speed and acceleration respectively. Eliminating v 0 and a (more details can be found in the supplementary materials), the propagated optical flow fieldF 0→t can be derived as: With Equation (2) and (3), we have two candidate representations, namelyF 0→t andF 0→t for the future optical flow.
We compare the warped and propagated flows with the ground truth at every pixel using the Sintel dataset [5] with t = 1. We test the optical flow fields for two flow methods, FlowNet2-S [18] and PWC-Net [47]. As shown in Table 1, in most cases, the warped flows are more accurate than the propagated flows using the uniform acceleration assumption. However, the propagated can also benefit the flow prediction in complex motion cases. For the FlowNet2-S based backbone model, the error of initial flow estimation will accumulate in the propagation process. Thus, selecting an accurate backbone of flow estimation is significant for flow prediction. In this paper, we select the state-of-the-art PWC-Net [47] as our optical flow estimator.

Flow Estimation
We then consider the case where the received current frame is in low-quality. We use the resolution degradation as the quality degradation model.
where D denotes a degradation mapping function, e.g. bicubic sampling, I t is the corresponding high-resolution counterpart and s represents the scaling factor. Our goal is to estimate optical flow fieldF 0→t .
In this paper, we only use one historical frame I 0 , the Equation (5) can be reformulated to E(I 0 ; bic(I ↓s t )), E is a flow estimation function, e.g. PWC-Net [47], bic(·) refers to bicubic up-scaling function.
We conduct ablation studies to investigate when the accuracy of propagated flowF 0→t orF 0→t is higher than directly estimationF 0→t by adjusting the scaling factor s ∈ {0, 2, 4, 6, 8, 10, 12, 14} with t = 1. As shown in Figure 5, with the increase of scaling factor s, the resolution of frame I ↓s t decreases, which means the amount of pixel information decreases, indicating a quality degradation. When the scaling factor s exceeds 12, the flow prediction performs better than flow estimation cause the received frame I ↓s t lost too much information.

Flow Fusion
Our model works in two cases, the lost case and the lossy case. For the lost case, we obtain the propagated flowF 0→t and warped flowF 0→t , as shown in Figure 2. As shown in Figure 3, the warped flow tends to miss parts of the pixels around the image boundary because of back-warping and the propagated flow still has some ringing artifacts around object edges. For the lossy case, besides the propagated flow and warped flow, we also obtain the estimated flow F 0→t . However, with the increase of scaling factor s, the accuracy of the estimated flow decreases rapidly, and the flow field presents large artifacts across the whole figure, as shown in Figure 3.
In this section, we propose a flow fusion module to refine those candidate flows. We refine the flow field using a U-Net. We concatenate the propagated flowF 0→t , warped flowF 0→t and estimated flowF 0→t if available as well as image I 0 as inputs to the fusion module. We formulate the initial flow fusion process of the lost case as follows: and the lossy case as follows: where image I 0 is used to provide high-frequency details, and F refers to the flow fusion module. Moreover, we learn the residual optical flow using the U-net, which can be added to the initial estimated optical flow for refinement. We formulate the refinement process as follows: where σ(u) is the learned sampling offset of pixel u, the σ(u) is constrained to [−k, k] by k × tanh(·) as the activation function to achieve a local receptive field of 2k + 1, and the r(u) is the learned residual flow.

Frame Synthesis
We align the input reference frame I 0 to the target frame at time t according to the back-projected fused flow F t→0 to synthesize an initial frame. We use the mean backpropagation process which works effectively in frame interpolation task [53,4]: where u is a pixel coordinate on I t , N (u) is the neighborhood of u, and w(d) = e −d 2 /σ 2 is the Gaussian weight for flow. At last, we synthesize the result I t by concatenate flow, up-scaled frame I ↓s t if available, using the image synthesis network φ. The synthesis process based on backwarping operation W is given by: We use the U-Net as the image synthesis network φ with the trainable parameter set Θ.

Implementation Details
Network Components. As shown in Figure 2, the model consists of four neural network components. We adopt the PWC-Net [47] as our flow estimation network. The flow prediction module also re-uses this estimation network multiple times. Except for the PWC-Net, the flow prediction module has no learnable parameters. The flow fusion module and frame synthesize module are both U-Net, the only difference is inputs and outputs, as shown in Figure 4.
Loss Function. We denote the synthesized frame as I t , and the ground-truth image is denoted by I GT t . We use the following pixel loss function to train our model: where ρ(x) = √ x 2 + 2 is the Charbonnier penalty function [8]. We set the constant to 1e −6 .
Training Dataset. We train the proposed model only on the Vimeo90K-Septuplet [53] training set. The septuplet dataset consists of 91,701 7-frame sequences with a fixed resolution of 448 × 256, which are extracted from 39000 selected video clips from Vimeo-90K. The low-resolution image distortion for the Vimeo90K dataset is generated by down-sampling the original high-resolution frames. We use the MATLAB function imresize with the bicubic mode to accomplish the dataset generation. When training, we use the first three high-resolution frames as previously received frames and use the down-scaled fourth frame as the current low-resolution frame for the lossy case. For the lost case, we do not use any current frames. We augment the training data by horizontal and vertical flipping as well as randomly crop frames to a size of 256 × 256.
Training Strategy. We use the AdaMax [26] optimizer with parameters β 1 = 0.9 and β 2 = 0.999 to train the model. The batch size is 8, and the initial learning rate is 1e −4 . We load the pre-trained weights of PWC-Net [47] when initializing and set the learning rate of PWC-Net to zero. We train the model for 80 epochs, then reduce the learning rate by a factor of 0.2 and fine-tune the PWC-Net for another 10 epochs. We train the model on an RTX-2080 Ti GPU card, which takes about five days to converge.

Experimental Results
In this section, we first introduce the evaluation datasets and then conduct ablation studies to analyze the contribution of each proposed component. Then, we compare the proposed model with state-of-the-art algorithms for next frame synthesis. Finally, we analyze our model for the task of the next group of picture synthesis.

Evaluation Datasets and Metrics
We evaluate the proposed model on multiple public datasets with different image resolutions. The lowresolution frames are all generated using MATLAB function imresize with the bicubic mode.
UCF101. The UCF101 dataset [46] contains videos with variety of human actions. We use the evaluation set the same with DVF [35] to evaluate video prediction.
Vimeo90K. There are 7824 7-frames in the test set of the Vimeo90K Septuplet dataset [53]. The image resolution of the test set is 448 × 256 pixels.
Middlebury. We use the Middlebury OTHER images as the evaluation dataset [1]. The image resolution of this dataset is around 640 × 480 pixels. Table 1. Analysis of flow propagation and estimation using the Sintel [5] dataset. For frame prediction, we take three previous frames I−2, I−1, I0 as inputs to propagate or warp the optical flow of the next frame, denoted byF0→1,F0→1. For frame estimation, we use the current frame I0 and the next frame I1 of full-resolution to estimate optical flowF0→1. We compare the output flow with the ground truth at every pixel using end-point-error (EPE). The numbers in red and blue represent the best and second-best results. clips market2 alley2 sleeping1 ambush6 alley1 temple2 cave4 Using FlowNet2-S [18] PropagationF  Figure 5. Analysis of the accuracy of optical flow estimation considering scaling factor on the Sintel [5] dataset. We estimate the optical flow E(I0; bic(I ↓s t )), where E refers to PWC-Net [47], bic(I ↓s 1 ) is the bicubic interpolated frame.
Vid4. The Vid4 [31] dataset contains four video sequences: city, walk, calendar, and foliage. Each sequence in the Vid4 has at least 30 video frames with 720 × 480 pixels. REDS. REDS is a newly proposed high-quality video dataset [39], which consists of 30 validation clips, and each with 100 consecutive frames at 720p. We extract every 7 frames to make the evaluation dataset.
HD. The HD [4] dataset consists of four 1920 × 1080, three 1280 × 720 and four 1280 × 544 videos. We use the 1920 × 1080 video for evaluation. We first convert the YUV video to regular RGB video using FFmpeg, then make multiple consecutive 7-frame groups to create the test set.

Model Analysis
We analyze the contribution of the two key components in the proposed network: the optical flow estimation module and the flow fusion module.
Optical Flow Estimation. To analyze how well our approach performs with different correspondence estimates, we consider two state-of-the-art optical flow algorithms [47,18]. As shown in Table 2 (third section), those methods all perform similarly well. And the PWC-Net based method performs better than FlowNet2-S based model because the PWC-Net learns a slightly accurate optical flow. Specifically, we fine-tune PWC-Net and achieve additional performance gain with this PWC-Net-ft. Thus, we use the finetuned PWC-Net for our algorithm.
Frame Prediction Assistant. Our video super-resolution model uses the previous high-resolution frames and current low-resolution frames to synthesize high-quality frames at the current time. We adopt warped flow and propagated flow to help the flow fusion module to produce an accurate future flow, as shown in Figure 2. We train four versions of our model, including only using flow estimation (PAonly-f-estimation), without using flow propagation (PAw/f-propagation), without using flow warping (PA -w/fwarping), and using all of them (PA). As shown in Table 2 (fourth section), the model using the propagated flow and warped frame performs better than the algorithm only relies on flow estimation. Furthermore, when the current lowresolution frame is lost, we conduct experiments to verify the contribution of flow propagation for video frame prediction. Our model for frame prediction using the propagated frame works better than the one without using it, as shown in Table 4. Thus, the propagated flow and warped flow provide a foundation for the lost case, and the lossy case.
Video Frame Prediction. In Table 4, we show comparisons on the UCF101 [46] and Vimeo90K [53] dataset for the task of next frame prediction. Our algorithm performs favorably against existing methods in the compared dataset, especially on the Vimeo90K [53] dataset with a 2.75dB gain over DVF [35] in terms of PSNR.
Video Frame Super-Resolution. As shown in Table 3, we compare our model with multiple kinds of frame super-resolution algorithms. Our model outperforms all other methods for all evaluation datasets in terms of PSNR and SSIM metrics. Our model performs favorably against the non-reference video super-resolution methods (e.g., EDVR [51]) mainly because our model transfers highresolution textures from the given reference images to produce outputs with rich textual details. Our approach outperforms the reference-based video super-resolution methods (e.g., AWnet [9], CrossNet [58]) mainly because we adopt the flow prediction process to assistant the frame fusion module to predict an accurate flow, thus the frame synthesize module could produce high-quality frames.
Moreover, we test the model using frames with multiple gaps, which is more piratical in real applications. For example, there is a 7-frames group of each clip in the Vimeo90K dataset. In the previous section, we train our model using  the first three frames as high-resolution reference frames, and the down-scaled fourth frame as the low-resolution current input. In this case, the frame gap is equal to one. In this section, we test our model for multiple frame gaps, namely, we use the first three high-resolution frames as a reference and the nth down-scaled frame as the current input. As shown in Figure 10, our model performs favorably against all the compared methods. Even trained with frame gap equals to one, the reference-based models (e.g., Cross-Net, AWnet, and our algorithm) always outperform nonreference based methods (e.g., EDVR). However, with the increase of the frame gap, the PSNR of the synthesized frame also decreases, because the long temporal distance enlarges the objection motion, resulting in inaccurate motion estimation. In the meantime, the content similarity of the historical reference frames and the current frame is also reduced, making it hard to synthesize high-quality frames. Moreover, we believe that if we train the model using a frame with a randomly selected frame gap, our model will be more robust to multi-gap frame synthesis.

Conclusion
In this work, we propose to enhance video quality using lossy frames in two streaming patterns, including the Figure 10. Comparisons of video super-resolution using different frame gaps on the Vimeo90K evaluation dataset [53]. Our model is trained using a frame gap equals to one, and we test the model using frames with multiple gaps. lost case, and the lossy case. For the lost case, we propose to use previously received high-resolution images to predict future frames. For the lossy case, we propose to use previously received high-resolution frames to enhance the low-quality current frames. Our proposed video enhancement model has a unified framework both for video prediction and video super-resolution. Extensive experimental results demonstrate that our method performs favorably against state-of-the-art methods in the lossy video streaming environment. However, our model can not produce images in real-time, we will focus on speeding up the algorithm in future work.