{'fraction_non_alphanumeric': 0.03994157118729173, 'fraction_numerical': 0.016022275984845027, 'mean_word_length': 4.804981452040275, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution

Video super-resolution (VSR) aims to utilize multiple low-resolution frames to generate a high-resolution prediction for each frame. In this process, inter- and intra-frames are the key sources for exploiting temporal and spatial information. However, there are a couple of limitations for existing VSR methods. First, optical flow is often used to establish temporal correspondence. But flow estimation itself is error-prone and affects recovery results. Second, similar patterns existing in natural images are rarely exploited for the VSR task. Motivated by these findings, we propose a temporal multi-correspondence aggregation strategy to leverage similar patches across frames, and a cross-scale nonlocal-correspondence aggregation scheme to explore self-similarity of images across scales. Based on these two new modules, we build an effective multi-correspondence aggregation network (MuCAN) for VSR. Our method achieves state-of-the-art results on multiple benchmark datasets. Extensive experiments justify the effectiveness of our method.


Introduction
Super-resolution (SR) is a fundamental task in image processing and computer vision, which aims to reconstruct high-resolution (HR) images from low-resolution (LR) ones. While single-image super-resolution methods are mostly based on spatial information, video super-resolution (VSR) needs to exploit temporal structure from multiple neighboring frames to recover missing details. VSR can be widely applied to video surveillance, satellite imagery, etc.
In contrast to previous methods that model VSR as separate alignment and regression stages, we view this problem as an inter-and intra-frame correspondence aggregation task. Based on the fact that consecutive frames share similar content, and different locations within a single frame may contain similar structures (known as self-similarity [20,29,9]), we propose to aggregate similar content from multiple corresponding patches to better restore HR results.
Inter-frame Correspondence Motion compensation (or alignment) is usually an essential component for most video tasks to handle displacement among frames. A majority of methods [2,37,32] design specific sub-networks for optical flow estimation. In [19,22,15,14], motion is implicitly handled using Conv3D or recurrent networks. Recent methods [38,41] utilize deformable convolution layers [3] to explicitly align feature maps using learnable offsets. All the methods establish explicit or implicit one-on-one pixel correspondence between frames. However, motion estimation may suffer from inevitable errors and there is no chance for wrongly estimated mapping to locate correct pixels. Thus, we resort to another line of solutions to simultaneously consider multiple correspondence candidates for each pixel, as illustrated in Figure 1(a). In order to preliminarily validate our idea, we estimate optical flow with a simple patch-matching strategy on the MPI Sintel Flow dataset. After obtaining top-K most similar patches as correspondence candidates, we calculate the Euclidean distance between the best-performing one and ground-truth flow. As shown in Figure 1(b), it is clear that the better result is obtained by taking into consideration more correspondences for a pixel. Inspired by this finding, we propose a temporal multi-correspondence aggregation module (TM-CAM) for alignment. It uses top-K most similar feature patches as supplement. More specifically, we design a pixel-adaptive aggregation strategy, which will be detailed in Sec. 3. Our module is lightweighted and, more interestingly, can be easily integrated into many common frameworks. It is surprisingly robust to visual artifacts, as demonstrated in Sec. 4.
Intra-frame Correspondence From another perspective, similar patterns within each frame as shown in Figure 1(c) can also benefit detail restoration, which has been verified in several previous low-level tasks [20,29,9,44,47,13]. This line is still new for VSR. For existing methods in VSR, the common way to explore intra-frame information is to introduce a U-net-like [31] or deep structure, so that a large but still local receptive field is covered. We notice that valuable information may not always come from neighboring positions. In fact, similar patches within nonlocal locations or across scales may also be beneficial.
Accordingly, we in this paper design a new cross-scale nonlocal-correspondence aggregation module (CN-CAM) to exploit the multi-scale self-similarity property of natural images. It aggregates similar features across different levels to recover more details. The effectiveness of this module is verified in Sec. 4.
The contribution of this paper is threefold.
• We design a multi-correspondence aggregation network (MuCAN) to deal with video super-resolution in an end-to-end manner. It achieves state-ofthe-art performance on multiple benchmark datasets. • Two effective modules are proposed to make decent use of temporal and spatial information. The temporal multi-correspondence aggregation module (TM-CAM) conducts motion compensation in a robust way. The crossscale nonlocal-correspondence aggregation module (CN-CAM) explores similar features from multiple spatial scales. • We introduce an edge-aware loss that enables the proposed network to generate better refined edges.

Related Work
Super-resolution is a classical task in computer vision. Early work used examplebased [8,9,46,7,39,40,33], dictionary learning [45,28] and self-similarity [44,13] methods. Recently, with the rapid development of deep learning, super-resolution reached a new level. In this section, we briefly discuss deep learning based approaches from two lines, i.e., single-image super-resolution (SISR) and video super-resolution (VSR).

Single-Image Super-Resolution
SRCNN [4] is the first method that employs a deep convolutional neural network in the image super-resolution task. It has inspired several following methods [5,17,34,18,21,36,10,48,49]. For example, Kim et al. [17] proposed a residual learning strategy using a 20-layer depth network, which yielded significant accuracy improvement. Instead of applying commonly used bicubic interpolation, Shi et al. [34] designed a sub-pixel convolution network to effectively upsample low-resolution input. This operation reduces computational complexity and enables a real-time network. Taking advantage of high-quality large image datasets, more networks, such as DBPN [10], RCAN [48], and RDN [49], further improve the performance of SISR.

Video Super-Resolution
Video super-resolution takes multiple frames into consideration. Based on the way to aggregate temporal information, previous methods can be roughly grouped into three categories. The first group of methods process video sequences without any explicit alignment. For example, methods of [19,22] utilize 3D convolutions to directly extract features from multiple frames. Although this approach is simple, the computational cost is typically high. Jo et al. [15] proposed dynamic upsampling filters to avoid explicit motion compensation. However, it stands the chance of ignoring informative details of neighboring frames. Noise in the misaligned regions can also be harmful.
The second line [23,26,16,2,25,37,32,11] is to use optical flow to compensate motion between frames. Methods of [23,16] first obtain optical flow using classical algorithms and then build a network for high-resolution image reconstruction. Caballero et al. [2] integrated these two steps into a single framework and trained it in an end-to-end way. Tao et al. [37] further proposed sub-pixel motion compensation to reveal more details. No matter if optical flow is predicted independently, this category of methods needs to handle two relatively separated tasks. Besides, the estimated optical flow critically affects the quality of reconstruction. Because optical flow itself is a challenging task especially for large-motion scenes, the resulting accuracy cannot be guaranteed.
The last line [38,41] conducts deformable convolution networks [3] to accomplish video super-resolution. For example, EDVR proposed in [41] extracts and aligns features at multiple levels, and achieves reasonable performance. The deformable network is however sensitive to the input patterns, and may give rise to noticeable reconstruction artifacts due to unreasonable offsets.

Our Method
The architecture of our proposed multi-correspondence aggregation network (MuCAN) is illustrated in Figure 2. Given 2N + 1 consecutive low-resolution frames {I L t−N , . . . , I L t , . . . , I L t+N }, our framework predicts a high-resolution central image I H t . It is an end-to-end network consisting of three modules: a temporal multi-correspondence aggregation module (TM-CAM), a cross-scale nonlocalcorrespondence aggregation module (CN-CAM), and a reconstruction module. The details of each module are given in the following subsections.

Temporal Multi-Correspondence Aggregation Module
Camera or object motion between neighboring frames has its pros and cons. On the one hand, large motion needs to be eliminated to build correspondence among similar content. On the other hand, accuracy of small motion (at subpixel level) is very important, which is the source to draw details. Inspired by the work of [30,35], we design a hierarchical correspondence aggregation strategy to handle large and subtle motion simultaneously.
As shown in Figure 3, given two neighboring LR images I L t−1 and I L t , we first encode them into lower resolutions (level l = 0 to l = 2). Then, the aggregation starts in the high-level (low-resolution) stage (i.e., from F l=2 t−1 ) compensating large motion, while progressively moving up to low-level/high-resolution stages (i.e., to F l=0 t−1 ) for subtle sub-pixel shift. Different from many methods [2,37,32] that directly regress flow fields in image space, our module functions in the feature space. It is more stable and robust to noise [35].
The aggregation unit in Figure 3 is detailed in Figure 4. A patch-based matching strategy is used since it naturally contains structural information. As aforementioned in Figure 1(b), one-to-one mapping may not be able to capture true correspondence between frames. We thus aggregate multiple candidates to obtain sufficient context information as in Figure 4. Fig. 4. Aggregation unit in TM-CAM. It aggregates multiple inter-frame correspondences to recover a given pixel at p t .

Inner Product
In details, we first locally select top-K most similar feature patches, and then utilize a pixel-adaptive aggregation scheme to fuse them into one pixel to avoid boundary problems. Taking aligning F l t−1 to F l t as an example, given an image patch f l t (represented as a feature vector) in F l t , we first find its nearest neighbors on F l t−1 . For efficiency, we define a local search area satisfying p t − p t−1 ≤ d, where p t is the position vector of f l t . d means the maximum displacement. We use correlation as a distance measure as in Flownet [6]. For f l t−1 and f l t , their correlation is computed as the normalized inner product of After calculating correlations, we select top-K most correlated patches (i.e., where Aggr is implemented as convolution layers. Instead of assigning equal weights (e.g., 1 9 when the patch size is 3), we design a pixel-adaptive aggregation strategy to enable varying aggregation patterns in different locations. The weight map is obtained by concatenating F l t−1 and F l t and going through a convolution layer, which has a size of H × W × s 2 when the patch size is s × s. The adaptive weight map takes the form of As shown in Figure 4, the final value at position p t on the aligned neighboring frameF l t−1 is obtained asF After repeating the above steps for 2N times, we obtain a set of aligned neighboring feature maps To handle all frames at the same feature level, as shown in Figure 2, we employ an additional TM-CAM, which performs self-aggregation with I L t as the input and producesF 0 t . Finally, all these feature maps are fused into a double-spatial-sized feature map by a convolution and depth-to-space operation (PixelShuffle), which is to keep sub-pixel details.

Cross-Scale Nonlocal-Correspondence Aggregation Module
Similar patterns exist widely in natural images that provide abundant texture information. Self-similarity [20,29,9,44,47] can help detail recovery. In this part, we design a cross-scale aggregation strategy to capture nonlocal correspondences across different feature resolutions, as illustrated in Figure 5.
To distinguish from Sec. 3.1, we use M s t to denote feature maps at time t with scale level s. We first downsample the input feature maps M 0 t and obtain a feature pyramid as where AvgP ool is the average pooling with stride 2. Given a query patch m 0

Edge-Aware Loss
Usually, reconstructed high-resolution images produced by VSR methods suffer from jagged edges. To alleviate this problem, we propose an edge-aware loss to produce better refined edges. First, an edge detector is used to extract edge information of ground-truth HR images. Then, the detected edge areas are weighted more in loss calculation, enforcing the network to pay more attention to these areas during learning.
In this paper, we choose the Laplacian filter as the edge detector. Given the ground-truth I H t , the edge map I E t is obtained from the detector and the binary mask value at p t is represented as where δ is a predefined threshold. Suppose the size of a high-resolution image is H × W . The edge mask is also a H × W map filled with binary values. The areas marked as edges are 1 while others being 0. During training, we adopt the Charbonnier Loss, which is defined as whereÎ H t is the predicted high-resolution result, and is a small constant. The final loss is formulated as where λ is a coefficient to balance the two terms and • is element-wise multiplication. Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) [42] are used as metrics in our experiments.

Implementation Details
Network Settings The network takes 5 (or 7) consecutive frames as input. In feature extraction and reconstruction modules, 5 and 40 (20 for 7 frames) residual blocks [12] are implemented respectively with channel size 128. In Figure 3, the patch size is 3 and the maximum displacements are set to {3, 5, 7} from low to high resolutions. The K value is set to 4. In the cross-scale aggregation module, we define patch size as 1 and fuse information from 4 scales as shown in Figure 5. After reconstruction, both the height and width of images are quadrupled.

Training
We train our network using eight NVIDIA GeForce GTX 1080Ti GPUs with mini-batch size 3 per GPU. The training takes 600K iterations for all datasets. We use Adam as the optimizer and cosine learning rate decay strategy with an initial value 4e − 4. The input images are augmented with random cropping, flipping and rotation. The cropping size is 64 × 64 corresponding to output size 256 × 256. The rotation is selected as 90 • or −90 • . When calculating the edge-aware loss, we set both δ and λ to 0.1.
Testing In the testing phase, the output is evaluated without boundary cropping.

Ablation Study
To demonstrate the effectiveness of our proposed method, we conduct experiments for each individual design. For convenience, we adopt a lightweight setting in this section. The channel size of network is set to 64 and the reconstruction module contains 10 residual blocks. Meanwhile, the amount of training iterations is reduced to 200K. Table 1. Ablation Study of our proposed modules and loss on the REDS testing dataset. 'Baseline' is without using the proposed modules and loss. 'TM-CAM' represents the temporal multi-correspondence aggregation module. 'CN-CAM' means the cross-scale nonlocal-correspondence aggregation module. 'EAL' is the proposed edgeaware loss. Temporal Multi-Correspondence Aggregation Module For fair comparison, we first build a baseline without the proposed ideas. As shown in Table 1, the baseline only yields 28.98dB PSNR and 0.8280 SSIM, a relatively poor result. Our designed alignment module brings about a 1.15dB improvement in terms of PSNR.

Components
To show effectiveness of TM-CAM in a more intuitive way, we visualize residual maps between aligned neighboring feature maps and reference feature maps in Figure 6. After aggregation, it is clear that feature maps obtained with the proposed TM-CAM are smoother and cleaner. The mean L1 distance between aligned neighboring and reference feature maps are smaller. All these facts manifest the great alignment performance of our method.
We also evaluate how the number of aggregated temporal correspondences affects performance. Table 2 shows that the capability of TM-CAM rises at first and drops with the increasing number of correspondences. Compared with taking only one candidate, the four-correspondence setting obtains more than 0.1dB gain on PSNR. It demonstrates that highly correlated correspondences can provide useful complementary details. However, once saturated, it is not necessary to include more correspondences, since weakly correlated correspondences actually bring unwanted noise. We further verify this point by estimating optical flow using a KNN strategy for the MPI Sintel Flow dataset [1]. From Figure 1(b), we find that the four-neighbor setting is also the best choice. Therefore, we set K as 4 in our implementation. Finally, we verify the performance of pixel-adaptive weights. Based on the experiments, we find that a larger patch in TM-CAM usually gives a better result. It is reasonable since neighboring pixels usually have similar information and are likely to complement each other. Also, structural information is embedded. To balance performance and computing cost, we set the size to 3. When using fixed wieghts (at K = 4), we obtain the resulting PSNR/SSIM as 30.12dB/0.8614. From Table 2, the proposed pixel-adaptive weighting scheme achieves 30.31dB/0.8648, which is superior to the fixed counterpart by nearly 0.2dB on PSNR, which demonstrates that different aggregating patterns are necessary for consideration of spatial variance.
More experiments of TM-CAM with regard to patch size and maximum displacements are provided in the supplementary file.
Cross-Scale Nonlocal-Correspondence Aggregation Module In Sec. 4.3, we already notice that highly correlated temporal correspondences can serve as supplement to motion compensation. To further handle cases with different  scales, we proposed a cross-scale nonlocal-correspondence aggregation module (CN-CAM). As listed in Table 1, CN-CAM improves PSNR by 0.12dB. Besides, Figure 7 reveals that this module enables the network to recover more details when images contain repeated patterns such as windows and buildings within the spatial domain or across scales. All these results show that the proposed CN-CAM method further enhances the quality of reconstructed images.
Edge-Aware Loss In this part, we evaluate the proposed edge-aware loss (EAL). Table 1 lists the statistics. A few visual results in Figure 8 indicate that EAL improves the proposed network further, yielding more refined edges. The textures on the wall and edges of lights are clearer and sharper, which demonstrates the effectiveness of the proposed edge-aware loss.

Comparison with State-of-the-art Methods
We compare our proposed multi-correspondence aggregation network (MuCAN) with previous state-of-the-arts including TOFlow [43], DUF [15], RBPN [10],  and EDVR [41] on REDS [27], Vimeo-90K [43] and Vid4 [24] datasets. The quantitative results in Tables 3 and 4 are extracted from the original publications. Especially, original EDVR [41] is initialized with a well-trained model. For fairness, we use author-released code to train EDVR without pretraining. On the REDS dataset, results are shown in Table 3. It is clear that our method outperforms all other methods by at least 0.17dB. For Vimeo-90K, the results are reported in Table 4. Our MuCAN method works better than DUF [15] with nearly 1.2dB enhancement on RGB channels. Meanwhile, it brings 0.25dB improvement on the Y channel compared with RBPN [10]. All these results verify the effectiveness of our method. Besides, performance on Vid4 is reported in the supplementary file. Several examples are visualized in Figure 9.

Generalization Analysis
To evaluate the generality of our method, we apply our model trained on the REDS dataset to test video frames in the wild. In addition, we test EDVR with the author-released model 1 on the REDS dataset. Some visual results are shown in Figure 10. We remark that EDVR may generate visual artifacts in some cases due to the variance of data distributions between training and testing. (d) DUF [15] (c) TOFlow [43] (e) MuCAN (ours) Fig. 9. Examples of REDS (top two rows) and Vimeo-90K (bottom two rows) datasets.

EDVR EDVR
Ours Ours LR LR Fig. 10. Visualization of video frames in the wild for EDVR [41] and our MuCAN.
In contrast, our MuCAN performs well in the real world setting and shows its decent generality.

Conclusion
In this paper, we have proposed a novel multi-correspondence aggregation network (MuCAN) for the video super-resolution task. We showed that the proposed temporal multi-correspondence aggregation module (TM-CAM) takes advantage of highly correlated patches to achieve high-quality alignment-based frame recovery. Additionally, we verified that the cross-scale nonlocal-correspondence aggregation module (CN-CAM) utilizes multi-scale information and further boosts the performance of our network. Also, the edge-aware loss enforces the network to obtain more refined edges on the high-resolution output. Extensive experiments have manifested the effectiveness and generality of our proposed method.