{'fraction_non_alphanumeric': 0.03088544654809715, 'fraction_numerical': 0.012335054503729202, 'mean_word_length': 5.012359873526876, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Texture-aware Video Frame Interpolation

Temporal interpolation has the potential to be a powerful tool for video compression. Existing methods for frame interpolation do not discriminate between video textures and generally invoke a single general model capable of interpolating a wide range of video content. However, past work on video texture analysis and synthesis has shown that different textures exhibit vastly different motion characteristics and they can be divided into three classes (static, dynamic continuous and dynamic discrete). In this work, we study the impact of video textures on video frame interpolation, and propose a novel framework where, given an interpolation algorithm, separate models are trained on different textures. Our study shows that video texture has significant impact on the performance of frame interpolation models and it is beneficial to have separate models specifically adapted to these texture classes, instead of training a single model that tries to learn generic motion. Our results demonstrate that models fine-tuned using our framework achieve, on average, a 0.3dB gain in PSNR on the test set used.


INTRODUCTION
Video frame interpolation (VFI) refers to the task of generating non-existent intermediate frames between any two consecutive frames in a video while preserving spatiotemporal consistencies. VFI allows up-conversion of frame rates for improved visual quality, and offers an important tool for video compression, where it can be used to replace or enhance conventional motion estimation [1] or to conceal errors in reconstructed frames [2]. Interpolating visually pleasing intermediate frames requires accurate modelling of motion. However, due to the variety of textures and motion patterns in real-world videos, it is difficult to capture the motion of different texture types using a single mathematical model. Hence frame interpolation remains a challenging task.
Existing VFI methods can be classified as either flowbased or kernel-based. Flow-based methods [3][4][5][6][7][8][9] generally involve two steps, namely motion estimation and frame synthesis, where the latter typically resorts to pixel-wise warping of the two adjacent frames utilising the motion information to predict the intermediate frame. However, these methods generally experience a degradation in performance under challenging conditions where finding reference pixels in nearby frames is made difficult, e.g. under illumination change, occlusion and large motion. On the other hand, kernel-based methods [10][11][12][13][14][15][16][17] estimate a convolution kernel for each output pixel and predict the pixel by convolving nearby regions in adjacent frames with the kernel. Although in this case each output pixel is synthesised by referring to multiple pixels in the adjacent frames and occlusion can be better handled, the complexity of captured motion is still limited by the size of the kernels.
The aforementioned methods are mostly based on convolutional neural networks (CNNs). These have advanced the state-of-the-art in VFI over the past few years with various innovations relating to their overall architectures and to specific processing stages. One common feature amongst them is that a single model is trained to interpolate all kinds of video frame. In contrast, previous work on video texture analysis and synthesis [19][20][21][22][23][24] has shown that different textures exhibit vastly different motion patterns which are better handled separately. For example, Zhang et al. [19] proposed an analysis-synthesis video compression framework in which different algorithms were adopted to synthesise dynamic and static textures.
A finer classification of video textures was proposed in [21,22], where the authors clustered homogeneous sequences based on texture-relevant features as well as the HM encoding statistics and found three video texture clusters exist which were referred to as static (rigid texture exhibiting perspective motion), dynamic discrete (texture with discernible parts undergoing perspective motion independently) and dynamic continuous (spatially irregular and unstructured texture moving as a continuum). Example frames from each class taken from the HomTex dataset [21] are shown in Fig.1.
While generalisation is a desirable property of deep learning models, it has also proved to be difficult due to the issue of Interpolation results of our texture-aware finetuned AdaCoF and the original AdaCoF [15] on sample sequences from static (top row), dynamic discrete (middle row) and dynamic continuous (bottom row) textures. Sequences are "PaintingTilting1", "RiceField" and "Shinny-BlueWater downsampled" from the HomTex dataset [21]. over-fitting. Motivated by the classification of video textures in previous work, in this contribution, we investigate the impact of video texture type on the performance of state-of-theart VFI models. Based on our observation, we hypothesise that it is harder to train a generic VFI model that performs well on all texture types than to train separate models, each specialising in a specific texture (i.e. to overfit texture class); we verify our hypothesis through experimentation. As a consequence, we propose a novel texture-aware frame interpolation (TAFI) framework that can generalise to any VFI method and improve its performance.

RELATED WORK
Conventionally, VFI has been addressed by estimating optical flow and warping two adjacent frames using the flow map [3]. Such methods rely heavily on the adopted flow estimation algorithm and have been generally found to suffer from occlusion and large motion. Recent flow-based methods improved upon the framework by deploying CNNs for flow estimation and frame synthesis. Liu et al. [4] trained an encoder-decoder network to estimate the optical flow from the intermediate frame to the two reference frames, which is then used by a sampling layer to interpolate each output pixel. The complexity of the captured motion was limited since equal flow in two directions was assumed. Jiang et al. [5] improved on [4] by enabling bidirectional flow. Niklaus et al. [6] incorporated context information during the warping process. Bao et al. [7] additionally estimated depth map to refine the estimated flow. Recently, Niklaus et al. [9] proposed to use softmax splatting for differentiable forward warping such that different reference pixels can map to the same output pixel. Although this allows more complex motion to be captured, such complexity is still limited due to the dependence on flow estimation and pixel-wise warping.
On the other hand, Kernel-based methods inherently allow more pixels to be sampled for predicting one output pixel. Niklaus et al. [11] proposed a CNN (improved later in [12] and [13]) to predict a kernel for each output pixel which is convolved with the input image to obtain the target pixel. Instead of predicting individual kernels for each pixel, Choi et al. [14] proposed an end-to-end network with channel attention that directly outputs the interpolated frame. Recently, there has been an increased number of works on VFI that employs deformable convolution [18]. Lee et al. [15] developed a CNN to predict the kernel weights together with their offsets for each output pixel. The flexibility of deformable kernels enabled more complex motion to be captured. Cheng et al. [16] adopted a similar approach but separable kernels were used. Shi et al. [17] enabled further degree of freedom by allowing the reference pixels to be sampled in an interpolated space-time volume instead of just from the existent frames.
While these VFI methods were developed to generalise on all types of videos, previous work on video texture analysis and synthesis adopted a different approach. In [19], Zhang et al. observed that textured regions in videos typically consume more bits to encode, and proposed to synthesise such region and encode only the warping parameters, where different synthesis methods were used for dynamic and static textures. In [21], Afonso et al. analysed video texture by clustering homogeneous videos based on their encoding statistics (e.g. prediction modes, bit allocation etc.) and showed that video texture can be categorised as static, dynamic discrete and dynamic continuous. A homogeneous texture dataset, HomTex, was produced. Katsenou et al. [22] confirmed the existence of such categorisation by clustering videos based on their spatiotemporal features. In our work, the implications of such texture categorisation for VFI are investigated.

Effect of Texture on VFI Performance
In this section we investigate the performance of VFI methods on different video textures. Specifically, the pre-trained versions of three CNN-based state-of-the-art VFI models with publicly available source code are evaluated: AdaCoF [15], CAIN [14] and GDConvNet [17]. The evaluation dataset used is HomTex [21] which contains 120 videos with 250 frames of 256 × 256 spatial resolution at 25 and 60 fps. Each video  in HomTex is (approximately) texturally homogeneous; there are 45, 50 and 25 sequences that are labelled dynamic continuous, dynamic discrete and static respectively. Each model is evaluated on the entire HomTex dataset, where for each sequence every second frame, I t , is considered the groundtruth and the two adjacent frames I t−1 and I t+1 are input to the model (in the case of GDConvNet 4 frames, namely I t−3 , I t−1 , I t+1 , I t+3 , are used as input due to the model design). Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [25] scores are computed using the original and interpolated frames as these are the most commonly used metrics in VFI [3][4][5][6][7][8][9][10][11][12][13][14][15][16][17]. Additionally, video multi-method assessment fusion (VMAF) [26] scores are also computed using the original and interpolated videos. The scores of each model are then grouped according to the video texture class, and their distributions are shown in Fig.2, where abbreviations "dyncon" and "dyndis" are used for dynamic continuous and dynamic discrete.
It is noticeable that the distributions of the scores obtained by the models vary with texture type. Specifically, all models scored the highest PSNR, SSIM and VMAF on static textures, which is expected since the perspective motion of rigid objects is relatively easier to capture. For the case of dynamic texture, all models see degraded performance across all metrics, and this decrease is even more severe for dynamic continuous textures which exhibit the most irregular motion intrinsic to the scenes containing water, smoke and fire, etc.
To assess whether the impact of texture on VFI performance is statistically significant, a one-way ANOVA is performed with results shown in Table 3.1. Here we see that in terms of all metrics, texture type has significant effect on the performance of the VFI models at p < 0.05 level. Furthermore, T-tests (Welch's T-test to account for unequal sample sizes) are performed between the scores of each model on each pair of textures to see if there are significant differences, and the results are summarised in Table 2. The last two rows of the table show that, in terms of all metrics, the average performance of the models on static textures is significantly different from that on dynamic discrete and dynamic continuous textures. Comparing between the two dynamic types, all models have significantly different SSIM and VMAF scores. Combining these results with Fig.2, we can confirm that for p < 0.05, all models performed best on static textures and worst on dynamic continuous textures. There are two possible reasons for this observation. The first is that the dynamic texture types fall outside of the distribution of the three models' training sets. We consider this reason less important since all models were trained on the Vimeo90K dataset [8] which involves a wide range of non-homogeneous videos that cover all types of textures. The second reason is that the complexity of motion patterns, and hence the levels of difficulties to interpolate different textures are inherently different, with dynamic continuous textures exhibiting the most unpredictable motion thus being the hardest to interpolate, and static textures being the easiest type. This implies that, rather than trying to train a single model that learns to model the motion of all textures, there might be potential gain in having separate models specialised in interpolation of one texture class, namely texture-aware frame interpolation.

Proposed Method
Under the proposed VFI framework, the aforementioned three texture types are treated separately. That is, given a VFI model, we generate three versions of it, each trained exclusively on one type of texture. In this work we focus on performance of models on homogeneously-textured videos so the inference on a test video is performed in the same way as the original model, but with the difference that the specialised versions are used according to the video texture type. We show through our experiments that it is more difficult to learn a generic model that can interpolate all types of videos than to learn a specialised model that overfits a specific texture class.

. Datasets
We focus on the three models used previously for analysis, i.e. AdaCoF [15], CAIN [14] and GDConvNet [17]. The test data used for evaluation are the 120 homogeneous sequences in HomTex [21]. For the purpose of training models tuned to a single texture type, the training dataset should contain homogeneous sequences. Therefore, we make use of three existing texture datasets, namely DynTex [20], BVI-Texture [23] and SynTex [24]. Specifically, DynTex contains 650 annotated videos of 720 × 576 spatial resolution at 25 fps, and we retained those that contain only one type of texture according to the annotations. BVI-Texture and SynTex each contain 20 and 196 full-HD resolution (1920 × 1080) homogeneous sequences at 60 fps. It should be noted that, since HomTex is composed of videos from DynTex and BVI-Texture, we removed the DynTex sequences that exist in HomTex and other sequences that are visually similar to them. This resulted in a total of 214 dynamic continuous, 222 dynamic discrete and 110 static sequences with each sequence containing at least 250 frames.

Training details
Due to the lack of a diverse homogeneous video texture dataset, the models are initialised with their pre-trained weights provided and fine-tuned on our training sets. The loss functions, optimisation strategies and other hyperparameters except learning rate are set identical to the original implementations of the models. To fine-tune a model on a specific texture class, each training batch is formed by randomly sampling 10000 triplets from videos of that texture where each time a sequence is randomly selected then a triplet (or quintuplet for GDConvNet) of 256 × 256 patches is sampled randomly from the space-time volume. The sampled frames are augmented via random horizontal and vertical flipping, color jittering and temporal order reversal. Batch sizes of 8, 8, 3 are used and the initial learning rates are set to be 10 −3 , 10 −5 , 10 −4 for AdaCoF, CAIN and GDConvNet respectively. The models are fine-tuned on each video texture for 10 epochs with learning rates decayed by a factor of 0.5 every 4 epochs. The computations are performed on Nvidia P100 GPU cards provided on the shared cluster BlueCrystal Phase 4 [27] at the University of Bristol.

Effectiveness of TAFI
In order to investigate whether there is benefit in texturespecific training instead of training the models on all texture classes, in this experiment we fine-tune each model on four training sets: three for the three texture classes as described above and also the combination of them (referred to as "mixed"). We name the fine-tuned versions after the targetted texture type, namely "dyncon", "dyndis", "static" and "mixed". The four fine-tuned versions, as well as the original "off-the-shelf" version (regarded as the baseline) of each model are then evaluated on HomTex. The average PSNR and SSIM scores obtained for each texture subset in HomTex (i.e. HomTex-dyncon, HomTex-dyndis and HomTex-static) and the overall HomTex dataset (HomTex-overall) are summarised in Table 3. We see from Table 3 that for all baseline models, texture class tuning increased their performance on test sequences from that class in terms of both PSNR and SSIM, although in general the variations in SSIM scores are marginal. Meanwhile, models fine-tuned on a class generally performed worse on other textures, implying some overfitting which is expected. The improvements due to class-based training are particularly evident for dynamic textures.
The performance variations are more obviously reflected by PSNR, and this is because all models use 1 distortion as a major component in their loss functions, so one can expect a lower mean-squared-error after training and hence the obvious variations in PSNR.
Comparing the specialised models that are fine-tuned on single texture types (which together form the TAFI model) against the model fine-tuned on the mixed training set, we see that in all cases the former managed to deliver increased gain in terms of both PSNR and SSIM. This result is consistent with our hypothesis that learning a generic model is relatively harder than specialising on a specific video texture class. It is noted that the models fine-tuned on the mixed set constantly exhibited worse performance on static textures while perfor- Table 3. Performance of the original (baseline) and texture-aware fine-tuned versions of AdaCoF, CAIN and GDConvNet on HomTex. The version indicates the texture the model is fine-tuned on. The combination of dyncon, dyndis and static versions forms the TAFI model. Models are evaluated on three individual texture types in HomTex as well as the whole HomTex dataset. Numbers in brackets denote change with respect to the baseline model. For each column, the best result is in bold text. Our TAFI framework consistently improved the overall performance of the baseline models. mance on the other two texture types are mostly improved. This can be attributed to the fact that the static sequences in the training set constitute a relatively small proportion (20%) in the mixed set, making the training more biased towards the dynamic texture types. Finally, comparing the performance of the models on the entire HomTex dataset, it is clear that for all three VFI models, the combination of their specialised versions (TAFI) achieved higher PSNR and SSIM than their baselines, with approximately 0.3dB gain in PSNR after texture-aware finetuning. Example qualitative interpolation results of the specialised AdaCoF models and the original AdaCoF are given in Fig.1, where it can be observed that the specialised versions (AdaCoF-TAFI) produce an intermediate frame with higher visual quality. This is particularly evident for dynamic textures where the original AdaCoF model fails to capture the more complex motion and produces numerous visual artefacts.

CONCLUSION AND FUTURE WORK
In this work, we applied the video texture categorisation proposed in previous work to the problem of video frame interpolation (VFI). The effect of video texture type on VFI was studied by evaluating three state-of-the-art models on a video dataset HomTex which contains homogeneous sequences of static, dynamic discrete and dynamic continuous textures. The results showed that the models perform differently on different textures with statistical significance. Motivated by this observation, we fine-tuned the three VFI models on each of the three types of textures as well as the combination of them, and found that although each model fine-tuned on a specific texture overfits that texture class, it outperformed the model trained on combination of textures, proving our hypothesis that it is beneficial in terms of interpolation quality to have separate models specialised in different textures, instead of training a single model to learn to interpolate all kinds of textures.
To further confirm these findings, VFI models should be completely re-trained on larger scale homogeneous video datasets which do not yet exist. A possible direction of future work is to construct such a dataset and train state-of-the-art VFI models on homogeneous videos from scratch. In addition, despite the improved performance, our framework requires three times more computation compared to the original VFI models and there is up to future research to develop a single model that can adapt to different textures. Finally, in this work we focused on performance of VFI on homogeneous videos and did not consider fusion of the specialised models to interpolate generic sequences. Development of such ensemble algorithms will also be part of our future work.