{'fraction_non_alphanumeric': 0.030538675455762693, 'fraction_numerical': 0.009887733031208157, 'mean_word_length': 4.757189445597391, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Technical Report on Visual Quality Assessment for Frame Interpolation

Current benchmarks for optical flow algorithms evaluate the estimation quality by comparing their predicted flow field with the ground truth, and additionally may compare interpolated frames, based on these predictions, with the correct frames from the actual image sequences. For the latter comparisons, objective measures such as mean square errors are applied. However, for applications like image interpolation, the expected user's quality of experience cannot be fully deduced from such simple quality measures. Therefore, we conducted a subjective quality assessment study by crowdsourcing for the interpolated images provided in one of the optical flow benchmarks, the Middlebury benchmark. We used paired comparisons with forced choice and reconstructed absolute quality scale values according to Thurstone's model using the classical least squares method. The results give rise to a re-ranking of 141 participating algorithms w.r.t. visual quality of interpolated frames mostly based on optical flow estimation. Our re-ranking result shows the necessity of visual quality assessment as another evaluation metric for optical flow and frame interpolation benchmarks.


I. INTRODUCTION
As one of the basic video processing techniques, frame interpolation, namely computing interpolated in-between images, is a necessary step in numerous applications such as temporal up-sampling for generating slow motion videos, frame rate conversion between broadcast standards and so on [1]. One of the main approaches in frame interpolation is motion compensation, which can be achieved in various ways based on block matching, frequency domain, optical flow, etc. [2] Among them, optical flow is the most popular one for this usage [1]. Typically, optical flow or its variations are estimated and then used to produce interpolation results. Thus, for a given frame interpolation technique, the quality of the results heavily depends on the optical flow algorithm adopted [3].
Regarding the evaluation of motion-compensated interpolation, currently, there is only one optical flow benchmark that is commonly used and offers an assessment of interpolated frames, which is the Middlebury benchmark [4]. It considers angular and endpoint errors between an estimated flow vector and the ground-truth flow to assess the accuracy of optical flow computation methods.
Moreover, it uses the estimated flow field to interpolate an in-between image for two video frames, and then computes the root mean square error (RMSE) and gradient normalized RMSE between the interpolated image and the ground-truth image. However, it is well known that mean square errors can be misleading and may not reliably reflect image quality as perceived by the human visual system (HVS) [5]. In the Middlebury interpolation evaluation web-page, some interpolated images have the same RMSE, but exhibit obvious differences in image quality (see Fig.1). Therefore, we propose that the evaluation of motion-compensated interpolation should take perceived visual quality assessment into consideration.
Regarding visual quality assessment methods, we take fullreference image quality assessment (FR-IQA) into consideration, since ground-truth in-between images are available in the Middlebury benchmark. There are several FR-IQA methods that consider the HVS, such as SSIM [5], MS-SSIM [6], FSIM [7], VSI [8]. These methods were designed to estimate image quality degradation due to common artifacts, namely the ones caused by processing such as data compression or by losses in data transmission. However, the artifacts induced by optical flow algorithms lead to interpolated images with different specific distortions. In this contribution, we show (in Table I) that five objective FR-IQA methods have rather low correlations with the evaluations made by human observers, regardless of whether the methods are based on the HVS or just on pixel-wise errors such as RMSE. More specifically, the method VSI, one of the best FR-IQA methods, when trained and tested on the LIVE database [9], yielded a Spearman rank-order correlation coefficient (SROCC) of 0.952 w.r.t. ground truth consisting of MOS from a controlled lab study for the LIVE database. But when applied for the interpolated images by optical flow algorithms, VSI gave only an SROCC of 0.610. This shows that current FR-IQA methods do not perform well on interpolated frames produced by optical flow algorithms because of their specific distortions. Therefore, a new FR-IQA method specifically designed for such images is needed. However, before the research in such FR-IQA methods can proceed, ground-truth data, namely subjective quality scores of such images need to be collected, a first set of which is provided by our study.
For subjective quality evaluation, lab-studies take the lead because of their reliability. In lab-studies, the experimental environment and the evaluation process can be controlled. However, it is time consuming and costly, which severely limits the number of images to be assessed. Crowdsourcing studies can be less expensive and the reliability of crowdsourcing has been proven to be acceptable with appropriate setup and certain training for the crowd workers [10]. Therefore, we applied subjective quality assessment of the images interpolated by optical flow algorithms with the help of crowdsourcing.
In this paper, we implemented paired comparisons of the interpolated images given by optical flow algorithms in the Middlebury interpolation benchmark with the help of crowdsouring and re-ranked them accordingly. After that, the old ranking according to RMSE in the Middlebury benchmark and the re-ranking according to our subjective study were compared. After that, the old ranking according to RMSE in the Middlebury benchmark and the re-ranking according to our subjective study were compared. Our study shows that current FR-IQA methods are not suitable for assessing the perceived visual quality of interpolated frames (as produced by optical flow algorithms).

II. RELATED WORK
So far, there is only one benchmark that is used for evaluating the performance of frame interpolation, namely the Middlebury benchmark. It was originally designed for the evaluation of optical flow algorithms. Since it provides the ground-truth in-between images to evaluate the interpolation performance of optical flow algorithms, some interpolation algorithms also made use of this benchmark for evaluation, [11], [1], [3].

A. Subjective Study
Absolute Category Rating (ACR) is a judgment where the test items are presented one at a time and are rated independently on a category of five ordinal scales, i.e., Bad-1, Poor-2, Fair-3, Good-4, and Excellent-5 [19].
ACR is easy and fast for implementation, however, it has several problems [20]. Participants may be confused when the concept of the ACR scale has not been explained sufficiently well. They may also have different interpretations of the ACR scale, in particular in crowdsourcing experiments because of the wide range of backgrounds and perceptual experiences of the crowd workers from around the world. Moreover, the perceptual distances between two consecutive scale values, e.g., between 1 and 2, should ideally be the same. However, in practice this can hardly be achieved [21]. Also it is not easy to detect when a participant intentionally or carelessly gives false ratings.
Alternatively, paired comparisons (PC) can solve some of the problems of ACR. In a PC test, items to be evaluated are presented as pairs. In a forced choice setting, one of the items must be chosen as the preferred one. The main advantage of this is that it is highly discriminatory, which is very relevant when test items have nearly the same quality.
However, when implemented naively, to compare N items would require N 2 comparisons, too many to be practical, when N is on the order of 100, for example. In our case, for each of the 8 sequences, we would have to compare N = 141 images, giving a total of 78,960 pairs. Instead, random paired comparisons randomly chooses a fraction of all possible paired comparisons. This is more efficient and has been proven to be as reliable as full comparisons [22]. After obtaining paired comparisons results, subjective quality scores can be reconstructed based on Thurstone's model [23], [24] or the Bradley-Terry model [25].

B. Thurstone's Model
Thurstone's model provides the basis for a psychometric method for assigning scale values to options on a onedimensional continuum from paired comparisons data. It assumes that an option's quality is a Gaussian random variable, thereby accommodating differing opinions about the quality of an option. Then each option's latent quality score is revealed by the mean of the corresponding Gaussian.
The result of a paired comparison experiment is a square count matrix C denoting the number of times that each option was preferred over any other option. More specifically, for n comparisons of option A i with option A j , C i,j gives the number of times A i was preferred over A j . Similarly, C j,i in the count matrix denotes the number of times that A j was preferred over A i , and we have C i,j + C j,i = n.
According to Thurstone' Case V, subjective qualities about two options A and B are modelled as uncorrelated Gaussian random variables A and B with mean opinions µ A , µ B and variances σ A 2 , σ B 2 , respectively. When individuals decide which of the two options is better, they draw realizations from their quality distributions, and then choose the option with higher quality. More specifically, they choose option A over option B if their draw from the random variable A − B (with mean µ AB = µ A − µ B and variance σ AB 2 = σ A 2 + σ B 2 ) is greater than 0. Therefore, the probability of a subject to prefer option A over B is: where Φ(·) is the standard normal cumulative distribution function (CDF). Thurstone proposed to estimate P (A > B) by the empirical proportion of people preferring A over B, which can be derived from the count matrix C as The estimated quality differenceμ AB can be derived from inverting Eq. 1, giving: known as Thurstones Law of Comparative Judgment, where Φ(·) −1 is the inverse standard normal CDF, or z-score. Leastsquares fitting or maximum likelihood estimation (MLE) can be then applied to estimate the scale values µ A for all involved stimuli A. For more details we refer to the technical report [26].

C. Study Design
In order to re-rank the methods in the Middlebury benchmark, we implemented paired comparisons based on Thurstone's model with least-squares estimation to obtain subjective judgments of the image qualities. In the benchmark, there are 8 sets of interpolated images, most of which generated by  141 optical flow methods. 1 Therefore, in our experiment, for each set of 141 interpolated images, we generated a random sparse graph with degree of 6 (i.e., each image was to be randomly compared to 6 other images), which resulted in 423 pairs of images. We ran the experiment using the Figure Eight [27] platform. In our crowdsourcing interface as shown in Fig.  2, crowd workers were asked to identify and select the image with better quality for each image pair (forced binary choice).

D. Quality Assurance and Quality Control
Before the actual paired comparison, there was a training session, in which workers were instructed how to compare the image quality. Since the visual differences between some images are not that obvious, in the instructions as shown in Fig. 3, we highlighted those parts in the images that are more degraded and hence showed more differences.
In our experiment, we had eight separate jobs, each containing the 423 image pairs of the same series. In each job, there were 22 tasks, each consists of one page. In order to make sure that crowd workers' performance was not effected by exhaustion, we showed 20 pairs of images per page, and payments were initiated for each completed page. For each pair of images to be compared, we collected 30 votes from the crowd workers.
In order to assure the reliability of the crowd workers, the unreliable ones need to be detected and disallowed to continue. This was done by requiring crowd workers to answer test questions. For the test questions we chose image pairs with the ground-truth in-between image as one of the images, and the other image of bad quality. Then the expected, correct answer was obviously given by choosing the ground-truth image as the one with a better quality.
Before crowd workers were allowed to start a job, they must pass a quiz which is composed entirely of test questions. This ensures that only crowd workers that proved to be able to work on the subject matter of the job, would be able to enter the job. Crowd workers that failed the quiz were permanently disqualified from working on the job. After passing the quiz, crowd workers were admitted to start the real job. During the job, they had to answer further hidden test questions. Once a crowd worker failed more than 30% of the hidden test questions, he or she was disqualified and removed from the job. Only crowd workers who passed the quiz and showed an accuracy above 70% on the hidden test questions were regarded as reliable.

A. Statistics for the Crowdsourcing Experiment
In our experiment, there were eight jobs, each one comparing 141 interpolated images pairwise (423 pairs per job). The average run time was 29 hours per job. In total, 3189 crowd workers participated in our experiment, some of them took multiple jobs. Before the real experiment, 54% of them did not pass the quiz thus were not allowed to contribute to the job. During the real job, 14% of them failed more than 30% of the test questions, thus been disregarded. In the end, 1033 crowd workers were accepted as trusted workers. Among the trusted workers, 79% had an accuracy of 90%-100% (where 100% means they passed all test questions), 10% of them had an accuracy of 80%-90% and 10% of them had an accuracy of 70%-80%.

B. Re-ranking Results
Given the results of the paired comparisons, we reconstructed corresponding quality values based on Thurstone's model using the code provided by [26]. In order to make the results of the 8 separate jobs comparable, we added two fictitious images as anchors. One of them represents the worst quality among all the images, and the other one is like the ground-truth image, with a quality that is better than that of all the other images. After reconstruction of the scale values for the 141 + 2 images in each series, we linearly rescaled the quality scores such that the quality of the imaginary worst quality image became 0, and that of the ground-truth image became 1. In this way, we rescaled the reconstructed scores to the interval [0, 1]. Reconstructed quality values are shown in Table IV and V, accompanied by their corresponding rankings. Note that for each set, we ranked them separately. The 'Average' in the first column was gained by taking the mean of the 8 quality values in the same row, which results in the rank according to average quality. Table VI and VII show the differences between the reranking (ranked according to subjective study) and their corresponding ranking in the Middlebury benchmark (ranked according to RMSE), denoted by 'new' and 'old' in the tables, respectively. It can be seen that the best three methods ranked by the subjective study (i.e., SuperSlomo, CtxSyn and DeepFlow2), ranked 1st, 5th and 9th in the Middlebury benchmark, respectively. Overall, 36 methods highlighted in color blue showed rank differences up to 5. However, 30 methods highlighted in color red gave differences of more than 30 between their new and old rankings.  As an overall analysis, Table II shows the bootstrapped (after 1000 iterations) SROCC correlation values accompanied with confidence intervals (95%) between the ranking in Middlebury benchmark (i.e., ranking according to RMSE) and the re-ranking according to our subjective study. Note that the confidence interval of SROCC was computed by transforming the rank correlation score into an approximate z-score using the Fisher transform [28]. In a nutshell, a confidence interval of probability p is given by tanh(arctan r±Φ −1 (p)/ √ n − 3), where r denotes the estimated SROCC, n is the sampling size, and Φ −1 is the inverse of the standard normal CDF. In order to visualize the result, we computed the disagreement level as 1 − SROCC as shown in Fig. 5. Fig. 6 shows the scatter plots of the RMSE values compared to subjective scores of each optical flow method. Those eight plots are displayed in descending order of SROCC between RMSE and subjective scores. It can be seen that starting from Urban with the highest SROCC (0.854) down to the lowest SROCC (0.152) given by Backyard, the scattered plots become more sparse, as to be expected from their decreasing correlation.

V. LIMITATIONS
The interpolated images available from the Middlebury benchmark are compressed and slightly downsized from the original images. The original interpolated images could not be made available by the maintainers of the Middlebury benchmark. The differences between their resolutions are shown in Table III. Since we used the downscaled, compressed public version of the images for the crowdsourcing study, our results may be biased to a small extent.
Another limitation of the experiment is the difficulty of the subjective study. The quality differences between some images are quite hard to distinguish. Therefore, in the instructions of the crowdsourcing experiment, we highlighted the main degraded parts according to our visual observation to help the crowd workers to focus on the critical parts of the images. We believe, that this can be further improved in future studies, e.g., by providing zoomed image portions that contain the most noticeable artifacts.

VI. CONCLUSION AND FUTURE WORK
We have adopted visual quality assessment to the Middlebury benchmark for frame interpolation based mostly on optical flow methods. Our study confirms that only using RMSE as an evaluation metric for image interpolation performance is not representative of visual quality. Also current FR-IQA methods do not provide satisfying results on those interpolated images. This is due to the fact that such images, especially the ones generated by optical flow algorithms have specific distortions that are quite different from artifacts commonly addressed by conventional IQA methods.
Therefore, we plan to develop a domain specific FR-IQA for frame interpolation based on optical flow estimation. For the reference there is the original ground truth frame. In addition, we can make use of the ground truth optical flow vector field which is also available for the frame to be interpolated. This amounts to a FR-IQA with side information given by optical flow. It requires feature extraction from images and additionally from the optical flow, in order to train a model for a FR-IQA method specifically for frame interpolation by means of optical flow. The use case of such a FR-IQA method is to serve as a visual quality metric in optical flow benchmarks. Moreover, we plan to apply VQA methods on the videos generated by frame interpolation as a further study. This, in turn, will allow us to consider temporal aspects in the quality assessment.