{'fraction_non_alphanumeric': 0.04247493205885109, 'fraction_numerical': 0.014829912847905538, 'mean_word_length': 5.020451339915374, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content

Blind or no-reference video quality assessment of user-generated content (UGC) has become a trending, challenging, heretofore unsolved problem. Accurate and efficient video quality predictors suitable for this content are thus in great demand to achieve more intelligent analysis and processing of UGC videos. Previous studies have shown that natural scene statistics and deep learning features are both sufficient to capture spatial distortions, which contribute to a significant aspect of UGC video quality issues. However, these models are either incapable or inefficient for predicting the quality of complex and diverse UGC videos in practical applications. Here we introduce an effective and efficient video quality model for UGC content, which we dub the Rapid and Accurate Video Quality Evaluator (RAPIQUE), which we show performs comparably to state-of-the-art (SOTA) models but with orders-of-magnitude faster runtime. RAPIQUE combines and leverages the advantages of both quality-aware scene statistics features and semantics-aware deep convolutional features, allowing us to design the first general and efficient spatial and temporal (space-time) bandpass statistics model for video quality modeling. Our experimental results on recent large-scale UGC video quality databases show that RAPIQUE delivers top performances on all the datasets at a considerably lower computational expense. We hope this work promotes and inspires further efforts towards practical modeling of video quality problems for potential real-time and low-latency applications. To promote public usage, an implementation of RAPIQUE has been made freely available online: \url{https://github.com/vztu/RAPIQUE}.


I. INTRODUCTION
Recent years have witnessed an explosion of user-generated content (UGC) captured and streamed over social media platforms such as YouTube, Facebook, TikTok, and Twitter. Thus, there is a great need to understand and analyze billions of these shared contents to optimize video pipelines of efficient UGC data storage, processing, and streaming. UGC videos, which are typically created by amateur videographers, often suffer from unsatisfactory perceptual quality, arising from imperfect capture devices, uncertain shooting skills, and a variety of possible content processes, as well as compression and streaming distortions. In this regard, predicting UGC video quality is much more challenging than assessing the quality of synthetically distorted videos in traditional video databases. UGC distortions are more diverse, complicated, commingled, and no "pristine" reference is available.
Traditional video quality assessment (VQA) models have been widely studied [1] as an increasingly important toolset used by the streaming and social media industries. While full-reference (FR) VQA research is gradually maturing and several algorithms [2], [3] are quite widely deployed, recent attention has shifted more towards creating better no-reference (NR) VQA models that can be used to predict and monitor the quality of authentically distorted UGC videos. One intriguing property of UGC videos, from the data compression aspect, is that the original videos to be compressed often already suffer from artifacts or distortions, making it difficult to decide the compression settings [4]. Similarly, it is of great interest to be able to deploy flexible video transcoding profiles in industry-level applications based on measurements of input video quality to achieve even better rate-quality tradeoffs relative to traditional encoding paradigms [5]. The decision tuning strategy of such an adaptive encoding scheme, however, would require the guidance of an accurate and efficient NR or blind video quality (BVQA) model suitable for UGC [6].
Prior work has mainly focused on spatial distortions, which have been shown to indeed play a critical role in UGC video quality prediction [6]. The exploration of the temporal statistics of natural videos, however, has been relatively limited. The authors of [6] have shown that temporal-or motionrelated features are essential components when analyzing the quality of mobile captured videos, as exemplified by those in the LIVE-VQC database [30]. Yet, previous BVQA models that account for temporal distortions, such as V-BLIINDS and TLVQM, generally involve expensive motion estimation mod-els, which are not practical in many scenarios. Furthermore, while compute-efficient VQA models exist, simple BVQA models like BRISQUE [8], NIQE [33], GM-LOG [11] are incapable of capturing complex distortions that arise in UGC videos. Complex models like V-BLIINDS [9], TLVQM [13], and VIDEVAL [6], on the contrary, perform well on existing UGC video databases, but are much less efficient, since they either involve intensive motion-estimation algorithms or complicated scene statistics features. A recent deep learning model, VSFA [19], which extracts frame-level ResNet-50 [34] features followed by training a GRU layer, is also less practical due to the use of full-size, frame-wise image inputs and the recurrent layers.
We have made recent progress towards efficient modeling of temporal statistics relevant to the video quality problem, by exploiting and combining spatial and temporal scene statistics, as well as deep spatial features of natural videos. We summarize our contributions as follows: • We created a rapid and accurate video quality predictor, called RAPIQUE, in an efficient manner, achieving superior performance that is comparable or better than stateof-the-art (SOTA) models, but with a relative 20x speedup on 1080p videos. The runtime of RAPIQUE also scales well as a function of video resolution, and is 60x faster than the SOTA model VIDEVAL on 4k videos. • We built a first-of-its-kind BVQA model that combines novel, effective, and easily computed low-level scene statistics features with high-level deep learning features. Aggressive spatial and temporal sampling strategies are used, exploiting content and distortion redundancies, to increase efficiency without sacrificing performance. • We created a new spatial NSS feature extraction module within RAPIQUE, which is a highly efficient and effective alternative to the popular but expensive featurebased BIQA model, FRIQUEE. The spatial NSS features used in RAPIQUE are suitable for inclusion as basic elements of a variety of perceptual transforms, leading to significant efficiencies which might also be useful when developing future BVQA models. • We designed the first general, effective and efficient temporal statistics model (beyond frame-differences) that is based on bandpass regularities of natural videos, and which can also be used as a standalone module to boost existing BVQA methods on temporally-distorted or motion-intensive videos.
The rest of this paper is organized as follows. Section II reviews previous literature relating to video quality assessment models, while Section III unfolds the details of the RAPIQUE model. Experimental results and concluding remarks are given in Section IV and Section V, respectively.

A. Traditional BVQA Models
Many early BVQA/BIQA models have been 'distortion specific' in that they were designed to quantify a specific type of distortion such as blockiness [35], blur [36], ringing [20], banding [21], [37], [38], or noise [22], [39] in compressed images and videos. Recent high-performing BIQA/BVQA models are almost exclusively learning-based, operating by training sets of generic quality-aware features, which are combined to conduct quality predictions [7]- [10], [12]- [15]. Learningbased BVQA models are more versatile and generalizable than 'distortion specific' models, in that the selected features are broadly perceptually relevant, while powerful regression models can adaptively map the features onto quality scores learned from the data in the context of a specific application.
Time-domain behavior is the key attribute that differentiates videos from still pictures. The perception of video correlates highly with motion and temporal change [52]. Amongst BVQA models, Video-BLIINDS [9] was the first to explore the use of (spatio-) temporal scene statistics of video using DCT coefficient statistics in the time-differenced domain. V-BLIINDS also involves calculating motion coherence and global motion features, which requires expensive motion estimation, to account for temporal masking effects.
Instead of using DCT transforms, Mittal et al. proposed a completely blind model called VIIDEO [47], which inspects the divisively normalized spatial statistics of frame differences. Bandpass filtering followed by divisive normalization was applied to frame differences, after which the inter-subband correlations are modeled over the temporal variation of the extracted generalized Gaussian parameters. As a highly experimental temporal-only model, VIIDEO includes no spatial features, hence does not perform well on natural UGC video datasets [30], [31].
Regarding the joint modeling of spatiotemporal statistics, Li et al. proposed to adopt 3D-DCT transforms of local spacetime regions from videos to extract quality-aware features [46]. More recently, the authors of [53] leveraged 3D divisive normalization transformed (DNT) and spatiotemporal Gaborfiltered responses of 3D-DNT coefficients of natural videos. The 3D transforms adopted therein, however, are too expensive for practical use; neither have these models been observed to perform well on UGC datasets [30], [32].
A very recent feature-based BVQA model called TLVQM [13] uses a two-level feature extraction mechanism to achieve efficient computation of a set of impairment/distortion-relevant features. Unlike NSS-based models, TLVQM relies on a comprehensive set of highly crafted features that measure motion, specific distortion artifacts, and aesthetics. TLVQM requires that a large number of parameters be specified by the user, which may affect its general performance on datasets or scenarios it has not been exposed to. The model currently achieves very good performance on natural video quality databases at a reasonable complexity.
VIDEVAL [6] is currently the SOTA feature-based BVQA model on recent large-scale video dataset like KoNViD-1k [31] and YouTube-UGC [32]. It employs feature selection and fusion on top of efficacious NSS-based models as well as distortion-based features. It is also a very compact model as it only utilizes 60 features. However, it has not been observed to efficiently scale to high-resolution and high-framerate videos.

B. Deep Learning-Based BVQA Models
Deep convolutional neural networks (CNNs) have been shown to deliver standout performance in a wide variety of low-level computer vision applications [17], [23], [25], [58]. Recently, the release of several large-scale psychometric visual quality databases [29]- [32], [51] have sped the application of deep CNNs to perceptual video and image quality modeling. To conquer the limits of small data size, researchers have either proposed to conduct patch-wise dataaugmentation during training [59]- [61], or to pretrain deep nets on larger visual sets like ImageNet [62], then fine tune on target quality databases. Several authors report remarkable performance on synthetic distortion databases [63], [64] or on naturally distorted databases [29], [51].
Deep CNN models have also been employed for natural video quality prediction. Kim et al. [26] proposed a deep video quality assessor (DeepVQA) to learn spatio-temporal visual sensitivity maps via a deep CNN and a convolutional aggregation network. The V-MEON model [65] leveraged a multi-task CNN framework which jointly optimizes a 3D-CNN for feature extraction and a codec classifier, and using fully-connected layers to predict video quality. Zhang et al. [27] leveraged transfer learning to develop a general-purpose BVQA framework based on weakly supervised learning and a resampling strategy. In the VSFA model [19], the authors applied a pre-trained image classification CNN as a deep feature extractor, then integrated the frame-wise deep features using a gated recurrent unit and a subjectively-inspired temporal pooling layer, reporting leading performance on several natural video databases [31], [50], [66]. The authors then built an enhanced version of VSFA, dubbed MDVSFA [67], by employing a mixed datasets training strategy on top, training a single VQA model on multiple datasets, and reporting superior performance on publicly available datasets. Several other popular CNN-based BVQA models [19], [26], [27], [65], [67] produce accurate quality predictions on legacy (single  synthetic distortion) video datasets [1], [68], but struggle on recent in-the-wild UGC databases [31], [50], [66].

III. RAPID AND ACCURATE VIDEO QUALITY EVALUATOR (RAPIQUE)
Prior statistics-based video quality models have been shown to be capable of capturing complex UGC distortions, such as FRIQUEE [12], TLVQM [13], and VIDEVAL [6]. However, these models are subject to time-consuming executions since either expensive motion estimation or high-order statistical features are required. CNN models are able to efficiently capture high-level features, which have also been observed to be useful quality indicators [19], albeit directly applying a CNN on high-resolution video frames is expensive. Here we propose an efficient two-branch framework, as depicted in Fig.  1, which combines quality-aware, low-level NSS features with high-level, semantics-aware CNN features. The NSS features operate on higher-resolution spatial and temporal bandpass feature maps, while the CNN feature extractor is applied on a resized low-resolution frames for practical considerations. We also adopt a sparse frame sampling strategy when extracting features, which further accelerates the runtime. We present the details of RAPIQUE in the following.

A. Natural Scene Statistics
It has been observed that the spatial wavelet/subband coefficients of natural images exhibit strong regularities (Gaussianity) following a divisive normalization transform (DNT) [7]. A simple but effective form of divisive normalization, called mean subtraction and contrast normalization (MSCN), has been observed to accurately characterize image naturalness in multiple feature transforms [8], [10], [12], [49]. We develop NSS-based features following the methodology of FRIQUEE [12], which leverages multiple perceptually-relevant feature transforms to extract a large number of statistical features. RAPIQUE uses simple yet effective low-order bandpass statistics, achieving comparable performance as the complex and time-consuming high-order features used in FRIQUEE. We were inspired by the successful and efficient basic features developed as products of spatially-adjacent MSCN responses, and log-derivative statistics in BRISQUE [8] and DESIQUE [49], respectively. Specifically, let Y (i, j) be a given intensity image or a transformed feature map. The MSCN operator is applied on Y (i, j) to further decorrelate and Gaussianize the local pixels:Ŷ where, (i, j) are spatial indices and C = 1 is a constant that prevents instabilities caused by having a small variance in the denominators. The factors µ(i, j) and σ(i, j) are the weighted local mean and standard deviation within a spatial window centered at location (i, j) calculated by: where w = {w k, |k = −K, ..., K, = −L, ..., L} is a 2D isotropic, truncated, unit-volume Gaussian weighting function. We used K = L = 3 in our implementations. It has been empirically observed that the MSCN coefficients of images or video frames have characteristic statistical properties that are altered by the presence of distortion, and therefore, quantifying these deviations can help enable the prediction of perceived quality [8], [40]. A well-known model is the generalized Gaussian distribution (GGD) with zero mean [8]: where β = σ Γ(1/α) Γ(3/α) and Γ(·) is the gamma function: The two parameters are the shape α and the spread σ, of the zero-mean symmetric GGD, which are estimated using a popular moment-matching based method [69]. These are used as features to predict perceptual quality.
Another statistical observation is that the sample distributions of products of pairs of neighboring pixels in the MSCN coefficient map along four directions -horizontal (H) , and secondary-diagonal (D2) (Ŷ (i, j)Ŷ (i + 1, j − 1)) also exhibit a regular statistical structure, which are well-modeled as following a zero mode asymmetric generalized Gaussian distribution (AGGD) [8], [47]: An AGGD model has four parameters: ν controls the shape of the distribution, while (σ l , σ r ) are scale parameters that control the spread along each side of the mode; and η is the mean of the distribution, given by η = (β r − β l ) Γ(2/η) Γ(1/η) . Apart from the second-order pair-product statistics, we also extract another supplementary set of features by modeling the log-derivative statistics of neighboring MSCN coefficient pairs as introduced in [49]. Specifically, the absolute pixel values of Y (i, j) are first logarithmically transformed: then seven types of log-derivative statistics (Eqs. (8)) along six paired orientations -horizontal (H: ∇ x Z(i, j)), vertical (V: ∇ y Z(i, j)), main-diagonal (MD: ∇ xy Z(i, j)), secondary-diagonal (SD: ∇ yx Z(i, j)), horizontal-vertical (HV: ∇ x ∇ y Z(i, j)), and two combined-diagonals (CDs: , are modeled as GGD, respectively, after which the estimated GGD parameters are gathered as additional statistical features for learning the eventual quality predictor. For each pair log-derivative feature map, a single scale NSS model is used to derive two parameters (α, σ) by fitting a GGD distribution using the same moment-matching procedure, yielding a total of 14 additional features.
The variance field (or 'sigma' field) in Eq. (3) has been previously shown to provide effective quality-aware features deriving from the same NSS/retinal model [10], [12]. We extract two additional quantities from the variance field (Eq. (3)): the mean φ σ and square of the reciprocal of the coefficient of variation (CoV): where the CoV is ρ = (φ σ /ω σ ) 2 and where informative to include all these statistical features in the final prediction model.

B. Spatial Features
We built a basic statistical feature extraction module using the NSS features mentioned in the previous section, as summarized in Table I. Given an input image or feature map, it extracts two features (α, σ) from the MSCN transforms, two features (φ σ , ω σ ) from the variance field, 16 features 4 × (ν, η, σ l , σ r ) from the AGGD fit of MSCN adjacent pair products along 4 directions, and 14 features 7 × (α, σ) from GGD fits of paired log-derivatives along 7 directions, yielding a total of 34 features, which we dub NSS-34 for simplicity.
We also hypothesized that the NSS-34 operator is able to extract valuable color quality information if applied in a chromatic space, which we deem to be a more unified and efficient approach than crafting specific, complex features in different feature spaces, as is done in FRIQUEE [12]. The proposed NSS-34 feature set is based on fast, low-level statistics derived from GGD and AGGD models only, which we will show to deliver superior efficiency in the experimental section.
The gradient magnitude (GM) of a video frame is defined as the root mean square of directional gradients along two orthogonal spatial directions. GM is computed by convolving with a linear filter such as the Roberts, Sobel, or Prewitt. We utilized the Sobel kernels: whereby the GM of an image or frame I(i, j) is calculated by: where * denotes the convolution operator. It has been observed that two dimensional difference-of-Gaussian (DoG) or Laplacian-of-Gaussian (LoG) operators well-characterize the multiscale receptive fields of retinal ganglion cells [70]. We also extract two bandpass maps, using LoG and DoG, and extract their corresponding NSS-34 features, respectively. The LoG of image I is: where the LoG kernel is defined as: where g σ (x, y) is an isotropic Gaussian function with scale parameter σ. We used a window size of 9×9 for LoG filtering. While the GM and LoG are used by RAPIQUE to amplify high-frequency responses relating to local frame structures, the DoG is configured to capture mid-frequencies, expressive of structure at larger bandpass scales. The DoG response is defined as the difference of the responses of two Gaussian filters with different standard deviations To avoid redundant information between the LoG and DoG, only the first level of an N -level DoG decomposition with k = 1.6, σ i = k i−1 , i = 1, ..., N − 1 is utilized. Fig. 6 shows the differences between the GM, LoG, and DoG responses on a sample video frame. Overall, the four luma channel feature maps (Y, GM, LoG, DoG) (where Y = 0.299R + 0.587G + 0.114B) are fed into the NSS-34 module to obtain useful statistical features.
Images are naturally multiscale, and distortions affect image structures across scales. Incorporating multiscale information in quality models provides significant performance improvements [8], [77]. Hence, we extract NSS-34 features from the four luma feature maps (Y, GM, LoG, DoG) at two scales: the original image scale and a reduced (by a factor two) resolution. However, we only extract NSS-34 features at the half scale on the twelve chroma feature maps, since frames are often compressed in YUV420 format, which already contain chroma information in reduced scale; additionally, it has also been observed that humans are more sensitive to luma distortions than chroma distortions [55]. To sum up, the entire collection of spatial features is collected by applying two-scales NSS-34 to the luma feature maps and single-scale NSS-34 to the chromatic maps, yielding a total of 34 × 2 × 4 + 34 × 1 × 12 = 680 features. It is worth noting that this 680-dim spatial model is an improved alternative to the SOTA 560-dim FRIQUEE model [12] since it achieves comparable performance as FRIQUEE, but is 20x faster.

C. Temporal Features
Prior BVQA methods accounting for temporal distortions, however, either rely on expensive motion estimation [9], [13], or underperform on UGC videos by only accounting for simple frame-difference statistics [4], [47], [48], even including complex CNN models [19], [26]. Here we attempt to exploit more general temporal scene statistics of natural videos to develop and improve BVQA models. To the best of our knowledge, we propose the first general, effective and efficient temporal statistics model based on bandpass regularities of natural videos along the time dimension, going beyond simpler frame-difference models [78].
Inspired by the efficacy of temporal bandpass statistics in the prediction of frame rate-dependent video quality [54], our proposed temporal model utilizes 1D temporal bandpass representations. Specifically, consider a bank of K temporal bandpass filters denoted h k , k ∈ {0, ..., K − 1}, where k denotes the subband index. The temporal bandpass responses of a video F (x, t) (where x = (x, y) and t represents spatial and temporal co-ordinates, respectively) is where * and Y k are 1D temporal convolution operations and the bandpass response of the k th filter, respectively. Note that frame differences are a special case of Eq. (19) (the high-pass component of a 2-tap Haar wavelets). Fig. 7 visually illustrates the bandpass responses of a natural video from the LIVE-VQA dataset [1] using 3-level Haar wavelet filters. Attempting to generalize the spatial NSS as mentioned in Sec. III-A to the temporal domain, we instead analyze the statistics of the temporal bandpass coefficients Y k (x, t), k = 1, ..., 7 (ignoring the lowest band k = 0) by again applying MSCN transforms, as in Eq. (1), to further decorrelate the subband representations, over a set of frame time samples t ∈ {t 0 , t 1 , ..., t N } (note that t does not need to be densely sampled). Note that in Eq. (1), µ k (x, t) and σ k (x, t) are replaced by the local mean and standard deviation within a spatial window centered at location (x, t), for each subband k.
We have found that the MSCN coefficients of the temporal bandpass coefficients of natural videos also exhibit a Gaussianlike appearance, as shown in Fig. 8, while the regularities are modified by the presence of distortion, strongly suggesting the possibility of quantifying deviations to predict perceived video quality. We model the distributions of subband MSCN 4k-360p 20 MOS+σ [1,5] coefficients again using the pre-defined GGD and AGGD distributions, by merely passing them into the NSS-34 feature extractor (Sec. III-B). Similar to the spatial feature processing, we also extract the temporal statistical features over two scales (original and half scale), yielding a 476-dim feature vector ((34 features/band)×(7 subbands)×(2 scales) = 476). Inspired by the efficacy of standard deviation pooling as first introduced in GMSD [79] and later also shown effective when utilized for temporal pooling in [6], [13], we calculate the 680 spatial features at two frames per second within each non-overlapping one-second chunk, then enrich the feature set by applying average and absolute difference pooling [80] of the frame features within each chunk, based on the hypothesis that the variation of spatial features also correlates with the temporal properties of the video. Finally, all of the chunkwise feature vectors are average pooled across all chunks to derive the final set of features over the entire video.

D. Deep Learning Features
CNN-based solutions have been observed to generally perform well on UGC picture quality problems [17], [27], [51] thanks to several recently released large-scale picture quality datasets [17], [51], [82]. Still, none of them have proven effective on UGC video quality databases [30]- [32]. However, the authors of [6] have shown that the simple feature vector from an FC-layer, without fine-tuning, to be a useful quality indicator if training a shallow regressor on top. Therefore, we, for the first time, propose to leverage the best of both worlds, by combining powerful quality-aware NSS features as described in Sec. III-A, III-B, III-C, with pre-trained deep learning features, by jointly training a regressor on them to predict the final quality score.
One issue encountered when dealing with quality prediction problems is the mismatch of picture sizes between the standard inputs of CNN models such as VGG-16 [83], ResNet-50 [34], and IQA-valid high-resolution images. Two possible solutions have been attempted to solve this. The authors of [17], [61] suggested applying a CNN on spatially sampled small patches, then aggregating the locally predicted scores to obtain global quality scores. The authors of [51] presented a CNN operating on full-sized images, but with either global average pooling (GAP) or spatial pyramid pooling (SPP) [84], feeding FC layers. These two schemes, however, increase the computation load of the CNN models. Since our proposed model is already armed with powerful spatial and temporal quality-aware features, we added CNN features only to exploit its ability to capture high-level semantic information, supplementing the low-level NSS features. In this regard, we aggressively downscaled the frames to fit the CNN model  Fig. 9. Scatter plots and nonlinear logistic fitted curves of (c) RAPIQUE versus MOS, compared against (a) TLVQM [13] and (b) VIDEVAL [6], using a grid-search SVR using k-fold cross-validation on KoNViD-1k [31], LIVE-VQC [30], YouTube-UGC [32], and the All-Combined set (Sec. IV-A), respectively.
inputs when extracting these semantic-aware features, yielding greater efficiency than previous CNN VQA models. Another reason to use a pre-trained CNN without fine-tuning is to prevent overfitting, since existing video quality datasets are of limited sizes. In our implementation, we used a ResNet-50 (2,048-dim) as a semantic feature extractor.

E. Learning a Video Quality Predictor
We summarize the feature extraction process as follows. Since our goal is to build an efficient BVQA model, we devised spatial and temporal sampling strategies to further improve its speed. Specifically, given an input video F (x, t), RAPIQUE uniformly samples 2 frames per second, based on which the 680-dim spatial NSS features (in Sec. III-B) are extracted, then average and absolute-difference pools these to obtain 680 spatial and 680 temporal variation features, respectively. RAPIQUE also uniformly samples 8 consecutive frames each second, then applies temporal Haar filter (Eq. (7)) to extract 7 bandpass responses, from which 476 features are calculated at each time sample. The above features are calculated at a higher resized resolution while maintaining the aspect ratio (we used 512p in our experiments). However, the CNN backbone (ResNet-50) operates on resized frames at a sparse temporal sampling of 1 frame/sec to attain an additional 2,048 features.
After obtaining all the spatial, temporal, and CNN features within each one-second chunk, we adopt a simple approach to concatenate them all into a totally 3884-dimensional feature vector for each video chunk, then average-pool each to obtain a single 3884 feature vector over the entire video. A shallow or deep regressor head can then be trained on the aggregated feature vector to predict the final video quality scores.

A. Experiment Settings
Datasets and Baselines: We used three recent BVQA datasets as testbeds for the performance evaluations: KoNViD-1k [31], LIVE-VQC [30], and YouTube-UGC [32], as summarized in Table II. We also used the combined set (denoted All-Combined) as introduced in where equations (20) and (21) are used to calibrate KoNViD-1k and LIVE-VQC, respectively (YouTube-UGC does not need to be changed). Here y adj denotes the adjusted scores, while y org is the original MOS. We refer the reader to [6] for details regarding assumptions and derivations of the calibration process.
Evaluation Method: We used a support vector regressor (SVR) as the back-end regression model to learn the featureto-score mappings [8], [9], [12], [13], [61]. We optimized Fig. 10. Performance comparison of SRCC/PLCC as a function of the percentage of the content used to train the compared blind VQA models on the composite All-Combined set. Note that this result is self-explanatory as we used a slightly different evaluation method (20 iterations, SVR with randomized search cross-validation) compared to previous experiments. the SVR parameters (C, γ) via a randomized grid-search on the training set. Following convention, we randomly split the dataset into training and test sets (80%/20%) over 20 iterations, and the overall median test performance was reported. All of the evaluated methods were implemented using the original release by the respective authors. Four performance metrics were used: the Spearman Rank-Order Correlation Coefficient (SRCC) and the Kendall Rank-Order Correlation Coefficient (KRCC) are non-parametric measures of prediction monotonicity, while the Pearson Linear Correlation Coefficient (PLCC) with corresponding Root Mean Square Error (RMSE) were computed to assess prediction accuracy. Note that PLCC and RMSE are computed after performing a nonlinear fourparametric logistic regression to linearize the objective predictions to be on the same scale as MOS [1]: Table III shows the main comparison results on the four evaluated datasets. It may be observed that RAPIQUE achieved the best performance on KoNViD-1k, even outperforming the most recent, dense deep learning models such as VSFA and MDVSFA. On LIVE-VQC, which contains many mobile videos exhibiting large camera motions [6], TLVQM, which contains numerous heavily crafted motion-relevant features, was the best performer. However, RAPIQUE ranked a clear second, indicating that the temporal NSS features in RAPIQUE are powerful indications of temporal and motionrelated distortions.

B. Main Evaluation Results
The most recent deep still picture quality models, Kon-Cept512 and PaQ-2-PiQ, have been observed to perform poorly on UGC-VQA datasets. One reason for this is that these models were trained on picture quality datasets [17], [51], containing strictly spatial content and distortions. A leading blind deep video quality model, V-MEON, also does not perform well, likely because it was trained on compression artifacts rather than on complex combinations of UGC distortions.
On the larger datasets, RAPIQUE delivered the second-best correlation against the subjective data on YouTube-UGC, only slightly worse than the current SOTA model VIDEVAL, while RAPIQUE ranked the best on the 3165-video composite set, All-Combined. Since VIDEVAL was created by a supervised feature selection process (using subjective labels) on the composite combined set, wherein YouTube-UGC accounts for a large portion, it would be expected to outperform on these two sets. The RAPIQUE model, on the contrary, is databaseagnostic, and also exhibited uniformly well performance on all four test sets. In this regard, RAPIQUE has the potential to perform better on future larger-scale datasets and in realworld application scenarios it has not been exposed to. The scatter plots and fitted curves of RAPIQUE predictions versus MOS in Fig. 9 visually demonstrate that the performance of RAPIQUE remains stable on video sequences from different databases, achieving smaller RMSE on larger databases.

C. Effects of Training Data Size
To study the degree of performance variation by the compared algorithms, we vary the training-test splits from 10% to 90% of the content used for training, using the rest for testing on the composite combined set. As might be seen in Fig. 10, the RAPIQUE model was able to already achieve better than 0.8 in PLCC provided only 50% of the data for training. When compared to SOTA methods, although RAPIQUE was not observed to outperform VIDEVAL when the fraction of training data was less than 40%, it delivered improved performances relative to VIDEVAL and TLVQM as the proportion of training data was increased, as shown in Fig. 10. This suggests that RAPIQUE is very data-efficient, with the potential to achieve even better results when largerscale datasets become available.

D. Ablation Study
To analyze the importance of each module in RAPIQUE, we conducted an ablation study. Fig. 11 shows the incremental performance attained when adding each module sequentially. It is worth mentioning the dataset biases of the evaluated benchmarks. For example, the authors of [6] observed that the LIVE-VQC videos generally contain more (camera) motions and temporal distortions than other databases, while spatial distortions predominate on KoNViD-1k and YouTube-UGC. It may be observed in Fig. 11(a) that the spatial NSS module (Sec. III-B) performs quite well on the UGC databases that mainly present spatial distortions, like KoNViD-1k and YouTube-UGC, indicating its efficacy in capturing authentic spatial distortions. LIVE-VQC, which mainly contains videos with large motions, challenges the spatial NSS module, aligning with the empirical observations made above [6]. Adding spatial variation and temporal NSS features (Sec. III-C) improves the performance of RAPIQUE on LIVE-VQC, indicating that these two types of temporal features capture important attributes of motion-intensive videos. Interestingly, we also noticed that including the SpatialNSS-Var features degraded performance on YouTube-UGC. It is possible that the SpatialNSS-Var features are redundant with SpatialNSS features on YouTube-UGC, causing the training algorithm to underperform. We also observed that temporal statistics did not contribute much to the assessment of Internet UGC videos from YouTube and KoNViD-1k (Flickr).
It is also important to note that including deep learning features (Sec. III-D) significantly boosts the performance over only using NSS features on all these UGC datasets, further validating our assumptions expressed in Sec. II-B, that high-level semantic features are also informative when conducting UGC video quality prediction. To better understand which types of videos are advantageously analyzed by the CNN features, we divided the combined set into three subsets of differing contents: 2,667 natural videos, 163 screen contents, and 209 gaming videos, as shown in Fig. 11(b). Notably, we observed that the CNN features provided more benefits on screen content and gaming videos than on natural videos. The new temporal statistical features yielded noticeable improvements relative to using only spatial features. Lastly, our deployment of CNN modules is essentially different from other methods [17], [51] in that RAPIQUE only requires a single pass of the resized frames (224x224), making it highly advantageous in application scenarios having high-speed requirements.

E. Performance on different deep features
To determine which kinds of deep features most effectively complement the proposed NSS features, we conducted another ablation study. We compared the performance of RAPIQUE variants that use features from different backbones: VGG-19, ResNet-50, PaQ-2-PiQ (trained on LIVE-FB [17]), and KonCept512 (trained on KonIQ-10k [51]). Since PaQ-2-PiQ was designed for local quality prediction, we included the predicted 3 × 5 local quality scores along with the single global score. For KonCept512, the 256-dim feature vector immediately before the last linear layer in the fully connected head was included. We also included VGG-19 and ResNet-50, except for they were pre-trained on ImageNet classification.
The overall performance results are tabulated in Table IV. It may be observed that combining NSS features with ResNet-50 yielded the best or top performances on all benchmarks,   13. Our proposed RAPIQUE model enables high-resolution video quality prediction at significantly lower runtimes than existing BVQA methods. Particularly, as seen in the plot our model is 2-150x faster than baselines, depending on resolution, and the higher, the faster. slightly better than KonCept512, suggesting that features pretrained on classification tasks provide valuable high-level semantic information to the quality assessment process. Moreover, using features pre-trained on a specific IQA dataset may limit model generalizability to future, unseen distortions. Another important reason why we prefer ResNet-50 over Kon-Cept512 is the gigantic model size of the InceptionResNetV2 [85], used as the backbone of KonCept512.

F. Complexity and Runtime Comparison
Apart from performance analysis, computational efficiency is also of great importance for BVQA models. Thus, we also study the model (feature) dimension and runtime comparisons in Table V. For a fair comparison, all the experiments were carried out in the same desktop computer, a Dell OptiPlex 7080 Desktop with Intel Core i7-8700 CPU@3.2GHz, 32G RAM, and GeForce GTX 1050 Graphics Cards. The models were implemented using their original releases on MATLAB R2018b and Python 3.6.7 under Ubuntu 18.04.3 LTS system. It should be noted that our comparison of computing complexity involves methods that use different sampling rate (or FPS), which is critical to model efficiency. However, we regard the design of FPS itself as an important aspect of BVQA algorithms, and thus our comparison still provides insights on developing more efficient BVQA models.
It may be seen that RAPIQUE is extremely efficient as compared to other complex top-performing BVQA models like TLVQM and VIDEVAL. Specifically, RAPIQUE is 10x faster than TLVQM, which also aims to efficiency. Fig. 12 shows the scatter plots of SRCC versus runtime, which indicates that RAPIQUE achieves comparable prediction accuracy, but with 20x less computational expense as compared to VIDEVAL, the current SOTA model on the UGC-VQA problem [6]. We observe however that, CNN models that benefit from optimized low-level implementations are generally faster than NSS models executed in MATLAB; we have observed a ∼10x speedup by switching from CPU to GPU on the CNNbased models, KonCept512, PaQ-2-PiQ, V-MEON, VSFA, and MDVSFA.
Predicting the quality of videos having multiple diverse resolutions is also a pressing problem, but has barely been discussed, since most video datasets only contain single-resolution contents. Thanks to the large-scale dataset, YouTube-UGC [32], which contains videos at five different resolutions, we were able to extend the complexity analysis to videos ranging from 540p to 4k, to study computational scalability with respect to video size. Fig. 13 compares computation time as a function of video resolution. We may observe that RAPIQUE has superior computational scalability in terms of data sizes, making it attractive and preferable for potential real-time, low-latency, and light-weight applications requiring high-resolution video inputs. Particularly, as seen in the plot our model is 2-150x faster than baselines, depending on resolution, and the higher, the faster. In Table VI we list the partial compute time of each sub-module in RAPIQUE on 1080p videos. Since all of the NSS-based features are implemented in MATLAB, a high-level prototyping tool, we would expect further accelerations to be possible (by orders-ofmagnitude) if implemented in low-level languages like C/C++, or GPU-friendly frameworks such as Tensorflow or PyTorch.

V. CONCLUSION
We have proposed an effective and efficient model for predicting the subjective quality of user-generated videos, which we call the Rapid and Accurate Video Quality Evaluator (RAPIQUE). The model, for the first time, leverages a composite of spatio-temporal scene statistics features and deep CNN-based high-level features in a two-branch framework, then jointly learns a regressor head for video quality prediction. Within the model, we developed new spatial scene statistics models in an efficient way and further extended the overall model to include normalized temporal bandpass responses, yielding the first general efficacious temporal NSS model for UGC video quality problems. Experiments on recent large-scale UGC video databases show the superior accuracy and efficiency of the proposed model in that it achieves competitive or substantially higher accuracy than both SOTA conventional as well as deep learning video quality models. RAPIQUE is computationally less expensive by orders-ofmagnitude than the most accurate benchmark methods and scales remarkably well with video resolution. To support reproducible research, an implementation of RAPIQUE is available on https://github.com/vztu/RAPIQUE.