{'fraction_non_alphanumeric': 0.03438185808339429, 'fraction_numerical': 0.014601316752011704, 'mean_word_length': 4.777852916314455, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DCVQE: A Hierarchical Transformer for Video Quality Assessment

The explosion of user-generated videos stimulates a great demand for no-reference video quality assessment (NR-VQA). Inspired by our observation on the actions of human annotation, we put forward a Divide and Conquer Video Quality Estimator (DCVQE) for NR-VQA. Starting from extracting the frame-level quality embeddings (QE), our proposal splits the whole sequence into a number of clips and applies Transformers to learn the clip-level QE and update the frame-level QE simultaneously; another Transformer is introduced to combine the clip-level QE to generate the video-level QE. We call this hierarchical combination of Transformers as a Divide and Conquer Transformer (DCTr) layer. An accurate video quality feature extraction can be achieved by repeating the process of this DCTr layer several times. Taking the order relationship among the annotated data into account, we also propose a novel correlation loss term for model training. Experiments on various datasets confirm the effectiveness and robustness of our DCVQE model.


Introduction
Recent years have witnessed a significant increase in user-generated content (UGC) on social media platforms like Youtube, Tiktok, and Weibo. Watching the UGC videos on computers or smartphones has even become part of our daily life. This trend stimulates a great demand for automatic video quality assessment (VQA), especially in popular video sharing/recommendation services.
UGC-VQA, also known as blind or No-Reference video quality assessment (NR-VQA), aims to evaluate in-the-wild videos without the corresponding pristine reference videos. Usually, UGC videos may suffer from complex distortions due to the diversity of capturing devices, uncertain shooting skills, compression, and poor editing process. Although many excellent algorithms have been proposed to evaluate video quality, it remains a challenging task to assess the quality of UGC videos accurately and consistently.
Besides the frame-level image information, temporal information is regarded as a critically important factor for video analysis tasks. Although many image quality assessment (IQA) models [35,67,33,61,22,14,62,60,64] can be applied to VQA base on a simple temporal pooling process [52], these models may not work very robustly because of the absence of proper time sequence aggregation. For instance, Fig. 1 shows three consecutive frames extracted from Video B304 in LIVE-VQA [48] dataset. As seen, the motion blur distortion appears on the actress's hand area. These frame images are most likely to be recognized as middle, even low quality when applying a sophisticated IQA method to them individually. However, the quality of this video was labeled as high by human annotators, because a very smooth movement of the actress can be observed when playing the video stream. RNNs [1,2,23] and 3D-CNNs [20,6,13,12,25] are potential models to integrate spatial and temporal information for NR-VQA. Though these algorithms perform well on many datasets, the difficulty of parallelizing RNN models and the non-negligible computational cost of 3D-CNNs make them infeasible to be applied to many internet applications that require quick responses, just like the online video sharing/recommendation services.
On the other hand, research on NR-VQA is often limited by the lack of sufficient training data, due to the tedious work to label the mean opinion score (MOS) for each video. Among the publicly available datasets, MCL-JCV [56], VideoSet [57], UGC-VIDEO [27], CVD-2014 [38], LIVE-Qualcomm [16] are generated in lab environments, while KoNViD-1k [18], LIVE-VQC [48], YouTube-UGC [58] and LSVQ [63] are collected in-the-wild. As mentioned above, one could consider VQA as a temporal extension of IQA task, thus can apply some frame-level distortion and ranking processes [28] to augment the small video datasets for algorithm development. However, in-the-wild videos are usually hard to synthesize, since they may suffer from compound distortions which cannot be exactly parameterized as a combination of certain distortion cases. Recently, a large-scale LSVQ dataset [63], which contains 39,075 annotated videos with authentic distortions is released for public research. We know that not many studies have been conducted so far based on this new dataset.
Additionally, most previous works take L1 or L2 loss [64,63,70,23,54] as the optimization criterion for model training. Since these criteria to some extent ignore the order relationship of quality scores of the training samples, the trained model may be not stable to quantify the perceptual differences between the videos with similar quality scores. For example, in our research we find that many existing NR-VQA models work well to identify both high and low quality videos, but struggle to distinguish the videos with middle quality scores. How to effectively quantify the difference between samples with similar perceptual scores therefore becomes the key to the success of a NR-VQA model.
To address the above problems, in this paper we put forward a new Divide and Conquer Video Quality Estimator (DCVQE) model for NR-VQA. We summarize our contributions as follows: (1) Inspired by our observation on the actions of human annotation, we propose a Divide and Conquer Transformer (DCTr) architecture to extract video quality features for NR-VQA. Our algorithm starts from extracting the frame-level quality representations. Regarded as a divide process, we split the input sequence into a number of clips and apply Transformers to learn the clip-level quality embeddings (QE) and update the frame-level QE simultaneously. Subsequently, a conquer process is conducted by using another Transformer to combine the clip-level QE to generate a video-level QE. After stacking several DCTr layers and topping with a linear regressor, our DCVQE model can be constructed to predict the quality value of the input video. (2) By taking the order relationship of the training samples into account, we propose a novel correlation loss to bring an additional order constraint of video quality to guide the training. Experiments indicate that the introduction of this correlation loss can consistently help to improve the performance of our DCVQE model. (3) We conduct plenty of experiments on different datasets and confirm that our DCVQE outperforms most other algorithms.

Related works
Traditional NR-VQA solutions: Many prior NR-VQA works are "distortion specific" because they are designed to identify different distortion types like blur [31], blockiness [39], or noise [49] in compressed videos or image frames. More recent and popular used models are deployed on natural video statistics (NVS) features, which are created by extending the highly regular parametric bandpass models of natural scene statistics (NSS) from IQA to VQA tasks. Among them, successful applications have explored in both frequency domain (BIQI [36], DIIVINE [37], BLINDS [44], BLINDS-II [45]) and spatial domain (NIQE [35], BRISQUE [33]). V-BLIINDS [46] combined spatio-temporal NSS with motion coherency models to estimate perceptual video quality. Inter-subband correlations, modeled by spatial domain statistical features in frame differences, were used to quantify the degree of distortion in VIIDEO [34]. 3D-DCT was applied on local space-time regions, to establish quality aware features in [26]. Based on hand-crafted features selection and combination, recent algorithms VIDEVAL [53] and TLVQM [21] demonstrated outstanding performance on many UGC datasets. The designation of these hand-crafted features is deliberate, though they are hard to be deployed in an end-to-end fashion for NR-VQA tasks.
Deep learning based NR-VQA solutions: Deep neural networks have shown their superior abilities in many computer vision tasks. With the availabil-ity of perceptual image quality datasets [15,19,41,64], many successful applications have been reported in the past decade [28,64,29,50,70]. Combining with a convolutional neural aggregation network, DeepVQA [59] utilized the advantages of CNN to learn spatio-temporal visual sensitivity maps for VQA. Based on a weakly supervised learning and resampling strategy, Zhang et al. [69] proposed a general purpose NR-VQA framework which inherited the knowledge learned from full-reference VQA and can effectively alleviate the curse of inadequate training data. VSFA [23] used a pretrained CNN to abstract frame features, and introduced the gated recurrent unit (GRU) to learn the temporal dependencies. Following VSFA, MDTVSFA [24] proposed a mixed datasets training method to further improve VQA performance. Although the above methods performed well on synthetic distortion datasets, they may be unstable to analyze UGC videos with complex and diverse distortions. PVQ [63] reported a leading performance on the large-scale dataset LSVQ. For a careful study on the local and global spatio-temporal quality, spatial patch, temporal patch and spatiotemporal patch were introduced in [63]. RAPIQUE [54], by leveraging a set of NSS features concatenated with learned CNN features, shown the top performance on several public datasets, including KoNViD-1k, YouTube-UGC, and their combination.
Transformer techniques in computer vision: Self-attention mechanismbased Transformer architecture shows its exceptional performance in natural language processing [55,9]. Recently, many researchers introduced Transformer to solve computer vision problems. ViT [10] directly run attention among image patches with positional embeddings for image classification. Detection Transformer (DETR) [5] reached a comparable performance with Faster-RCNN [43] by designing a new object detection systems based on Transformers and bipartite matching loss for direct set prediction. Through conducting contrastive learning on 400 million image-text pairs, CLIP [42] shown impressive performance to solve different zero-shot transfer learning problems. For IQA tasks, Transformer also shows its powerful strength. Inspired by ViT, TRIQ [66] connected Transformer with MLP head to predict perceptual image quality, where sufficient lengths of positional embeddings were set to analyze the images with different resolutions. IQT [7] achieved outstanding performance by applying Transformer encoder and decoder on the features of reference images and distorted images. Through introducing 1D CNN and Transformer to integrate short-term and long-term temporal information, a recent work LSCT [65] demonstrated excellent performance on VQA. Our proposal is also derived from Transformer, inspired by our observation on the actions of human annotation. Experiments on various datasets confirm the effectiveness and robustness of our method.

Divide and Conquer Video Quality Estimator (DCVQE)
Human judgements of video quality are usually content-dependent and affected by their temporal memory [3,11,32,47,51,68,57]. In our investigation, we notice that many human annotators like to give their opinions on the quality of a video after the following two actions: first, watch the video quickly (usually in the fast forward mode) to get an overall impression of its quality, then they may scroll mouse forward and backward to review some specific parts of the video for their final decisions. Inspired by this observation, we propose a hierarchical architecture, dubbed Divide and Conquer Video Quality Estimator (DCVQE) for NR-VQA. Our model is worked by extracting three levels of video quality representations from frames, video clips to whole video sequence progressively and repeatedly, somewhat similar to the reverse processes of human annotation. An additional correlation loss term is also presented to bring an additional order constraint of video quality to guide the training. We find that our method can effectively improve the performance of NR-VQA. We will describe our work in detail in the following paragraphs.

Overall Architecture
The left side of Fig. 2 represents the architecture of a key video quality analysis layer, Divide and Conquer Transformer (DCTr) in our proposal. In order to simulate the second action of human annotation mentioned above, we split the input sequence into a number of clips, and introduce a Transformer module T ransf ormerD to learn quality representations for each clip. As shown in Fig.  2, for the k th DCTr layer, we split the whole sequence into I clips, and each clip covers J frame-level quality representations generated by the previous layer. For the i th (1 ≤ i ≤ I) clip C k i , a module T ransf ormerD i is applied to combine the two levels of quality embeddings (QE) generated by the previous layer, that is, all the J frame-level QE F k−1 i,j (1 ≤ j ≤ J) and the video-level QE Q k−1 , to simultaneously learn the clip-level QE C k i and update the frame-level QE F k i,j for the current layer. We further simulate the first action of human annotation by integrating the learned clip-level QE to generate a video-level QE. The other Transformer module T ransf ormerC with a topped average pooling layer are proposed to merge all clip-level quality representations C k i to predict the videolevel QE Q k for the current layer. In general, our proposed video quality analysis module is constructed in a divide and conquer format, so we coin it as a Divide and Conquer Transformer (DCTr) layer.
The overall architecture of our DCVQE model is shown in the right side of Fig. 2. As seen, we stack several DCTr layers (3 layers for our proposal, refer to supplementary material for details about this setting) to extract the final video-level QE Q k for the input video. The final quality score is predicted by topping this embedding Q k with a regressor. In practice, to improve the temporal sensitivity of our model, we progressively expand the coverage of video clips in the DCTr as the layers deepen. For example, supposing that each clip in the k th DCTr layer covers J frame-level QE from the previous layer, the coverage of each clip in the (k+1) th DCTr layer will be 2J, which means two neighbor clips of the k th DCTr layer are combined to extract the clip-level QE in the (k+1) th DCTr layer. For the first DCTr layer, the input frame-level QE F 1 * , * is generated by our feature extractor described in subsection 3.2, and the input video-level QE Q 1 is initialized randomly. Both of them are integrated with positional embeddings P * , * [9] for the following processes.
T ransf ormerD and T ransf ormerC are the two most important modules in our DCVQE. As shown in Fig. 3, they have similar architecture to the classic Transformer [55]. Three fully connected layers are used to convert the input features into Query, Key, and Value tensors. Split operations are then adopted to group these three tensors into H heads, respectively. For the input frame-level feature with the shape of B ×S ×D, where B, S and D denote the batch size, sequence length and dimension of the frame embedding respectively, the shapes of its corresponding grouped Query, Key and Value tensors will be H×B×S/H×D. The attention weights can be computed by the dot-product operations between the grouped Query and grouped Key tensors. To make T ransf ormerD module more concerned with the clip-level quality information, except the first position is reserved for the input video-level QE, we mask out the attention weights outside a predefined temporal range by using sequence masks (the dashed square plotted in Fig. 3) and adopt a Softmax layer to normalize the rests. The normalized weights are then applied to the grouped Value tensor to update the quality features. A stack operation is conducted on the head axis to recover the shape of the tensor. Here we also introduce the residual technique [17] to improve the robustness of T ransf ormerD module, so the final output of this module is the addition of the recovered features and residual. T ransf ormerC module is designed to update the video-level QE for each DCTr layer. The difference between T ransf ormerC and T ransf ormerD modules is that T ransf ormerC module does not contain sequence masks and residual block, as shown in Fig. 3.

Feature Extraction
To avoid the additional distortions that may be caused by applying some preprocessing steps to the input data, such as image resizing and filtering, we take the original full-size image frames as the input of our model. The pretrained Resnet-50 [17] is adopted as the CNN backbone of our feature extractor. To increase the sensitivity of the feature extractor to capture frame-level distortions, we first conduct a typical IQA task to fine-tune the CNN backbone. As shown in the top figure of Fig. 4, all CNN layers except the last group of CNN blocks are frozen for fine-tuning. We find this process can help to transfer the learned knowledge from ImageNet [8] to our application. It will be verified by our experiments.
Based on the fine-tuned backbone, as the bottom figure of Fig. 4 shows, the quality feature of each input frame is then extracted by adopting a concatenation operator to the outputs of global average (AVG) pooling and global standard deviation (STD) pooling processes [23,24]. Since both the dimensions of the outputs of AVG pooling and STD pooling are 2048, the frame-level QE for each frame will be a vector with a dimension of 4096. In practice, refer to [23], we further add a fully connected (FC) layer to reduce the dimension of the framelevel QE to 128. It is an efficient way to balance the accuracy of frame-level feature representation and overall computational cost.

Correlation loss
Prior works usually use the following L1 loss to optimize the model: where p n , n ∈ [1, N ] represents the n th predicted MOS, g n denotes its corresponding ground truth, N is the total number of videos. Mathematically, this criterion to some extent ignores the relative order relationship of quality scores of the training samples in a batch, it sometimes may lead to limited ability to quantify the perceptual differences between the videos with similar quality scores. For example, considering the training processes of two NR-VQA models based on L1 loss criterion, for two video samples A and B with the MOS ground truths of 7.0 and 7.1, if their quality scores are predicted as P A1 = 7.1, P B1 = 7.0 and P A2 = 6.9, P B2 = 7.0 by the two models respectively (where P A1 denotes the quality score of video A predicted by model 1, and so forth), the same strengths coming from the L1 losses (0.1) of these two training samples will be contributed to optimize the models correspondingly. Though it is hard to say which model is better than the other, in our opinion, it may be easier to optimize model 2 because of a positive correlation between the ground truths and its loss. Based on this idea, a new correlation loss, aiming to bring an additional order constraint of video quality to guide the training procedure, is proposed as follows: where p m represents the prediction of a training sample in a batch with size N , and p n denotes the n th prediction which is picked to compare the ranking order with each element in the prediction set. g m and g n stand for the corresponding ground truth scores of p m and p n , respectively. This loss equation can be considered as that, we take the n th video in the batch as an anchor and compute the positive or negative correlations between predictions and ground truths for this anchor and any other one in the batch. Eq. 2 can be further simplified as: wherep andḡ represent the mean values of predictions and ground truths, respectively. We find that this simplified L c term is actually equivalent to the Spearman correlation coefficient without normalization. A total loss L, which combines the above two loss equations (1) and (4), is finally constructed to optimize our DCVQE model as where α and β represent the weights of L 1 loss and L c loss. We will discuss the optimal settings for these two weights in subsection 4.4.

Datasets
We conduct experiments on four in-the-wild VQA datasets: KoNViD-1k [18], LIVE-VQC [48], YouTube-UGC [58], and LSVQ [63]. In these datasets, KoNViD-1k contains 1,200 unique contents with a duration of 8 seconds; LIVE-VQE, the smallest among them, consists of 585 video contents; YouTube-UGC contains 1,500 UGC video clips sampled from millions of YouTube videos; A recently released dataset LSVQ, containing 39,075 video samples with diverse durations and resolutions, is the largest publicly available UGC datasets for research right now. We also follow the suggestion given in [53,40] to select YouTube-UGC dataset as the anchor and map MOS values of KoNViD-1k and LIVE-VQC data onto a common scale to generate an All-Combined dataset for performance evaluation. The typical random 60-20-20 strategy [23] is applied to split data into three sets, i.e. 60% for training, 20% for evaluation, and the remaining 20% for testing. Especially, for LSVQ we follow the setting in [63] to first generate a Test-1080p set and then conduct the random 80-20 splitting on the rests to generate training and testing sets for experiments.

Evaluation metrics
Here we adopt four commonly used criteria, Spearman Rank-Order Correlation Coefficient (SRCC), Kendall Rank-Order Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC) and Root Mean Square Error (RMSE) to measure prediction monotonicity and prediction accuracy. For fair comparisons, we conduct each evaluation 100 times individually and report the median values of these four metrics as their final results.

Implementation Details
We first fine-tune ImageNet-pretrained Resnet-50 backbone on KonIQ-10k dataset [19] with random 80-20 splitting and maximum 20 epochs. The CNN backbone with the best performance of SRCC on the test set is selected to construct the feature extractor for our VQA task. The frame-level features are extracted using this feature extractor, where all frames of each video are considered without sampling. The temporal range of the sequence mask in the proposed T ransf ormerD (see Fig. 3) is empirically set to 15. For a tradeoff between GPU memory, running speed, and performance, we set the maximum length of each input video to 600 (i.e. only the first 600 frames of a video are considered if the video is longer than this maximum length) and the maximum training epoch to 75. An evaluation process will be conducted after each epoch, and the model with the lowest loss on the validation set will be stored.

Ablation Studies
In this subsection, we perform ablation studies to better understand the different components of the proposed DCVQE model. CNN Backbone: Training of the CNN feature extraction backbone can also be regarded as a typical IQA task. Two series of architectures based on ResNet and ViT backbones are fine-tuned fully or partially in our study. Here we note the partial fine-tuning means that only the last blocks of these models, topped with average pooling and fully connected layers (see Fig. 4) are trainable. We determine that fully fine-tuned backbones act negatively with regard to the complexities of the networks. i.e. deeper structure results in worse performance, and partial fine-tuned backbones usually work better than fully fine-tuned ones for both two series of architectures. We also incorporate a new attention-based IQA architecture PHIQNet [65] to our study, and similar performance as partial fine-tuning on ResNet is observed (refer to supplementary material for details). In our application, for a tradeoff between model complexity and performance, also a fair comparison with previous work [23], we select the partially fine-tuned ResNet-50 as the feature extraction backbone to construct our DCVQE model.
Temporal Range of Sequence Mask: We introduce a group of experiments on KoNViD-1k dataset to study how the temporal range affects the DCVQE model. By directly replacing the proposed DCTr layers with the traditional Transformer layers [55], a baseline Transformer model is constructed for performance comparisons. Test results are plotted in Fig. 5. From these figures, we can see that the best temporal range for the baseline Transformer model is about 9, and the performance of this baseline model slightly decreases as the temporal range increases. Especially if we set the range to "all", which means that all the frame-level representations are involved in the self-attention processes [55] of the Transformer, the performance of this baseline model drops dramatically. Switching to our DCVQE, we can see that the overall performance stays at a high level. The best performance can be observed when the temporal range reaches 15. The results of the "all" setting for DCVQE, for which one frame only conducts self-attention with all frames in its own clip, also demonstrate the robust performance of our DCVQE model.
Correlation Loss: To show how the proposed Correlation Loss (CL) contributes to our VQA task, the performances of DCVQE with different weight settings of CL and L1 loss terms are reported in Table 1. The best performance can be identified when L1 Loss weight α reaches 0.7 and CL weight β reaches   Here we note that our CL does not intend to play a critical role in model optimization since it tries to describe the relative order relationships between videos, particularly for those videos with similar quality scores, so the overall improvement by adding this CL term may not look very significant. Overall, we confirm that our model can achieve better performance with the introduction of CL (Eq. 4). The rest experiments are conducted with the setting of α = 0.7 and β = 0.3 if not specified.
Component by Component Ablation: We further stack the key parts in our DCVQE component-by-component to analyze their impacts individually and gradually. To additionally study the effectiveness of our new CL term, a well-known pairwise ranking loss (PW-RL) [4], which is similar to our proposal, is also involved in our experiments. As shown in Table 2, compared with Vanilla-Transformer (row 1), only introduces the Divide operation (row 2) does not impact the system's performance. In row 3 we employ the AVG pooling operator to integrate clip-level embeddings to generate video-level embeddings, Table 2. Component-by-component comparisons of several key parts in our DCVQE on KoNViD-1k dataset. Here we note that L1 loss is a default setting for all tests. Rows 3 and 4 apply AVG pooling (AVG) and T ransf ormerC (T ransC ) respectively to conduct the "conquer" operation. "TR 15"s in rows 5, 6, 7, and 8 mean the temporal ranges of sequence masks are set to 15  and the results show that this simple "conquer" operation can slightly improve the performance. In row 4 the conquer operation is switched to the proposed T ransf ormer C , where similar correlation scores with a lower RMSE are presented (compared with row 3). The temporal range of sequence mask 15 is introduced to tests in rows 5 and 6, where we can see that this change apparently benefits the system. Furthermore, the losses CL and PW-RL are compared directly in rows 7 and 8. The results confirm that the proposal CL provides the best performance for our VQA task.

Comparison with the State-of-the-art Methods
We compare our method with 13 state-of-the-art VQA methods, which can be roughly divided into NR-IQA based methods: BRISQUE [33], GM-LOG [61], HIGRADE [22], FRIQUEE [14], CORNIA [62], HOSA [60], Koncept-512 [19], PaQ-2-PiQ [64], and NR-VQA based methods: V-BLIINDS [46], TLVQM [21], VSFA [23], VIDEVAL [53] and RAPIQUE [54]. For NR-IQA based methods, we abstract one frame feature per second and regard the average value of the features as the video representation to train a support vector regressor (SVR) for prediction [54,53,63]. Test results are reported in Table 3 and the best solution for each evaluation metric is marked in bold. Since it is hard to reproduce VIDEVAL and RAPIQUE methods, in Table 3 we directly cite the results given in [53,54] (marked as "*"). Different from our default 60-20-20 data splitting setting, these two methods used the random 80-20 strategy to split data for training and testing. For a fair comparison, we train an additional version of the DCVQE model under the same 80-20 data splitting setting. Test results of this additional DCVQE model are marked as " †". As seen from this table, our model (DCVQE †) significantly outperforms the current top method RAPIQUE on KoNViD-1K dataset by 4.37% SRCC and 2.45% PLCC, and greatly exceeds the second place method VIDE-VAL on YouTube-UGC dataset by 5.62% SRCC and 5.67% PLCC. TLVQM method performs best on LIVE-VQC dataset. Looking into this dataset, we can see that most videos are captured using mobile devices, thus camera motion blurring is one of the common distortions affecting the video quality. TLVQM introduces several hand-crafted motion-related features to handle this issue, so it achieves the best performance, especially compared with all the deep learning based solutions. Nevertheless, besides TLVQM, our proposal performs the best among the remaining. The evaluation on the All-Combined dataset of KoNViD-1k, LIVE-VQC, YouTube-UGC also confirms the robustness of our model. As shown in the column "All Combined", our method (DCVQE †) surpasses the second place method (RAPIQUE) by 2.84% SRCC and 1.12% PLCC. To demonstrate how well our model is learned from a given dataset, in Fig. 6 we also visualize the correlation between the video quality predictions and their ground truths (top row), as well as the video-level QE learned by the "conquer" operation of the last DCTr layer (bottom row). From the top row, we can see a strong correlation between our predictions and their ground truths. The figures in the bottom row plot the dimension-reduced video-level QE, where the gray value of each point is assigned with the normalized ground truth MOS score of the corresponding video sample, that is, the darker a point, the greater its corresponding MOS score. From these figures, we can observe that the points with higher MOS scores scatter on one side while the points with lower MOS on the other. It indicates that the separability of quality for those video samples has been significantly enhanced by using our DCVQE model. Additionally, we compare our model with several others on the recently released large-scale LSVQ dataset. The PVQ model [63], which was proposed along with this dataset, is also involved in our evaluation. Test results listed in Table  4 confirm the excellent performance of our model again.

Conclusions
Inspired by our observation on the actions of human annotation, in this paper, we propose a new Divide and Conquer Transformer (DCTr) architecture to extract the video quality features for NR-VQA. Starting from extracting the framelevel quality embeddings (QE) of an input video, two types of Transformers are introduced to extract the clip-level QE and video-level QE progressively in our DCTr layer. Through stacking several DCTr layers and topping with a regressor, a hierarchical model, named Divide and Conquer Video Quality Estimator (DCVQE) is constructed to predict the quality score of the input video. We also put forward an additional correlation loss regarding the order relationship among the training data to guide the training. Experiments confirm that our proposal outperforms most other methods. What is more, our model is purely deep learning based, compared with other top methods where NSS/NVS features are more or less needed, so we believe our proposal is more practical.

Optimal Number of DCTr Layers
We conduct the ablation study on KoNViD-1K dataset to find out the optimal number of DCTr layers to construct our DCVQE model. As listed in Table 3, only one DCTr layer does not adequately solve the VQA problem, while stacking 3 DCTr layers significantly increases the performance. Further increases in layers does not improve the performance. As a result, we set 3 as the optimal number of DCTr layers for our model.

More Studies on the Proposed Correlation Loss
To find out how the proposed correlation loss additionally helps to improve NR-VQA, we apply the proposed losses to train the baseline Transformer and DCVQE models, respectively. The well-known pairwise ranking loss (PW-RL) is also involved in our study. Learned from subsection 4.4 of the paper, the best performance can be achieved with a temporal range selected from 9 to 15, so we only conduct the experiments under 3 different range settings of 9, 12, and 15. The test results are shown in Fig. 1, where we can see that no matter which architecture and temporal range are selected, the introduction of our correlation loss can consistently help to improve VQA performance. The PW-RL is also comparably well to optimize our DCVQE model. However, its solution reaches the highest RMSE. The reason is that the PW-RL will be converted to cross-entropy loss for training so that the optimization strength might be too strong for pairs with wrong ranking orders but small Mean Opinion Score (MOS) differences. Fortunately, our correlation loss can better handle this situation because both ranking orders and MOS differences are considered.
Additionally, to show how the proposed correlation loss and architecture benefit real VQA tasks, we provide MOS prediction results of 4 KoNViD-1k sample videos in Table 4. From this table, we can see that (1) DCVQE cl maintains the order relation among the samples but DCVQE l1 and Vanilla-Transformer fail, and (2) both the Mean Absolute Errors (MAEs) of DCVQE cl and DCVQE l1 are lower than that of Vanilla-Transformer thanks to the new hierarchical architecture of DCVQE.

Computational Cost Analysis
Compared with the Vanilla-Transformer, our DCVQE model has a lower computational cost. For example, to calculate the attention weights for one single frame, the time complexity of the Vanilla-Transformer is O(DN ), while that of our DCVQE is O(D * N C ) because an input video will be split into a number of clips for processing (where D denotes the dimension of feature, N is the total frame size and C is the clip number).