{'fraction_non_alphanumeric': 0.030297201115706454, 'fraction_numerical': 0.01433105703568337, 'mean_word_length': 4.593329747175901, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Subjective Quality Evaluation of a Full Resolution Video with Eligible Number of Reference Pictures

The paper presents a new concept of video presentation for subjective quality evaluation methods with a reference given in parallel. The idea is to split a full resolution video into an n-picture matrix and encode each cell differently to compare aspects of processing such as encoding parameters or lossy compression algorithms. Conducted experiments show that it is possible to get more accurate results, shortening the time of evaluation at least by half with a less complicated and cheaper experimental station.


Introduction
Subjective user-perceived video quality ratings provided by a human audience and expressed as a mean opinion score (MOS) are still considered to be the most reliable measure of compressed video quality [1,2]. Aiming at unification, in its recommendations the International Telecommunication Union (ITU) describes the methods of conducting subjective experiments. There are methods that use single stimuli and others which use more than one picture to score quality. Additional pictures are used as a reference, which enables comparison and helps to formulate assessment on a given scale. When stimuli are displayed sequentially, such as in the case of pair comparison (PC) [3], degradation category rating (DCR) [3], or subjective assessment methodology for video quality (SAMVIQ) [4], the experiment takes more time and the measurement accuracy is decreased by memory effects [7]. This problem does not occur in the case of methods that use stimuli given in parallel, such as in the case of the simultaneous double stimulus for continuous evaluation (SDSCE) [5] or triple stimulus continuous evaluation scale (TSCES) [6], but in turn they require duplication of video encoders and monitors, which multiplies the price of the experimental station. This is a big drawback, especially in the case of new technologies such as 8K ultra-high-definition (UHD) television, for example. The other problem is synchronizing video signals and the unnatural viewing angle, for example, when the observer has to compare 8K UHD pictures. For those reasons, the number of pictures is limited to two for SDSCE or three for TSCES. This paper proposes a new method with multiple stimuli displayed in parallel on the same monitor. A similar concept was introduced by ITU [3], where two full frame pictures were displayed on the same monitor, but this could work in the case of lower resolution video only. The proposed method does not have this restriction: the full resolution picture is segmented into an n-parts matrix and each cell can be independently processed. The screen-splitting matrix defines the number and surface area of sub-images simultaneously given to the observer. The final image does not contain any lines separating the divisions (Fig. 1).
The number n of stimuli can be changed according to experiments needs. The limit is the observers' ability to perceive and evaluate more than one video at the same time.

Experiment
To evaluate the proposed method, three experiments were conducted: PC, PC-1×2, and PC-2×2. The PC experiment was a pair comparison (PC) method in accordance with ITU recommendations [3]. The observers were to choose the video of a better quality in each pair of images presented alternately one after each other. In the PC-1×2 experiment, each half of the frame was coded with different parameters and observers had to choose the better one. In the case of PC-2×2, the screen was divided into two rows and two columns and observers were asked to point out which quarter had the best quality and which had the worst. The purpose of the PC experiment was to obtain reference results for PC-1×2. The result of the PC-2×2 experiment was supposed to answer the question of whether the observer was able to recognize the quality of an image divided into quarters, each of which was coded with different parameters. Thirty-nine observers participated in the PC experiment, 16 in the PC-1×2 test, and 39 in the PC-2×2 test.
To allow a comparison of the obtained results, the same video was used in all experiments. This was a rendered animation composed of two characteristic scenes (Fig. 2). For both scenes, the encoded with a certain bitrate was displayed at least once in any quarter of the screen

Data analysis
In PC-2×2, the best part was always represented by a quarter encoded with 6 Mbps and the worst by one encoded with 3 Mbps. Therefore the difficulty of recognizing the quality of the image fragments in each set was constant. In the PC and PC-1×2 methods, the difficulty level varied for each set depending on the quality levels of the selected pair of images. Therefore, the results of the PC and PC-1×2 methods cannot be directly related to the results of PC-2×2.
As expected, in the case of both the PC and PC-1×2 methods, the least votes were assigned to sequences encoded with the lowest bitrate (3 Mbps) and the most to the videos encoded with the highest bitrate (6 Mbps). The number of votes cast increases evenly with the increase of the bitrate, which means that in most cases observers were able to recognize and indicate an image of better quality. The increase in the number of votes is greater for PC-1×2, where two stimuli were presented simultaneously. The video with the lowest bitrate was chosen as the better one in only 2.6% of presentations in PC-1×2 and in only 4.9% of cases in the PC method. Assuming that choosing a video with a higher bitrate is the correct answer, the percentage of correct votes was calculated and is presented in Table 1 and Table 2.
The correctness of PC-1×2 votes is higher than that achieved with the classical PC-  image was symmetrical to the centre of the screen to ensure the same probability of image distortion in each part. Due to the technical limitations of the laboratory equipment, a standard progressive MPEG-2 MP@HL (Main Profile @ High Level) with 1920 × 1080 px, 24 fps, and constant bitrate had to be chosen, but the concept is valid for any resolution and compression algorithm.
The content was designed especially to obtain compression artefacts that are typical for MPEG-2: blocking, blurring, ringing, staircase, basis pattern artefacts, and mosquito noise [8].
Variation of the quality of the test sequences was achieved only by changing the bitrate in the coding process. Bitrate values were selected experimentally in pre-tests to provide clearly perceptible differences in quality. Four bitrate levels (n = 4) were chosen: 3, 4, 5, and 6 Mbps. Observers were not shown the source (uncompressed) video and were not informed about the compression parameters used.
In the case of PC and PC-1×2, in accordance with recommendations [3], all n(n -1) = 12 possible combinations were tested (Fig. 3). Because both sequences were displayed in parallel on the same screen in PC-1×2, the test time was halved in comparison with PC. In PC-2×2, the number of sequences evaluated simultaneously increased to four, therefore an observer would have to watch the same test sequence (with different sets of bitrates in individual quarters) 24 times. To lessen the observers' fatigue and to shorten the duration of the experiment, 10 variations were chosen randomly (Fig. 4). In this test material, each sub-image The results from the PC-2×2 test are displayed in Fig. 6, which shows a general summary, for the whole study, of the total number of votes cast for specific bit streams considered to be the best (green) or the worst (red) quarters of the image.
As the observer was watching all four bitrate representations at the same time, all votes cast for 6 Mbps as the best part and all votes cast for 3 Mbps as the worst may be indexed as correct. Therefore the percentage of votes cast correctly is very high: 80.51% in the case of the best quality and 87.95% in case of the worst. Next, in order of quality, 5 Mbps was denoted as the best by 18.46% of votes, while 4 Mbps was denoted as the worst by 11.28% of votes. A the same time there were two votes (0.51%) by observers indicating that the 4 Mbps and 3 Mbps streams were the best and three votes(0.77%) indicating that 5 Mbps streams were the worst. No observer voted for 6 Mbps as the worst. In general, it was easier for observers to identify quarters with the worst quality than those with the best. This shows, among other things, that humans focus primarily on visual distortions caused by excessive compression of the material. The better the image quality is, the more difficult it becomes to assess.
The other research question was whether the correctness of the vote depends on the quarter in which the material is displayed. Figure 7 shows the number of correct indications for each quarter of the screen, taking into account indications of both the best and the worst parts.
The correctness obtained in PC-2×2 was high for each quarter. The lowest observed correctness level was 75.21%, in this case for the best quarter in the lower right quarter. The highest observed accuracy level was 92.31% for the worst part in the upper right quarter. In all quarters of the screen there was a consistent percentage of correct indications with an average value of 80.77% for the best quarter and 88.04% for the worst (average calculated for three quarters of the screen, as there was no 3 Mbps sequence in the lower left corner). None of the image quadrants were privileged by observers. The number of correct votes cast for the worst part was slightly higher than the number of correct votes cast for the best. This also shows once again that it is easier for observers to judge a decrease in image quality than an improvement. It may be assumed that there is no problem focusing attention

Conclusion
Our idea of test material presentation made it possible to maintain more natural viewing conditions and to use a cheaper and more flexible experimental station. In the PC-1×2 experiment, limiting the image quality information to half of the full screen did not disturb the image quality assessment, but the duration of the test was halved. Simultaneous presentation of the two stimuli resulted in increases in the number of scores cast correctly. What is more, the human audience was able to recognize and correctly evaluate four quarters coded with different bitrates displayed in parallel. Obtaining such promi-