{'fraction_non_alphanumeric': 0.03278433809819764, 'fraction_numerical': 0.009633312616532008, 'mean_word_length': 5.319587628865979, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Comprehensive evaluation of no-reference image quality assessment algorithms on KADID-10k database

The main goal of objective image quality assessment is to devise computational, mathematical models which are able to predict perceptual image quality consistently with subjective evaluations. The evaluation of objective image quality assessment algorithms is based on experiments conducted on publicly available benchmark databases. In this study, our goal is to give a comprehensive evaluation about no-reference image quality assessment algorithms, whose original source codes are available online, using the recently published KADID-10k database which is one of the largest available benchmark databases. Specifically, average PLCC, SROCC, and KROCC are reported which were measured over 100 random train-test splits. Furthermore, the database was divided into a train (appx. 80\% of images) and a test set (appx. 20% of images) with respect to the reference images. So no semantic content overlap was between these two sets. Our evaluation results may be helpful to obtain a clear understanding about the status of state-of-the-art no-reference image quality assessment methods.


Introduction
There are three main categories of objective image quality assessment (IQA) algorithms. Namely, IQA problems can be classified based on the availability of the reference, pristine images. In full-reference image quality assessment (FR-IQA), the reference, pristine (distortion-free) image and the distorted image are given to estimate the perceptual quality of the distorted image. In contrast, no-reference image quality assessment (NR-IQA) algorithms solely rely on the distorted images. Similarly, in reduced-reference image quality assessment (RR-IQA) the reference image is not available, but partial information about it is known.
The evaluation of objective IQA algorithms is based on experiments carried out on publicly available image quality databases which consist of images labelled with quality scores. These databases can be divided into two categories based on the distortion types. The first one contains dozens of pristine (distortion free), reference images and the distorted images are derived from the reference images using different levels of artificial distortions and different types of artificial distortions, such as Gaussian blurring, JPEG compression, JPEG2000 compression, color diffusion, etc. The second one consists of authentically distorted images captured by various imaging devices, such as mobile camera. As a consequence, the images are afflicted by a highly complex mixture of multiple distortions. Table 1 summarizes the main characteristics of major publicly available image quality databases.

Structure
The rest of this study is organized as follows. After this short introduction, Section 2 gives a brief overview about NR-IQA methods with a special attention to those algorithms which are evaluated in this study on KADID-10k [13] database. Section demonstrates experimental results and analysis. Finally, a conclusion is drawn in Section 4.

Methods
As already mentioned, the aim of NR-IQA is to estimate the perceptual quality of a given image without any information about the pristine (distortion free), reference image. Due to the lack of any knowledge about the reference medium, NR-IQA is considered more challenging than FR-IQA or RR-IQA. Although the reference image is not available in NR-IQA, assumptions can be made about the distortion types found in an image, such as JPEG2000 compression noise. Distortion-specific methods assume one specific distortion type in the image, while general-purpose algorithms work over various types of distortions. Furthermore, general-purpose can be divided into natural scene statistics (NSS) based, learning based, and human visual system (HVS) based groups. Another classification of NR-IQA methods divides existing methods into opinion-aware and opinion-unaware classes. Opinion-aware methods utilize subject scores during the training process, while opinion-unaware methods derive features from the pristine, reference images of the database and perceptual quality of distorted images is quantified as the deviation from the pristine images' features. In the followings, the examined methods are briefly discussed.
DIIVINE [14] is a two-stage framework which involves distortion identification and distortion-specific quality assessment. It is based on NSS. Namely, a set of neighboring wavelet coefficients were modelled by a Gaussian scale mixture model. Moreover, steerable pyramid decomposition was used to extract statistics from the distorted images. [15] derives NSS features by discrete cosine transform coefficients modeling using generalized Gaussian distribution. The parameters of the generalized Gaussian distribution were applied as quality-aware features.

BLIINDS-II
BRISQUE [16] applies scene statistics of locally normalized luminance coefficients to train a support vector regressor (SVR) for perceptual quality prediction.
NIQE [17] measures the distance between the natural scene statistics (NSS) based features calculated from the pristine images to the features extracted from the input image. The features are modeled as multi-dimensional Gaussian distributions.
CurveletQA [18] extracted statistical features (the coordinates of the maxima of the log-histograms of the curvelet coefficients, the energy distributions of both orientation and scale in the curvelet domain) from the image's curvelet representation. Moreover, image distortion and quality prediction stages are trained using a support vector machine (SVM).
SSEQ [19] contains an image distortion and quality prediction engine. Furthermore, it extracts a 12-dimensional local entropy feature vector.
GRAD-LOG-CP [20] utilizes the joint statistics of gradient magnitude map and the Laplacian of Gaussian features to train a support vector regressor (SVR) for perceptual image quality prediction. PIQE [21] is an opinion-unaware method and calculates perceptual quality of an image through block-wise distortion estimation. First, the mean subtracted contrast normalized (MSCN) coefficients are determined for each pixel in the input image. Second, the input image is divided into 16 × 16 blocks and high spatially active blocks are identified relying on the MSCN coefficients. In each block, distortion is evaluated due to blocking artifacts and noise relying on the MSCN coefficients. A threshold criteria is also applied to classify blocks as distorted (blocking artifacts, Gaussian noise) blocks. The quality score is computed as the mean of scores in the distorted blocks. [22] is an opinion-unaware method. It integrated natural image statistics features from multiple sources, such as normalized luminance, mean subtracted and contrast normalized coefficients, gradient statistics, statistics of log-Gabor filter responses, and statistics of colors. Subsequently, a multivariate Gaussian model is learned from pristine image patches. Perceptual quality is quantified by measuring the deviation from the learned distribution using a Bhattacharyya-like distance.

IL-NIQE
BMPRI [23] introduced the concept of multiple pseudo reference images (MPRI). Specifically, the distorted images were further degraded. Subsequently, similarities between the distorted image and the MPRIs were measured. To this end, a traditional FR-IQA metric was applied. Specifically, local binary pattern features were computed to describe the similarities between the distorted image and the MPRIs. Finally, the similarity scores were aggregated to obtain the input image's perceptual quality.
SPF-IQA [24] extracted different statistical (fractal dimension distribution, first digit distribution in gradient magnitude domain, first digit distribution in wavelet domain, color statistics) and perceptual features (colorfulness, global contrast factor, dark channel feature, entropy, mean of phase congruency) from the input image and fused them together. Finally, the fused feature vector is mapped onto perceptual quality scores with the help of Gaussian process regression (GPR) using rational quadratic kernel function.
SCORER [25] proposed a set of derivative kernels which were utilized to filter the Y, Cb, and Cr channels of the input image. As a result, a set of filtered images was obtained for further processing. Subsequently, N interest points were detected on each filtered image relying on features from accelerated segment test (FAST) [28]. Specifically, each interest point is used to describe a 3 × 3 block around the interest point by taking all values from the block. The extracted 2400-dimensional feature vectors are mapped onto perceptual quality scores with the help of a trained support vector regressor (SVR).
ENIQA [26] extracts features in two different domains. Namely, mutual information between color channels and the two-dimensional entropy is determined first. Subsequently, two-dimensional entropy and the mutual information of the filtered sub-band images are determined. Based on the extracted features, a support vector machine (SVM) and a support vector regressor (SVR) is trained for distortion and quality prediction, respectively.
In MultiGAP [27], an input image is run through an Inception-V3 pretrained convolutional neural network which carries out all its defined operations. Moreover, global average pooling layers are attached to each Inception module to extract image resolution independent features. Subsequently, the features of the Inception modules are concatenated and mapped onto perceptual quality scores with the help of an support vector regressor (SVR) with Gaussian kernel function. In this study, results obtained by Gaussain process regression (GPR) head with rational quadratic function is also presented.

Experimental results and analysis
In this section, we report on the experimental results obtained on KADID-10k [13] database. The rest of this section is organized as follows. In Subsection 3.1, the evaluation metrics are defined. Subsection 3.2 describes the experimental setup. In Subsection 3.3, the performance of the examined NR-IQA algorithms is reported on the entire KADID-10k [13]. Subsection 3.4 reports the performance on individual distortion levels, while Subsection 3.5 presents the experimental results with respect to the distortion types.

Evaluation metrics
As already mentioned, the evaluation of IQA algorithms is based on the correlation between the predicted and groundtruth scores measured on an image quality database. In the literature, there are three major evaluation: Pearson's linear correlation coefficient (PLCC), Spearman's rank order correlation coefficient (SROCC), and Kendall's rank order correlation coefficient (KROCC). The latter two measure the prediction monotonicity of an IQA method, because they operate merely on the rank of the data points and ignore the relative distance between data points. MATLAB provides functions for the computation of these performance metrics. The PLCC between two data sets, A and B, is defined as (2) where the rank(·) function gives back a vector whose ith element is the rank of the ith element in the input vector. KROCC is defined as where n is the length of A and B, n c denotes the number of concordant pairs between A and B, and n d is the number of discordant pairs.

Experimental setup
For learning-based methods, the distorted images of KADID-10k [13] were divided into a training (appx. 80% of images) and a test set (appx. 20% of images) with respect to the reference images. As a consequence, there was no semantic overlap between these two sets. Moreover, average PLCC, SROCC, and KROCC were measured over 100 random train-test splits. In contrast, opinion-unaware methods were trained on the reference images and tested on the distorted images. Furthermore, PLCC, SROCC, and KROCC are reported.

Overall performance
The performance of the examined algorithms in terms of average PLCC, SROCC, and KROCC (100 random train-test splits) is summarized in Table 3. From these results, it can be observed that SCORER [25]

Performance over different distortion levels
In this subsection, the performance of the examined NR-IQA algorithms is given over different distortion levels. Specifically, Tables 4, 5, and 6 summarize the average PLCC, SROCC, and KROCC values over the different distortion levels of KADID-10k database [13].

Performance over different distortion types
In this subsection, the performance of the examined NR-IQA algorithms is given over different distortion types. Specifically, Tables 7, 8, and 9 summarize the average PLCC, SROCC, and KROCC values over the different distortion types of KADID-10k database [13]. As already mentioned, Table 2 summarizes the different distortions types found in KADID-10k [13] and their numeric codes.