{'fraction_non_alphanumeric': 0.024468610973801284, 'fraction_numerical': 0.009886307464162136, 'mean_word_length': 4.855764592378196, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Impact of applying super resolution to low resolution face images on the performance of deep neural networks

Face images captured in unconstrained environment differ in various aspects such as expression, illumination, resolution, occlusion, pose etc. which makes face recognition task difficult. The face images captured by the camera from a distance will have low resolution and lack many finer details that makes face recognition a challenging task. Super resolution (SR) is a process of generating high resolution (HR) images from one or more images. In this work, we apply super resolution to low resolution (LR) images of faces to find the impact on the deep models performance. To achieve this, we create dataset with face images captured in unconstrained environment. Later we designed a CNN model with eight layers and trained on the dataset created. Our deep model with low memory requirement and less parameters achieves an accuracy of 99.75% on test dataset and outperforms fine-tuned VGGFace by a small margin. The performance of our deep neural network and fine-tuned VGGFace was observed on low resolution images pre and post-super resolution. The deep neural network-based model available in OpenCV, SRGAN super resolution model and INTER_CUBIC interpolation are used to generate HR images. The HR images generated by OpenCV, SRGAN are better than INTER_CUBIC interpolation. The results show that HR images generated by applying SR to low resolution face images improve the image quality in terms of Mean squared error (MSE), Structural similarity index measure (SSIM) and Peak to signal noise ratio (PSNR). However, the results indicate that improvement in the image quality does not significantly improve performance of deep model.


Introduction
In the field of computer vision, Face recognition has grabbed the attention of researchers as it finds many applications in law enforcement, biometrics, law enforcement and so forth [1]. Face recognition in unconstrained environment is still a challenging task as the face images vary in terms of pose, illumination, resolution, occlusion etc. [2][3][4]. Most early approaches in face recognition extract the features like SIFT [5], LBP [6], PCA [7], HOG [8] from the face images and train the classifier to recognize the different classes. But the approach to perform face recognition has changed after AlexNet a deep neural network won the ImageNet competition by a large margin [9]. These deep convolutional networks have multiple layers which extract the features and learn representations that have different levels of abstraction. The face images captured with from surveillance camera has variations in terms of pose, resolution, expression etc. Generally, camera is placed to cover wide angle, so the face images captured at longer distance will have low resolution and lacks many finer details. Figure 1 shows the face image captured by surveillance camera with wide viewing angle has lot of image details missing. Face recognition with low resolution images is an active research area as demand for surveillance camera-based applications increased. Generally, to solve face recognition of lowresolution face images they construct high resolution faces by applying super resolution. Threshold resolution or minimal resolution above which the face recognition performance is steady, but if resolution decreases below then that the performance degrades [10]. The quality of an image can be assessed in two different ways i.e. subjective or objective. Subjective methods depend on the perception of the human whereas objective methods are related to computational models. Objective image quality measurement models are classified into three types based on the information available about the reference image. [11] Full-reference methods -They compare test image with ground truth or reference image. Reduced reference methods-Test image quality is assessed by comparing the features extracted from it with that of the reference image.
No-reference methods-The quality of test image is assessed without any reference image. Image quality techniques such as Mean Square Error (MSE), Peak signal to noise ratio (PSNR) and Structured similarity index method (SSIM) are generally used to assess the quality of the image. [12] MSE (Mean Squared Error)-MSE also referred as Mean squared deviation (MSD) gives the average of squared difference between the ground truth and the estimated value. The higher the MSE value indicates that the image quality is degraded. PSNR (Peak signal to noise ratio) -It is defined as ratio between the maximum power of the signal to the power of the noise that impacts the quality. It is expressed in decibel (DB) form for comparison between two images. The higher the PSNR value indicates the image quality is good. SSIM (Structured similarity index method)-It measures the similarity between the reconstructed image and the actual image. The SSIM value lies between 0 to 1, with 1 indicating high similarity (same as the original) and 0 low similarities.
The process of obtaining single or multiple HR images from one or more low resolution (LR) images is referred as super resolution (SR) [13]. To study the pre-and post-of applying of super resolution to the low-resolution face images impact the performance of deep neural network, the paper discusses about the dataset created with variations in pose, illumination, resolution, occlusion etc. in section 3. In section 4 the performance of deep neural network designed with eight convolutional layers followed by fully connected neural network and VGGFace fine-tuned model on the dataset created is observed. We study the impact of low-resolution images, pre and post of applying super resolution on the deep neural network designed and fine-tuned VGGFace in section 5.

Related work
Broadly super resolution methods are catagorized into (i) Multi-image super resolution (ii) Example-Based super resolution [14]. Many applications like aerial and satellite imaging, processing of medical images, ultra sound imaging [15], improving facial and text images, enhancing video and compressed images, iris recognition make use of super resolution [16]. To obtain high resolution images they  3 increase the pixels in each unit area of the image to get finer details [17]. There is difference between SR, Interpolation and Image restoration. Interpolation does not store high frequency details in contrast to SR which does and it is generally applied to single image [18]. In image restoration the quality of the image is improved but the size of the input and output are same whereas in SR the output image size increases. SR algorithms can be classified based on the factors like how many LR images used, domain and reconstruction method. Based on the domain they are classified into frequency or spatial domain. In the same way they are classified into single image or multiple images based on the number of LR images used. The assessments of SR algorithms are carried out using both the methods, subjective and objective. SR algorithms are also applied to facial images which have the advantage due to standard and established dataset as compared to applications like processing the medical images, satellite imaging etc. They found that CAS-PEAL database provides the ground truth of resolutions which is expected for SR [19]. To improve the performance Face recognition with very low resolution they designed a data constraint for super resolution [20].

Dataset
Dataset with face images of 80 subjects is created in unconstrained environment with half of the subjects images downloaded from internet and the other half subjects captured from the camera placed at a fixed location in the college campus. Deep neural network available in OpenCV is used for face detection. Figure 2 show face images of a particular subject captured by the camera with variations in resolutions. We split the dataset in the ratio of 70 : 15 : 15 (training set : validation set : testing set).
Data augmentation was applied to increase the face images of subject which were less in number by performing simple transformations like flip, scale and noise as shown in Figure 3.

Deep neural network model
A deep model with eight layers of convolution was designed, which is followed by a fully connected layer. The model accepts input image of size 224x224 pixels and uses activation function Rectified linear unit. The Figure 4 shows architecture of deep neural network. VGGFace deep neural network trained on huge dataset consisting of 2.6M face images with 2622 different subjects [21]. Instead of training VGGFace from scratch, we fine tune the model by freezing the top layers which learn the generic features. Figure 5 shows the fine-tuned VGGFace where the top layers weights are freezed. Table 1 shows the deep neural network performs marginally better than the fine-tuned VGGFace on the dataset created with the advantage of having less parameter with low memory space requirement. Our deep neural network achieved 99.94% training accuracy and 99.75% testing accuracy, whereas finetuned VGGFace achieved 99.82% training accuracy and 99.68% testing accuracy on our dataset. Our model has around 1/8 th of the VGGFace parameters and requires less memory requirement.             Table 2 show that the PSNR value of the HR images generated by DNN in OpenCV outperforms INTER_CUBIC and SRGAN based SR.
The Figure 17 (a) shows the low-resolution face image, which is not recognized by our Model but after generating the HR image using SRGAN based SR as shown in figure 17 (b) the model was able to correctly classify. By using the same image shown in figure 17 (a) when HR image is generated using DNN in Opencv as shown in figure 17 (c) it was not correctly classified.
The figure 18 (a) shows the low-resolution face image the which is not recognized by our Model but after generating the HR image using SRGAN based SR and DNN available in OpenCV as shown in figure 18 (b) and 18 (c) they were correctly classified. The figure 19 (a) shows the low-resolution face image the which is recognized by fine-tuned VGGFace but after generating the HR image using SRGAN based SR and DNN available in OpenCV as shown in figure 19 (b) and 19 (c) they were mis classified by it.    As shown in the Table 3 when 200 low resolution images (few images from IMFDB and few from our data set) were given as it is our model could classify 168 correctly and Fine-tuned VGGFace classified 174 correctly. After converting to HR using INTER_CUBIC the performance slightly reduced. There were cases where low resolution image which was correctly classified by the model was later misclassified when we applied super resolution.

Conclusion
A deep neural network with eight CNN layers has been developed and trained on the dataset created with face images captured in unconstrained environment with variations in pose, resolution, occlusion etc. Our CNN model with less numbers of parameters performs marginally better than fine-tuned VGGFace on our dataset. As Low-resolution face images captured lack many details which pose challenge for face recognition, so they were converted to high resolution images using super resolution and pre & post analysis has been done. We generated HR images using SRGAN and Deep neural network available in Opencv. Generating HR images improves the image quality in terms of MSE, SSIM and PSNR but the performance on recognition rate cannot be assured. As in a few cases LR images which are not correctly recognized by the Deep learning models after converting them to HR images using the SR algorithms, they were correctly classified, at the same time we observed a few lowresolution images which are correctly classified by the Deep models were later mis classified by the Deep models, when those low-resolution images were converted to HR using the SR algorithms used. We found only marginal improvement in the performance after enhancing the low-resolution models using super resolution model. However, we have to explore other super resolution methods in the future to come to conclusion.