{'fraction_non_alphanumeric': 0.03435326842837274, 'fraction_numerical': 0.01808066759388039, 'mean_word_length': 4.885675306957708, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ComplexFace: a Multi-Representation Approach for Image Classification with Small Dataset

State-of-the-art face recognition algorithms are able to achieve good performance when sufficient training images are provided. Unfortunately, the number of facial images is limited in some real face recognition applications. In this paper, we propose ComplexFace, a novel and effective algorithm for face recognition with limited samples using complex number based data augmentation. The algorithm first generates new representations from original samples and then fuse both into complex numbers, which avoids the difficulty of weight setting in other fusion approaches. A test sample can then be expressed by the linear combination of all the training samples, which mapped the sample to the new representation space for classification by the kernel function. The collaborative representation based classifier is then built to make predictions. Extensive experiments on the Georgia Tech (GT) face database and the ORL face database show that our algorithm significantly outperforms existing methods: the average errors of previous approaches ranging from 31.66% to 41.75% are reduced to 14.54% over the GT database; the average errors of previous approaches ranging from 5.21% to 10.99% are reduced to 1.67% over the ORL database. In other words, our algorithm has decreased the average errors by up to 84.80% on the ORL database.


I. INTRODUCTION
Face recognition is one of the most active research topics in computer vision [1,2]. During the past several years, a large number of algorithms have been proposed for face recognition [3,4], for which extraction of effective features is most critical. Traditional feature transformations such as principal component analysis (PCA) [5,6], Fisher discriminant analysis (FDA) [7] and Locality Preserving Projection (LPP) [8], are widely used for feature manipulation in face recognition. Later, those features are replaced by local features such as Local Binary Patterns (LBP) [9,10], Local Phase Quantisation (LPQ) [11], as well as Scale-invariant feature transform (SIFT) [12]. For datasets with simple background, traditional features can achieve very competitive results. Unfortunately, for uncontrolled environment, the face recognition performance with such traditional features may degrade dramatically. It is challenging to improve the performance of face recognition because facial images are affected by variation in illumination, facial expression and poses.
In recent years, Convolutional Neural Networks (CNNs) have achieved great success in computer vision such as objection detection, objection recognition, image segmentation, action recognition, and especially face recognition [13][14][15]. There are two main characteristics that make CNNs achieve such outstanding success. First, parallel GPUs greatly improve the training efficiency of CNNs. Second, a lot of labelled training data are available to learn rich features by the training algorithm of CNNs, which can automatically learn the features of facial images by tuning the large number of weights. However, CNN requires large-scale training datasets with labels. This means that highly efficient hardware and large-scale annotated data are the key to obtain the high performance of CNNs. This is, however, not available in applications in which there is no GPU for the target hardware or the number of training samples are limited, which leads to overfitting for CNNs. In such cases, non-CNN methods are needed to achieve competitive performance of face recognition. Among these approaches, generating new representations from original images and fusing multiple representations is a simple and effective method to improve the robustness of face recognition.
Data augmentation via generating new presentations from original images with mathematical functions can enlarge the training datasets and can achieve higher classification accuracy. These approaches can mainly be categorized into two categories. In the first type of approaches, new representations are obtained via the face structure, including symmetrical characteristic of faces and the mirror image of face [16][17][18][19]. In the second category, new representations are constructed by the perturbation and distribution of original samples [20][21][22]. Different from these two categories of approaches, Xu et al. [23] developed a novel method to automatically produce approximately axissymmetrical virtual face images. Zhang et al. [24] proposed a method to operate small regions of the original image to obtain new representations. Moreover, singular value decomposition has also been applied to generate new representations in [25,26]. Both sparse representation based classification [27,28] and collaborative representation based classification (CRC) [29] have been used with an assumption that a test sample is represented as a linear combination of an entire set of training samples and plays an important role in face recognition [23] [30][31][32]. They can achieve good performance if there are sufficient training samples for a given class object. Based on this fact, we design our method to obtain new representation from original sample to enlarge the face dataset.
Data sample fusion can offer more supplementary information from multiple samples or information sources and obtain better performance in image classification, which has been shown in many existing methods [18, 19, 24-26, 33, 34]. In these methods, it is a good and effective way that new representations from original samples are generated via related arithmetic operators and then new representations and original images are fused by appropriate weights. Traditional fusion methods have a drawback that the weights for multiple samples need to be manually set via empirical knowledge [24,25,35]. To address this problem, we exploit a method to transform the original image and the new representation into a complex number image. This scheme can be regarded as two kinds of images are fused or two features are fused without weight setting. Thus, our method avoids the parameter setting while more complementary information can be available for face recognition.
In addition to complex numbers have been used to represent samples, quaternions are also exploited in pattern classification [36]. A quaternion is composed of four real numbers, so some mathematical operations of a quaternion can be decomposed as operations of two complex numbers. As higher-order extensions of real numbers, complex numbers and quaternions not only allow more information to be integrated into a denotation, but also can exploit crossbred information of original numbers. In this sense, the corresponding method can be viewed as a feature level fusion method.
In existing methods, multi-sample fusion only works in the 2D space so that the supplementary information obtained by multiple samples is limited. If the fusion is in the 3D space, more supplementary information will be available. Motivated by the fact that kernel based methods [37] can be applied to represent and classify test samples by using all the training samples in the feature space, we present a novel method to transform different complex samples including test complex images and training complex images into a new space. This process can be viewed as fusion of two kinds of samples in the 3D space. We apply collaborative representation based classification [29]to conduct the face recognition experiments.  In this paper, we proposed a novel algorithm, namely, ComplexFace, for face recognition with small datasets, as shown in Fig. 1. Our algorithm uses the data augmentation strategy to generate new representations from original images, and then fuse these two sets of images by applying the complex numbers approach--the original sample as the real part of a complex number and the new representation as the imaginary part of the complex number, which allows to combine the pair-wise images for complementary information. This avoids parameter-setting problem typical for other fusing approaches, which makes it convenient in real-world applications. We assume that the test sample can be expressed by a linear combination of all the training samples, then we apply kernel function to complex numbers images and transform them into a new space. This scheme enables two kinds of samples or features to be fused in a 3D space, thus, more complementary information can be available for more accurate face recognition. Collaborative representation based classification CRC is then built in the new space. Extensive experiments on the GT dataset and the ORL dataset show that the proposed approach outperforms other facial recognition methods, including collaborative representation based classification (CRC) [29], an improvement to the nearest neighbor classifier (INNC) [38], ADWF [26] , KRBM [39] , SVD fusion [25] and method in [24]. For further performance comparisons, we conducted our experiments in comparison to derived images [20] and symmetrical faces [18]. All the experimental results show that our method achieved impressive accuracy rates for face recognition with limited samples.
The remainder of this paper is organized as follows. Section 2 discusses related work. Section 3 presents the proposed approach in detail. Section 4 discusses the experimental results. Section 5 concludes the paper.

II. RELATED WORK
Data augmentation by generating new presentations from original samples has been widely used to improve classification accuracy. Passalis et al. [16] proposed a novel face recognition method to handle pose variations by using facial symmetry. Mirror images or symmetrical faces of original faces are applied as new presentations in Xu et al. [17,18,40], which achieves favourable face recognition performance. Liu et al. [19] proposed a framework that fused virtual mirror synthesized training samples as bases and hierarchical multi-scale local binary patterns (LBP) features for face classification. Zhang et al. [41] exploited a new face recognition method based on SVD perturbation for single-example image per person. Shan et al. [22] proposed a sample expansion method to generate virtual samples. Xu et al. [23] developed a novel method to automatically produce approximately axis-symmetrical virtual face images. Zhang et al. [24] proposed a novel method to represent an image by first operating the nonoverlapping blocks with certain arithmetic, and then merging modified blocks to obtain new representations of original images. Singular value decomposition is applied to generate new representations in [25,26].
Several sample fusion methods have been proposed to improve face recognition performance. Xu et al. [18] integrated both original images and symmetrical faces for face recognition. This method may result in ugly virtual samples due to the use of the symmetrical faces'. Liu et al. [19] fused hierarchical multi-scale local binary patterns and virtual mirror samples to perform face recognition. Zhang et al [26] proposed a simple and effective method that automatically fuses the original samples and two groups of virtual samples obtained by singular value decomposition. Werghi et al. [42] presented a novel approach to fuse shape and texture local binary patterns (LBPs) on a mesh for 3D face recognition. Ding et al. [43] proposed a new scheme to extract "Multi-Directional Multi-Level Dual-Cross Patterns" (MDML-DCPs) from face images, which outperforms the state-of-the-art local descriptors for both face identification and face verification tasks.
Deep learning has also been exploited to address face recognition task. Deepface [44] took the last hidden layer neuron activations of deep convolutional networks to obtain the features. This study used a common method that augments the dataset via obtaining new samples in a multi-scale and multichannel way. Following DeepID2 [45], DeepID2+ [46], DeepID3 [47] and method in [48,49] have been proposed to further improve the performance for face recognition. Schroff et al. proposed a method named FaceNet [50] that applied a deep convolutional network trained to directly optimize the embedding itself, which reduced the error rate in comparison to DeepId2+.

III. THE PROPOSED METHOD
In this section, we describe the different components of our ComplexFace method as shown in Fig.1 in detail. We firstly input facial images. We then discuss the method for generating new representations from original samples. Following we transform both the original samples and the new representations into complex numbers. Afterwards we show how to transform test complex samples and training complex samples into a new space. In the end, the collaborative representation based classification is used to conduct our experiments.

A. Image Transformation
There are s classes among n original training samples represented as x represents the j -th sample of the k -th class. We define 11 11 ... ,..., ...   Fig. 2 shows some of original images and corresponding new representations from the ORL dataset.

B. Integrating the Original Images and Their New Representations
where  is obtained by Eq. (10) or Eq. (11) . In other words,  is one of the  and  .
In short, the main steps of our algorithm are described as follows: Step 1. For the set of original training sample set X and test sample set Y , we apply Eqs. (1) and (2)  Step 3. We apply kernel functions to transform the samples into new space by using Eqs. (5), (6), (7) and (8).
Step 4. Collaborative representation based classification is applied to training samples via Eqs. (9) and (11). Then we calculate the residual error for each training sample by Eq. (12). Finally we classify the test sample z into the cls -th class by using arg min

IV. EXPERIMENTAL RESULTS
We perform an extensive evaluation of our method on the most widely used and representative benchmark face datasets: the Georgia Tech (GT) face database and the ORL face database. All the experiments demonstrated that the performance of our method is significantly better than that of the compared methods.

A. Datasets
The ORL face database [51] consists of 40 subjects in total. For each subject, there are 10 images with different ages, genders and nations. Some of the images in ORL have various properties such as pose, expression and decorative face (e.g. glasses or no glasses). At the same time, there is 20% scale change in those images. All the images were acquired against a dark homogeneous background with the subjects in an upright and frontal position. Each image was resized to 46×56 pixels in our experiments. Fig. 3 shows some of facial images in ORL dataset.

Fig. 3. Sample facial images in ORL dataset
The Georgia Tech (GT) face database is composed of 50 persons, each person has 15 colourful images with clutter background. GT is provided by the Georgia Institute of Technology and is freely available to everyone [52]. The size of the images in GT is 640*480 pixels, we used down-sampled face images of size 30×40 pixels in our experiments. The details for obtaining these images are presented in [17]. Those images have different expressions, various illuminations and diverse scales while including two angles: frontal face and side face. Fig. 4 shows some of facial images in GT dataset.

B. Performance Analysis for the Proposed Method
In this section, we describe the extensive experimental results of our method: ComplexFace. In all our experiments, we apply the kernel function   to transform the samples into a new space. The parameter  is set to 0.01. We define  as 10 10 according to the analytical results discussed in detail in Section 4.3. To illustrate the outstanding performance of our algorithm, we compare ComplexFace with the standard collaborative representation based classification (CRC) [29], an improved nearest neighbour classifier (INNC) [38], ADWF [26], KRBM [39], and the method in [24]. All the experiments are described below and ComplexFace shows significant performance improvement over all existing methods.

(1) Experiments on the ORL dataset
For each class on the ORL database, we treated first 6 facial images as the training samples and the remain 4 facial images as the test samples. To illustrate how the number of training samples to affect the performance, we also evaluated our method by using the first 7, 8 or 9 facial images as the training samples and the remaining images (3, 2 or 1) as the test samples respectively. Table I shows all the experimental results. Just as shown for the GT dataset, our method also greatly outperforms all the other methods over the ORL database. Our method reduces the average classification error rates of with CRC (8.70%) and INNC (8.33%) to 1.67%, which is more than 80% drop in average error rate. This means that ComplexFace algorithm is very useful in improving the accuracy of face classification. It is worth mentioning that when the number of training samples is 8 or 9, both classification error rates are 0, which is an outstanding result.

(2) Experiments on the GT dataset
On the GT face database, the first 9, 10, 11, 12 or 13 face images of each class is treated as training samples and the remaining images is regarded as test samples. All the comparable experimental results are shown as Table II, which show that our method greatly outperforms all the other methods. The average error rate (our method is 14.62%) was 18.88% lower compared to that of CRC (33.50%). What's more, the average error rate of our method is 14.62%, which is 27.13% lower compared to that of KRBM (41.75%). Both results demonstrated that ComplexFace algorithm has achieved significant improvement over all existing methods compared here.

C. Extended Experiments for Our Method
To further understand the performance of ComplexFace algorithm, we conducted additional performance comparisons and parameter tuning experiments as shown in this section.

(1) Comparison with other fusion methods
One of the most important contributions of our method is the use of complex numbers to fuse the new representation and the original image to provide more information for face recognition. To analyse the performance of our fusing method, we conducted experiments to compare with the methods in [18] and [41]. The former method [18] exploited the symmetry of the face to generate new samples. We integrate the symmetrical face obtained by this work and the original image to perform experiments. The latter method [41] was proposed by Zhang et al. In comparative experiments, we set n as 5/4 and =0. 25  with more details in [41]. The experimental results shown in Table III illustrate that our method achieves outstanding results compared to the other methods. It is worth mentioning that we maximally decrease the average errors by 84.80% on the ORL database (The average error by using Derived images + original images is 10.99%, and our method's average error is only 1.67%). That is a very encouraging result.  (2) Evaluation for the value of coefficients There are two solutions for eq. (9), one is And the other is For further evaluation, we test our method using different parameters for eq. (10) and eq. (11) respectively. Table IV shows the classification errors on the GT face database. In this table, for example, 7 , 10  represents that we use eq. (10) to obtain parameter  and use the parameter 7 10   in experiments. In the similar way, 7 , 10  represents that we use eq. (11) to obtain parameter  and use the parameter 7 10   in experiments. If the  is fixed, the classification error using the parameter  is a little bit better than the classification error using the parameter  (22.69% vs 22.76%, and 14.54% vs 14.62%).

TABLE IV Experimental results (classification error rates% ) on the GT face database
For the ORL dataset, similar to the GT database, we repeated our experiments via the four situations as shown in Table V. No matter what the values we set for the coefficients, we can obtain the same classification error rates. That means our method is robust with low dependency on the coefficient parameters of the algorithm.  To further evaluate parameter  , we also conducted an experiment on the GT database. The results are shown in Fig. 6, The horizontal coordinate represents the exponent for parameter  ,and the vertical coordinate represents that the error rate obtained by different training number and different value of the parameter  . Again, all the curve has the minimum error rate if the 10 =10  .

V. CONCLUSIONS
Data augmentation by fusing raw images with transformed images is helpful for improving the performance of face recognition algorithms. In this paper, we proposed ComplexFace, a novel algorithm for face recognition with limited training samples. It first generates new representations from the original images and the fuse both into complex numbers by taking the new representation as the imaginary part of the complex number and the original sample as the real part of the complex number. Then we apply kernel functions to transform complex number samples into a new representation space. Our experimental results of this collaborative representation based classification showed that our fusion approach is able to achieve significant performance improvement compared to the state-of-the-art face recognition methods for limited samples, reducing their error rates by up to 84.80% on the ORL dataset. The source code of our program will be provided upon request.