{'fraction_non_alphanumeric': 0.03834969456859033, 'fraction_numerical': 0.010089970530260154, 'mean_word_length': 4.925754596970047, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-label Learning Based Deep Transfer Neural Network for Facial Attribute Classification

Deep Neural Network (DNN) has recently achieved outstanding performance in a variety of computer vision tasks, including facial attribute classification. The great success of classifying facial attributes with DNN often relies on a massive amount of labelled data. However, in real-world applications, labelled data are only provided for some commonly used attributes (such as age, gender); whereas, unlabelled data are available for other attributes (such as attraction, hairline). To address the above problem, we propose a novel deep transfer neural network method based on multi-label learning for facial attribute classification, termed FMTNet, which consists of three sub-networks: the Face detection Network (FNet), the Multi-label learning Network (MNet) and the Transfer learning Network (TNet). Firstly, based on the Faster Region-based Convolutional Neural Network (Faster R-CNN), FNet is fine-tuned for face detection. Then, MNet is fine-tuned by FNet to predict multiple attributes with labelled data, where an effective loss weight scheme is developed to explicitly exploit the correlation between facial attributes based on attribute grouping. Finally, based on MNet, TNet is trained by taking advantage of unsupervised domain adaptation for unlabelled facial attribute classification. The three sub-networks are tightly coupled to perform effective facial attribute classification. A distinguishing characteristic of the proposed FMTNet method is that the three sub-networks (FNet, MNet and TNet) are constructed in a similar network structure. Extensive experimental results on challenging face datasets demonstrate the effectiveness of our proposed method compared with several state-of-the-art methods.


Introduction
Facial attribute classification is an important and fundamental research area in computer vision and pattern recognition. The task of facial attribute classification is to predict the attributes of a facial image, including gender, attraction, race, etc. Recently, facial attribute classification has received increasing attention with a wide range of applications, such as face verification [1,2,3], face recognition [4,5,6], face retrieval [7]. However, it remains a challenging problem, because of the large facial appearance variations caused by pose, illumination and occlusion, etc.
Early works on facial attribute classification usually characterize the facial attributes based on the histogram representation [2,3,8]. For example, Kumar et al. [2] propose to firstly extract the low-level features from different regions of a face, and then predict facial attributes with the Support Vector Machine (SVM) for face verification. Cherniavsky et al. [8] develop a generative facial feature representation method based on the Haar-like features and investigate a semi-supervised method to predict facial attributes with SVM.
Recent research mainly focuses on using the Deep Neural Network (DNN) to predict facial attributes. Luo et al. [9] combine discriminative decision trees with the deep Sum-Product Network (SPN) for facial attribute classification. In [10,11,12], the authors firstly extract facial features using DNN and then classify facial attributes with SVM. Ehrlich et al. [13] learn the shared feature representation for facial attributes by directly operating on faces and facial landmark points. Rudd et al. [14] address the problem of imbalanced data to predict multiple facial attributes.
Generally speaking, methods for facial attribute classification can be divided into two categories: single-label learning based methods [2,8,10,11,12] and multi-label learning based methods [13,14]. The single-label learning based methods predict facial attributes separately and thus do not consider the correlation between facial attributes. In contrast, the multi-label learning based methods, which attempt to predict facial attributes simultaneously by using labelled data, have drawn increasing attention. However, in real-world applications, only some commonly used attributes are provided with labelled information, while the other attributes have unlabelled data. Therefore, these methods [13,14] fail to deal with the facial attribute classification problem when unlabelled information is available (recall that these methods are based on super-vised learning).
Motivated by the above observations, we propose a novel facial attribute classification method, which performs transfer learning based on multi-label learning. More specifically, we take advantage of the transfer DNN technique to predict facial attributes that do not have labelled information in the target domain. To effectively exploit the labelled data in the source domain, we use the multi-label learning technique to predict multiple facial attributes simultaneously, considering the correlation between facial attributes. Fig. 1 shows an illustration of the correlation between different facial attributes. For different learning problems, some carefully designed networks are used, where these networks share the same structure at the former layers of the networks and they only differ at the latter layers. Therefore, the networks can be effectively trained via finetuning.
In this paper, we propose an effective deep transfer neural network method, termed FMTNet, which consists of three sub-networks for facial attribute classification. The first subnetwork is the Face detection Network (FNet) for face detection. FNet is initialized by using the model learned from a large scale ImageNet dataset [15], and then is fine-tuned by using the facial images. The second sub-network is the Multi-label learning Network (MNet) for facial attribute classification with supervised learning, where multiple attributes are predicted simultaneously. Based on FNet, MNet is fine-tuned by using labelled attributes in the source domain. The network structures at the former layers of both MNet and FNet are the same, whereas the main difference is that multiple fully-connected layers are independently constructed in MNet. The third subnetwork is the Transfer learning Network (TNet) for facial attribute classification, when labelled information is not available in the target domain. Based on MNet, TNet makes use of unsupervised domain adaptation to improve the performance of facial attribute classification.
The main contributions of this paper are summarized as follows: (1) Instead of using single-label learning for each attribute [4,7], the proposed method effectively performs facial attribute classification based on multi-label learning for the labelled attributes in the source domain. Especially, we propose an effective loss weight scheme to explicitly exploit the correlation between facial attributes based on attribute grouping, which can significantly improve the generalization performance of the proposed method.
(2) Based on multi-label learning, the proposed method leverages transfer learning to predict facial attributes for the unlabelled attributes in the target domain. The transfer neural network successfully transfers the features from the source domain (with labelled information) to the target domain (without labelled information), even when the probability distributions between the two domains are significantly different. Therefore, the proposed method alleviates the dependency on fully labelled training data, especially in the absence of labelled information for some attributes.
The remainder of the paper is organized as follows: In Section 2, some related work is discussed. In Section 3, the details of the proposed FMTNet method for facial attribute classification are described. In Section 4, the experimental results are reported. In Section 5, the conclusions are presented.

Related Work
The proposed method is closely related to deep learning, multi-label learning and transfer learning. In this section, we briefly discuss the related work.

Deep Learning
The recent great success of deep learning based facial attribute classification is triggered by a growing number of works [10,11,12,16,17] on learning compact and discriminative features. Compared with the traditional feature extraction methods (such as Scale-Invariant Features Transform (SIFT) [18] and Histogram of Oriented Gradients (HOG) [19]), the deep learning based methods [10,11,16] have shown astounding performance improvement. For example, Zhang et al. [10] combine the deep features obtained from each poselet of the face region with the deep features of the whole facial image as the final features, and then use SVM for facial attribute classification. Liu et al. [11] employ DNN for face localization and apply another neural network to extract features in the face region for attribute classification. Zhong et al. [16] extract the features of each attribute based on the hierarchical DNN, and use the linear SVM to classify facial attributes. All the above methods rely on DNN to extract features and require an additional classifier to classify facial attributes. Furthermore, they usually do not consider the correlation between facial attributes. Different from these methods, our proposed method presents an end-toend network, which effectively exploits the correlation between facial attributes, thus considerably improving the performance of facial attribute classification.

Multi-label Learning
Multi-label learning [20,21,22,23,24], which aims to learn multiple different but related labels simultaneously, has received much attention so far. For example, Xu et al. [46] analyze the local Rademacher complexity of empirical risk minimization (ERM)-based multi-label learning algorithms and then propose a new method that not only results in a sharp generalization error bound, but also provides a tight approximation of the low-rank structure. In [47], the authors present an additional sparse component to deal with the tail labels for the multi-label learning task. In [50], You et al. present the privileged multi-label learning (PrML) method to exploit the correlation between labels. For the problem of multi-label learning with missing labels, Jain et al. [48] develop a scalable and generative framework, which is based on a latent factor model for the label matrix and an exposure model for missing labels.
Recently, the DNN based multi-label learning method has been proposed for facial attribute classification in [14]. The features learned by DNN exhibit the hierarchical structure (such as pixels, edges, object parts and objects), where the low-level features usually share the similar representation. Therefore, the  [11], we obtain the correlation between different attributes. The facial attributes with high correlation can be clustered into the same group. multi-label learning based on DNN uses the shared features at the low-level layers and separately learns the features of multiple facial attributes at the high-level layers. However, such a method focuses on the imbalance problem of the data and does not explicitly consider the correlation between facial attributes. In this paper, we decompose the multi-label learning problem into a number of binary classification problems, where we solve all these problems using a single neural network at once. Furthermore, we not only consider multiple facial attributes learning, but also investigate the relationship between the different attributes based on attribute grouping.

Transfer Learning
The general performance of the traditional classifier trained with a limited number of labelled data may not be satisfactory, while the manual annotation of abundant training data for various tasks costs too much manpower. Fortunately, transfer learning [25,26,27] is an effective technique to improve the performance of the classifier in the target domain given only the annotated data in the source domain, which greatly reduces the labelling cost. Transfer learning also refers to unsupervised domain adaptation [28,29,30], and it can adapt the features from the labelled source domain to the unlabelled target domain. Long et al. [28] consider both the marginal and conditional distributions between the source and target domains via a Joint Distribution Adaptation (JDA) method. In [29], the authors propose to reduce the domain difference by jointly matching the features and reweighing the instances via a Transfer Joint Matching (TJM) method. Han et al. [49] propose a sparse multi-label transfer learning framework, which first learns a multi-label encoded sparse linear embedding space, and then maps the target data onto the learned space for circumventing the problem of multiple tags in practical applications.
With the development of deep learning, the DNN based transfer learning methods [31,32] usually add the adaptation layers to effectively reduce the discrepancy between the source domain and the target domain. For example, Long et al. [32] generalize DNN to domain adaptation, which is formulated as the Deep Adaptation Network (DAN). To the best of our knowledge, our proposed method is the first attempt for the domain adaptation of facial attribute classification. More importantly, we propose to use the deep transfer neural network based on multi-label learning for facial attribute classification.

The Proposed Method
The proposed facial attribute classification method (i.e., FMTNet) mainly consists of three sub-networks: 1) the Face detection Network (FNet) for face detection; 2) the Multilabel learning Network (MNet) for predicting multiple facial attributes simultaneously; and 3) the Transfer learning Network (TNet) for unlabelled facial attribute classification. These three sub-networks are designed to share the same convolutional layers (i.e., the first 13 convolutional layers in VGG-16 [33]), so that they can be easily fine-tuned. Fig. 2 shows the overall framework of the proposed FMTNet method for facial attribute classification. A detailed description of the proposed method is given in the following subsections.

FNet (Face detection Network)
FNet is designed to perform face detection, which outputs the positions of the faces. In this paper, we mainly follow the work of Faster R-CNN [34] to train FNet. Specifically, as shown in Fig. 2, we first take an image (of any size) as the input and use VGG-16 [33], which has 13 shareable convolutional layers, to obtain the feature map of the whole image. Then, we use the Region Proposal Network (RPN) to obtain a set of rectangular face proposals, each of which has a face score. The RoI pooling layer combines the convolutional features with the predicted bounding boxes, so that FNet can process a theoretical valid back-propagation. Finally, we get the face regions with reference to a series of pre-defined boxes by using the box regression.
The above training scheme shows the discriminative capability of FNet for face detection. Most importantly, FNet is used as the basis of MNet and TNet (i.e., these three sub-networks share the same network structures for their former 13 convolutional layers).

MNet (Multi-label learning Network)
The second sub-network is the Multi-label learning Network (MNet), which simultaneously predicts multiple facial attributes with labelled data. Specifically, we fine-tune the former 13 convolutional layers (which are the same as those in FNet) and train the latter three fully-connected layers for each attribute, as shown in Fig. 2. To reduce the model complexity of MNet, we decrease the dimensions of two fully-connected layers used in VGG-16 [33]. Specifically, we reduce the dimensions of the 'fc14' fully-connected layer from 4,096 to 512, and the dimensions of the 'fc15' fully-connected layer from 4,096 to 256. As shown in the experiments, we still obtain on par results with the lower dimensions of the fully-connected layers, thus reducing computational complexity.
MNet predicts one attribute in conjunction with the other attributes. The structure of MNet (see Fig. 2) contains two parts: (1) the shared layers which are collaboratively trained for all the attributes, and (2) the independent layers which are separately trained for each attribute. In this paper, we tackle the multilabel learning in an attribute-by-attribute style and decompose the multi-label learning problem into a number of binary classification problems. Moreover, in order to improve the generalization performance of MNet, we exploit the correlation between facial attributes based on attribute grouping.
Assume that there are I facial attributes to be predicted. Let us define the softmax loss term, L i , associated with the i-th attribute, and the corresponding class label (denoted as y i ) is written as follows: where and N is the number of training examples; p n,y i represents the probability of the i-th attribute class calculated by the softmax function; x n,y i represents the value of the last fully-connected layer predicted in the i-th attribute class; and y i can take on C i values, ranging from 1 to C i . In this paper, we consider the facial attribute classification problem as a binary classification problem (i.e., C i = 2). In other words, an example, which owns the corresponding label, is treated as a positive sample (i.e., y i = 1); otherwise, it is considered a negative sample (i.e., y i = 2). Of course, each facial attribute classification on MNet is not limited to the binary classification, and it can be easily extended to multi-class classification.
Thus, we derive the loss function of MNet as follow: where λ i denotes the loss weight for the i-th attribute. In MNet, multiple attributes are trained together, where each attribute is associated with a loss weight (see Eq. (3)). The loss weight has significant influence on the performance of facial attribute classification (see Section 4.2.1 for the experimental results). Moreover, Liu et al. [11] show that the facial attributes have clear grouping patterns. In other words, the facial attributes can be clustered into several groups, where the attributes in one group show high correlation and those between groups have low correlation (see Fig. 1). Therefore, based on this observation, we propose to define the loss weight for each attribute by taking advantage of attribute grouping, which effectively considers the relationship between different attributes. The details of the proposed loss weight scheme are given as follows.
Firstly, we cluster all the I facial attributes into G groups (we use the clustering method in [11]). Suppose that there are g m attributes (and their corresponding loss weights are defined as λ g 1 , λ g 2 , ..., λ g m ) for the g-th group. The loss weights for all the g m attributes are set to be the same, and their sum is equal to 1/G. Therefore, Figure 2: The overall framework of the proposed FMTNet method for facial attribute classification. The first sub-network is FNet for face detection. The second sub-network is MNet for facial attribute classification with supervised learning, where multiple facial attributes are predicted simultaneously. The third sub-network is TNet for facial attribute classification when labelled data are not available. The three sub-networks share the same structure at the former layers but they differ at the latter layers.
If there is only one attribute in a group, we directly set its corresponding loss weight to be 1/G. For example, we can assume that 6 facial attributes are clustered into 3 groups, where one group (Group 1) has 2 attributes and the other two groups (Group 2 and Group 3) have 3 attributes and 1 attribute, respectively. In this case, the loss weight for each attribute in Group 1 is 1/3 × 1/2 = 1/6, while that for each attribute in Group 2 is 1/3 × 1/3 = 1/9. The loss weight for the attribute in Group 3 is 1/3. Thus, the sum of the loss weights for the attributes in each group is the same (i.e., 1/3). In other words, the proposed loss weight scheme can effectively balance the loss weight of each group (no matter how many attributes are in this group) and prevents TNet from overfitting to one group (especially when many attributes exist in that group). As a result, the performance of MNet can be improved. Subsequently, we train MNet by using the stochastic gradient descent (SGD) algorithm to minimize the loss function on the training data.
Compared with traditional single-label learning methods [2,8,10,11,12], which independently train different attributes, MNet effectively overcomes this problem by training multiple attributes simultaneously. Accordingly, the shared layers in MNet significantly reduce the number of network parameters (recall that the DNN based single-label learning methods need to estimate the network parameters for each attribute).
Different from the DNN based multi-label learning method [14], the proposed MNet successfully exploits the correlation between these attributes based on attribute grouping. Recall that we cluster the attributes, which have high correlation, into one group, and then assign the loss weights according to different attribute groups. Therefore, the proposed loss weight scheme takes advantage of attribute grouping to assign the loss weights. It is worth pointing out that attribute grouping is a key step in the proposed method. This is because that not only the proposed loss weight scheme in MNet is based on attribute grouping, but also the performance of TNet is greatly affected by the results of attribute grouping (see Section 4.3.3).

TNet (Transfer learning Network)
The third sub-network is the Transfer learning Network (TNet), which explores the transferability of one attribute with labelled information to another attribute without labelled information. To optimize the training process, TNet shares the similar network structure as MNet and it is fine-tuned based on MNet. In transfer learning, the domain with labelled information is treated as the source domain, and the domain without labelled information is considered as the target domain. Moreover, data in these two domains are usually under different probability distributions. As discussed previously, the attributes in the source domain are trained using the multi-label learning method. In this paper, to deal with different data distributions in the two domains, we introduce a Reproducing Kernel Hilbert Space (RKHS) that is a high-dimensional space, where the domain discrepancy is measured by using the Multi-Kernels Maximum Mean Discrepancies (MK-MMD) criterion proposed by Gretton et al. [35]. Fig. 3 gives an intuitive example of transfer learning.
Assume that X s = {x s 1 , x s 2 , ..., x s m } consists of m data with the labelled information Y s in the source domain, and X t = {x t 1 , x t 2 , ..., x t n } consists of n data in the target domain. Thus, D s = {(X s , Y s )} represents the source domain, and D t = {(X t )} represents the target domain. Furthermore, let p denote the probability distribution of the source domain and q denote the probability distribution of the target domain. H k is assumed to be a RKHS defined on a topological space χ with the characteristic kernel k. The MK-MMD d k (p, q) is defined as the distance between the probability distributions p and q in the RKHS. Therefore, we formulate the squared form of MK-MMD as follows: where E p [·] is the mean of p and E q [·] is the mean of q. φ(·) is a feature mapping function, which maps the features from the original feature space to the RKHS space [35].
In MK-MMD, we select a kernel from a particular family of kernels K (here, the multiple Gaussian kernels are used for optimal kernel selection). Let us denote {k u } d u=1 as a set of positive definite functions, where k u : χ × χ → R. Hence, where β u is the coefficient of the u-th kernel. We constrain β u to guarantee that the derived kernel is characteristic.
The characteristic kernel, k(·), is defined as Accordingly, the distance between two probability distributions is computed as follows: where x s , x s are two features in the source domain and x t , x t are two features in the target domain. E x s ,x s [·] is the mean of x s and x s , and E x s ,x t [·] and E x t ,x t [·] are similarly defined. The objective of transfer learning is to minimize the domain discrepancy between the source domain and the target domain, which is effectively measured by the distance between the probability distributions from the source domain and the target domain (i.e., MK-MMD). Therefore, we have: where D F (= d 2 k (p, q)) denotes the domain discrepancy between two domains in the fully-connected layer.
We respectively embed three MK-MMD loss layers in the last three layers of TNet, and thus the total loss of the whole TNet is the sum of the softmax loss and three MK-MMD losses, which can be written as: where L is the total loss of the whole TNet, which is trained by the SGD method. The D F i denotes the MK-MMD loss in the last i-th layer, and L s represents the softmax loss of the source domain (α is the loss weight). The loss of TNet (see Eq. (9)) consists of two terms (i.e., the MK-MMD loss and the softmax loss). The objective of minimizing the MK-MMD loss is to reduce the discrepancy between the distributions of the source and target domains, while the objective of minimizing the softmax loss is to enhance the discriminability of the features in the source domain. In this paper, the MK-MMD loss is based on the kernel mean matching, where only features of data are used (while the labeled information is not used). Meanwhile, the softmax loss is defined in the source domain, where the labelled information is considered. By combining these two terms (i.e., the MK-MMD loss and the softmax loss), the discriminability of the features in the target domain is enhanced.
An example is given in Fig. 2, where the features learned from the 'Male' attribute are transferred to the 'Pointy Nose' attribute. The 'Male' attribute has a large amount of labelled training data in the source domain; whereas, the 'Pointy Nose' attribute has unlabelled training data in the target domain. Although the network structures of these two attributes are the same, the discrepancy of the features obtained in the latter layers is large. As shown in the experiments, the performance of feature transferability significantly drops if we perform direct transfer (i.e., train the network with the source data having labelled information, and then directly extract the features of the target data for classification). However, based on the MK-MMD loss function, the domain discrepancy between the 'Male' attribute and the 'Pointy Nose' attribute is effectively measured. As a result, by minimizing the MK-MMD loss, the difference between the two attributes is greatly reduced so that the feature transferability is significantly improved.
It is worth pointing out that TNet and MNet share the same feature extraction layers (i.e., the former 13 convolutional layers), while the latter fully-connected layers are different (see Fig. 2 for an illustration). Considering the limited number of training data, TNet directly uses the parameters of the first 6 convolutional layers, while fine-tuning the parameters of the subsequent 7 convolutional layers of MNet (i.e., we freeze 'conv1 1' -'conv3 2' to learn generic features and fine-tune 'conv3 3' -'conv5 3' to correct the slight domain biases on the convolutional layers of VGG-16 [33]). This is because that the fine-tuning strategy is usually much easier to obtain the network parameters which are learned by taking advantage of multilabel learning based on the labelled facial attributes data.
Although recent works [31,32] learn the transferable features with DNN, which also uses the MK-MMD loss function, the proposed TNet has some differences: 1) The proposed TNet focuses on facial attribute classification, while the methods in [31,32] are proposed for office image classification. 2) The proposed TNet is fine-tuned based on MNet and thus the network structures of TNet and MNet are similar. As a result, TNet takes both advantages of transfer learning and multi-label learning. In contrast, the methods in [31,32] mainly perform transfer learning via the adaptation network.

The Overall FMTNet
In the previous subsections, we have developed all the ingredients for the proposed FMTNet method. Specifically, we use FNet for face detection, use MNet for predicting multiple facial attributes simultaneously with labelled data, and use TNet for predicting facial attributes without labelled information. The training process of the overall FMTNet method is described in algorithm 1.

Experiments
In this section, we show the effectiveness of the proposed method for facial attribute classification. In Section 4.1, the datasets and parameter settings are described. In Section 4.2, the performance of MNet is evaluated. The influence of loss weight is given in Section 4.2.1. The comparison between single-label learning and multi-label learning is shown in Section 4.2.2. Performance comparison with several state-of-theart multi-label learning methods is described in Section 4.2.3. Finally, in Section 4.3, the performance of the proposed FMT-Net is evaluated. The performances obtained by direct transfer and transferring under different correlations are given in Sections 4.3.1 and 4.3.2, respectively. Performance comparison with several state-of-the-art transfer learning methods is described in Section 4.3.3.

Datasets and Parameter Settings
We train and evaluate different facial attribute classification methods on the CelebA [36] and LFWA [37] datasets. The CelebA dataset has 202,599 facial images of 10,177 identities, for each training sample x i ∈ D t do 20: Forward pass to obtain the feature representation of x i ; 21: Back propagate to update the network parameters W t via Eq. (9); 22: end for 23: end while and 40 binary attribute annotations are provided for each facial image. The CelebA dataset is divided into three parts: training, validation and test. More specifically, the training set, the validation set and the test set respectively contain 162,770 images, 19,867 images and 19,962 images. The LFWA dataset, with more than 1,680 identities, contains more than 13,000 facial images collected from the web. Each image in this dataset has 73 binary attribute annotations, where the positive or negative values indicate the presence or absence of the corresponding attributes, respectively. We use half of the LFWA dataset for training and half of the LFWA dataset for test.
Recall that the proposed method focuses on facial attribute classification, where the detected faces are used to perform subsequent classification. Therefore, we do not compare FNet with other face detection methods in this paper. In fact, the performance of the Faster R-CNN for face detection is reported in [38]. In this experiment, we use the training data from the CelebA dataset to train FNet and then use FNet as the basis of both MNet and TNet.

Results on the Multi-label Learning Network
In this experiment, we evaluate the performance of MNet using 40 attributes, as done in [11]. The detailed setting of MNet is given as follows: Firstly, MNet is initialized based on FNet. Then, we use 162,770 images on the CelebA dataset to train MNet. Finally, we test MNet with 19,962 images on the CelebA dataset. For the LFWA dataset, we use 6,571 images to train MNet and use 6,571 images to evaluate its performance.

Influence of Loss Weight
In this subsection, we evaluate the performance of MNet with different loss weight schemes on the CelebA and LFWA datasets, respectively. In Table 1, we show the classification accuracy obtained by MNet with different loss weight schemes, where MNet with the same loss weight scheme (i.e., the loss weight for each attribute is set to 1.0) and MNet with the proposed loss weight scheme are represented by 'MNet Eq' and 'MNet Prop', respectively. In 'MNet Emp', we emphasize the attributes of one group among all groups (i.e., the attributes in one group are assigned with higher loss weights than those in the other groups). In particular, in this experiment we choose one group containing multiple attributes (e.g., the group 2 shown in Fig. 1) and assign the large loss weight (i.e., 1.0) to each attribute in this group. We assign the small loss weight (i.e., 0.1) to each attribute in the other different groups.
From the experimental results in Table 1, we can see that 'MNet Prop' achieves better results than 'MNet Eq' on both CelebA and LFWA datasets. In comparison to 'MNet Eq', the improvement of 'MNet Prop' on the LFWA dataset is more obvious (about 2% improvement). The improvement is more significant for some individual attributes (for example, 'MNet Prop' achieves about 4% higher classification accuracy for the 'Attractive' attribute with regard to 'MNet Eq' on the LFWA dataset ). The proposed 'MNet Prop' effectively exploits the correlation between facial attributes by taking advantage of attribute grouping. More specifically, the loss weights corresponding to different facial attributes are assigned according to the results of attribute grouping (see Eq. (4)). In this manner, 'MNet Prop' balances the loss weights for different facial attributes, which not only considers the differences among the facial attributes (note that 'MNet Eq' treats each facial attribute equally), but also avoids over-fitting to one group (note that 'MNet Emp' tends to focus on training one group, leading to the problem of over-fitting).
Note that the performance of facial attribute classification is greatly affected by large facial appearance variations caused by pose, illumination, occlusion, etc. However, some facial attributes are more sensitive to the facial appearance variations than the other facial attributes. In other words, the classification task for some facial attributes is more difficult than that for the other facial attributes. For example, the variations caused by the face pose have more influence on the 'Pointy Nose' attribute than the 'Gray Hair' attribute. Thus, the classification performance obtained by the proposed method on the 'Gray Hair' attribute (about 98% classification accuracy on the CelebA dataset) is much better than that on the 'Pointy Nose' attribute (about 78% classification accuracy on the CelebA dataset).

Single-label Learning vs. Multi-label Learning
Secondly, we compare the proposed MNet with two methods (Single-SVM and Single-DNN) that are based on singlelabel learning (each attribute is learnt separately) on the CelebA and LFWA datasets, respectively. Specifically, Single-SVM denotes the method that uses DNN to learn the features of facial attributes, and then applies an SVM classifier for classification. 'Single-DNN' represents the method that uses DNN to extract facial features and predict facial attributes. The classification accuracy comparison is reported in Fig. 4.
From Fig. 4, we can observe that the proposed MNet achieves better results than the two competing methods that are based on single-label learning, which validates the effectiveness of the proposed method. This is because that multi-label learning effectively exploits the correlation between different facial attributes. In contrast, the network based on single-label learning does not consider the relationship among the facial attributes. Therefore, the single-label learning based methods do not make full use of the intrinsic information of facial attributes.
In summary, compared with the single-label learning, the multiple attributes can be simultaneously exploited in MNet. Furthermore, the proposed loss weight scheme assigns different loss weights for different attributes, which makes MNet based on multi-label learning more effective than the network based on single-label learning.

Comparison with the State-of-the-art Methods
Finally, we compare the performance of MNet with the following state-of-the-art facial attribute classification methods trained with the labelled information: FaceTracer [39], two versions of PANDA [10] (i.e., PANDA-w and PANDAl), two versions of LNets+ANet [11] (i.e., LNets+ANet (w/o) (without pre-training) and LNets+ANet), and MT-RBM (PCA) [13]. FaceTracer [39] uses the features of HOG and the color histogram to train an SVM for facial attribute classification. PANDA [10] employs hundreds of poselets, which are aligned to predict facial attributes. LNets + ANet [11] cascades two deep neural networks to detect the face and then learns the facial attributes from the detected parts. MT-RBM (PCA) [13] learns the joint feature representation of the faces and facial landmark points for predicting facial attributes.
Classification accuracies obtained by all the competing methods are reported in Table 2. For the CelebA dataset, we compare MNet with all the competing methods. The results obtained by the competing methods on the CelebA dataset are taken from [13]. For the LFWA dataset, we compare MNet with all the competing methods except for MT-RBM (PCA) [13], because that MT-RBM (PCA) [13] does not release the code on the LFWA dataset. Moreover, the LFWA dataset is not suitable for MT-RBM (PCA) to train and test, since the scale of the LFWA dataset is small and the data distribution of the LFWA dataset is unbalanced. The results obtained by the competing methods on the LFWA dataset are taken from [11]. We follow the same evaluation protocol provided in [11,13].
As shown in Table 2, MNet significantly outperforms the other competing methods on the CelebA dataset. Moreover, MNet achieves similar average accuracy compared with LNet + ANet (and obtains much better results than FaceTracer, PANDA-w and PANDA-l) on the LFWA dataset. LNet + ANet uses multiple patches cropped from the face region for data augmentation, which generates much more training data than our proposed method (using half of the images on the LFWA dataset). Thus, the training complexity of LNet + ANet [11] is much higher than that of MNet. In general, MNet achieves better or comparable performance compared with the state-ofthe-art methods.

Results of the Proposed Method
In this section, we evaluate the performance of the proposed FMTNet method for transfer learning on the CelebA dataset. The reason why we do not evaluate the proposed FMTNet on the LFWA dataset is that the data distribution of this dataset is unbalanced (for example, almost all the images are labelled with the 'Male' attribute and the 'Pointy Nose' attribute, while only a few images are labelled with the 'Heavy Makeup' attribute), making the LFWA dataset difficult to be used for evaluating the performance of transfer learning.

Direct Transfer
Firstly, we perform the direct transfer method from one attribute to other attributes on the CelebA dataset. In other words, we train a model with the labelled data of one facial attribute in the source domain, and then we use the trained model to directly predict other facial attributes in the target domain. The classification accuracy obtained on one attribute, which is directly transferred to other attributes on the CelebA dataset is reported in Fig. 5. Specifically, we use 162,770 images to train the 'Attractive' attribute which has the labelled information (while other attributes can also be used, and we observe similar results). Then, we predict the other attributes with this trained model.
As shown in Fig. 5, because the model is trained with the labelled information of the 'Attractive' attribute, the best results are obtained in predicting the 'Attractive' attribute. Furthermore, the accuracy obtained on the three attributes -'Young', 'Lipstick' and 'Heavy Makeup' -is slightly better than that obtained on the other attributes, because these three attributes belong to the same group as the 'Attractive' attribute. However, the results for other transfer tasks are worse, since we train the model without using the labelled information of these attributes. It is worth pointing out that although the 'No Beard', 'Oval Face' and 'Pointy Nose' attributes belong to the same group as the 'Attractive' attribute, the classification accuracy obtained on these attributes is not high. Therefore, the direct transfer method is not desirable and the performance is not stable.

Transfer under Different Correlations
The correlation between the source and target domains will greatly affect the performance of unlabelled facial attribute classification. In this subsection, we evaluate the performance of FMTNet under different correlations between the source and target domains.   into the same group (see Fig. 1). Therefore, these attributes have high correlation. On the other hand, the four attributes ('Arched Eyebrow', 'Gray Hair', 'Male' and 'Smiling') and the 'Pointy Nose' attribute belong to different groups [11]. In other words, these attributes have low correlation to each other. The performance comparison on different transfer learning tasks over the training process is reported in Fig. 6. From Fig. 6, the proposed FMTNet obtains better performance for the four transfer tasks ('Attractive' → 'Pointy Nose', 'Heavy Makeup' → 'Pointy Nose', 'No Beard' → 'Pointy Nose' and 'Oval Face' → 'Pointy Nose') than for the other four transfer tasks ('Arched Eyebrow' → 'Pointy Nose', 'Gray Hair' → 'Pointy Nose', 'Male' → 'Pointy Nose' and 'Smiling' → 'Pointy Nose'). This is mainly because the features from the source attribute and the target attribute are similar when they belong to the same group. Therefore, FMTNet achieves better performance when the correlation between the source attribute and the target attribute is high. In contrast, when the source and target attributes belong to different groups, the performance of the transfer learning task drops because of the significant difference of the data distributions between the two attributes. In summary, the more correlated the two attributes are, the better the transferring performance is (when the value of α is fixed to 1.0).
It is worth pointing that the red curve (i.e., 'Attractive' → 'Pointy Nose') is not rather stable at the initial 3000 iterations compared with the other curves. This is mainly because that the imbalanced training data are used at the beginning of the whole training process. However, the training accuracy becomes stable as the number of iterations increases, since the classification capability of CNN is gradually enhanced with the number of iterations. Recall that the number of training data for each transfer task is the same. But some facial attributes have many labelled data, while some other facial attributes only have a small number of labelled data. For example, there are totally 4,000 training data for each facial attribute, where 2,500 images are labelled as 'Heavy Makeup', while only 1,000 images are labelled as 'Attractive'. The small number of labelled data in the source domain (for example, in the task of 'Attractive' → 'Pointy Nose') will cause that the training accuracy is not stable at the beginning of the training process. In contrast, the training accuracy for some transfer tasks (for example, in the task of 'Heavy Makeup' → 'Pointy Nose') is more stable since the labelled data in the source domain are sufficient. Actually, Rudd et al. [14] have mentioned the problem of imbalanced data for facial attribute classification.

Comparison with the State-of-the-art Methods
In this subsection, we compare the performance of the proposed FMTNet with several state-of-the-art transfer learning methods, including TCA [41], MIDA [42], ITL [43] and GFK [44]. TCA [41] uses the MMD-regularized PCA, which is a conventional transfer learning method, to learn some transfer components across domains. MIDA [42] reduces the discrepancy between the source and target domains by maximizing the independence between the source and target features. ITL [43] consistently learns a domain-invariant feature space and optimizes an information-theoretic metric on the target domain. GFK [44] bridges the source and target domains by interpolating the intermediate subspace.
We use 5,000 images with the labelled information on the CelebA dataset as the source data, and 2,000 images without the labelled information on the CelebA dataset as the target data (also as the test data). The facial attributes are randomly chosen as the source data and the target data. For the traditional transfer learning methods, we extract the features from the source data and target data with single-label learning and FNet, respectively. Then, we use the logistical regression technique to predict the facial attribute.
Firstly, we choose the eight attributes (i.e., 'Attractive', 'Heavy Makeup', 'No Beard', 'Oval Face', 'Arched Eyebrow', 'Gray Hair', 'Male' and 'Smiling') as the source data and the 'Pointy Nose' attribute as the target data. According to the experimental results in Section 4.3.2, the performance of FMTNet is significantly affected by the correlation between the source and target attributes. Specifically, when the source and target attributes belong to the same group, FMTNet can achieve excellent results (with α = 1). FMTNet uses the MK-MMD loss to measure the difference between the source domain and the target domain, which makes it more discriminative than the other transfer learning methods. However, when the source and target attributes are from different groups, the mean accuracy obtained by FMTNet becomes worse (with α = 1). This is mainly due to the fact that when the attributes belong to different groups, the large value of α makes the trained model easily overfit to the source attribute. Therefore, when the source and target attributes belong to different groups, we also set the value of α to be 0.1. In other words, the network focuses more on the minimization of the three MK-MMD losses (mainly reducing the domain discrepancy), when the value of α is small. Therefore, FMTNet uses the MK-MMD loss to measure the difference between the source domain and the target domain, which makes it more discriminative than the other transfer learning methods. The mean accuracy obtained by all the competing methods is shown in Fig. 7, where the loss weight α of the softmax loss is set to 1.0 or 0.1. The 'Same Group' represents the attributes (i.e., 'Attractive', 'Heavy Makeup', 'No Beard' and 'Oval Face') belonging to the same group as the 'Pointy Nose' attribute. The 'Different Groups' represents the attributes (i.e., 'Arched Eyebrow', 'Gray Hair', 'Male' and 'Smiling') belonging to different groups compared with the 'Pointy Nose' attribute.
As shown in Fig. 7, the mean accuracy obtained by FMTNet outperforms that obtained by the competing state-of-the-art methods. Specifically, if the source and target attributes belong to the same group, the mean accuracy obtained by FMTNet achieves the top performance when the value of α is set to 1.0, as shown in Fig. 7 (a). And if the source and target attributes belong to different groups, the mean accuracy obtained by FMTNet achieves the top performance when the value of α is set to 0.1, as shown in Fig. 7 (b). In summary, the loss weight α should be chosen according to the relationship between two attributes for better performance. More specifically, if the source and target attributes belong to the same group, the loss weight α should be assigned with a large value. Otherwise, α should be assigned with a small value.
To show the feature transferability of the proposed method, Fig. 8 demonstrates the t-SNE embeddings [45] of the samples for the 'Attractive' → 'Pointy Nose' task with the original features and the transferred features (on the source and target domains), respectively. We observe that, with the original features, the data distributions on the source and target domains are quite different. However, with the transferred features obtained by the FMTNet method, the data distributions on the two domains become more similar. This observation further verifies the effectiveness of MK-MMD, which significantly reduces the domain discrepancy.
To further demonstrate the effectiveness of the proposed FMTNet method, we randomly choose 20 facial attributes as the source data and 20 facial attributes as the target data under two different cases (i.e., the attributes in the source domain and in the target domain are from the same group or from different groups). The performance comparison among all the competing methods is given in Table 3. As shown in Table  3, the proposed FMTNet (we set α = 1 when the source and target attributes belong to the same group and α = 0.1 when the source and target attributes belong to different groups) obtains the top mean accuracy among all the competing methods. Therefore, the loss weight α should be chosen according to the results of attribute grouping (i.e., whether the source and target attributes belong to the same group or not).
The proposed FMTNet method takes advantage of some existing components. However, our method is not simply stacked by these components. In FMTNet, three carefully designed networks (i.e., FNet, MNet and TNet) are used, where these networks share the same structure at their former layers and they differ at their latter layers. Therefore, the networks can be effectively trained via fine-tuning. In other words, TNet can be effectively fine-tuned based on MNet (which is initialized by FNet). Especially, a loss weight scheme is proposed to explicitly exploit the correlation between facial attributes based on attribute grouping, which can improve the generalization performance of the proposed method.
In general, compared with the other transfer learning methods, the proposed FMTNet, which transfers features from the source domain to the target domain, is very effective. The reasons are summarized as follows: (1) Instead of directly performing transfer learning for facial attribute classification, we take advantage of multi-label learning to learn the network parameters (MNet) and then fine-tune the transfer network (TNet) based on MNet. In other words, we use a hierarchical training strategy, where TNet is fine-tuned based on MNet, and MNet is fine-tuned based on FNet. Therefore, TNet and MNet can effectively trained via fine-tune.
(2) We improve existing multi-label learning and transfer learning components by exploiting the correlation between facial attributes and propose a loss weight scheme based on attribute grouping.

Conclusions
For the task of facial attribute classification, we have presented a novel method, termed FMTNet, which consists of three   different sub-networks -FNet, MNet and TNet -for face detection, multi-label learning and transfer learning, respectively. MNet in FMTNet predicts multiple facial attributes simultaneously for the labelled facial attributes. Moreover, MNet reduces feature redundancy with the proposed loss weight scheme, resulting in significant performance improvements. Based on MNet, TNet in FMTNet predicts facial attributes with unlabelled information by using MK-MMD. The proposed method, which combines multi-label learning with transfer learning, is general and can be applied to other computer vision tasks.