{'fraction_non_alphanumeric': 0.03327142687793891, 'fraction_numerical': 0.017188920061643024, 'mean_word_length': 4.741379310344827, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FDN: Feature Decoupling Network for Head Pose Estimation

Head pose estimation from RGB images without depth information is a challenging task due to the loss of spatial information as well as large head pose variations in the wild. The performance of existing landmark-free methods remains unsatisfactory as the quality of estimated pose is inferior. In this paper, we propose a novel three-branch network architecture, termed as Feature Decoupling Network (FDN), a more powerful architecture for landmark-free head pose estimation from a single RGB image. In FDN, we first propose a feature decoupling (FD) module to explicitly learn the discriminative features for each pose angle by adaptively recalibrating its channel-wise responses. Besides, we introduce a crosscategory center (CCC) loss to constrain the distribution of the latent variable subspaces and thus we can obtain more compact and distinct subspaces. Extensive experiments on both in-the-wild and controlled environment datasets demonstrate that the proposed method outperforms other state-of-the-art methods based on a single RGB image and behaves on par with approaches based on multimodal input resources.


Introduction
Facial analysis is one of the most studied topics in the past decades and plentiful methods have been proposed for various 2D facial problems, such as face detection, face recognition and face alignment. In recent years, 3D facial analysis has received more and more attention. Being an important basic of the nonverbal communication of humans, 3D head pose estimation can be utilized for human-computer interaction, human attention modelling, group behavior analysis, etc.
Since head pose estimation is a 3D problem, early methods (Martin, Van De Camp, and Stiefelhagen 2014;Meyer et al. 2015) using depth images can achieve high estimation performance. However, their applications are primarily limited by device, i.e., RGB-D cameras. Recently, with the breakthrough of deep convolutional neural networks (CNN), accurate facial landmark detection methods (Zhu et al. 2016;Bulat and Tzimiropoulos 2017) promote the performance of landmark-based head pose estimation from RGB images to Figure 1: Overview of our FDN. The feature decoupling (FD) module is proposed to achieve the decoupled features of yaw, pitch, and roll, respectively, from the identical feature map input. GAP denotes global average pooling. a large extent and get rid of the limitation of devices. Nevertheless, such two-stage methods incur extra error and computation caused by landmark detection process which can not be neglected.
To tackle these problems, landmark-free methods (Ranjan, Patel, and Chellappa 2017;  choose to regress three pose angles directly using features extracted from CNN. However, all these landmark-free methods regard head pose estimation as a straightforward regression or regression with classification problem under the assumption that the identical feature is applicable for the predictions of all three angles. Though existing landmark-free methods have achieved convincing performance in this manner, using identical feature is inappropriate as it still suffers from three main disadvantages. First, the feature subspaces of different angles are hybrid and non-separable, which leads to degradation of performance as features of yaw, pitch and roll have complementary but not identical information. Second, the exclusive features of each angle are not highlighted and are underutilized, which restricts the model from learning discriminative features for each angle. Third, it is inconsistent with human behavior as we will not focus on the same areas twice when asked to judge different pose angles (e.g., yaw angle vs. pitch angle) respectively from the same image.
To the best of our knowledge, none of the previous methods tries to deal with each angle on its characteristics, but to follow an image feature extraction and angle regression pipeline. In contrast, we argue that the feature subspaces of different angles are non-identical, and the customized features for each angle is beneficial to make the final predictions. Afterwards, we propose a novel three-branch network architecture in this paper, termed as Feature Decoupling Network (FDN), a more powerful architecture for landmarkfree head pose estimation from a single RGB image, as shown in Figure 1. In our FDN, we add a feature decoupling (FD) module for the identical feature extracted from backbone CNN to learn discriminative features for each pose angle respectively. Furthermore, based on our proposed FDN, we propose a cross-category center (CCC) loss to implement decoupling of latent variable subspaces. With the novel three-branch architecture, our FDN has three main advantages. First, benefited from FD module and CCC loss, it realizes the decoupling of latent variable subspaces, which makes the model focuses on the prediction of each pose angle separately. Second, it learns to highlight exclusive features of each angle meanwhile suppress less useful ones which is beneficial for final predictions. Third, being consistent with human behavior, it is comprehensible.
In summary, the main contributions of our work can be summarized as follows: • We propose a Feature Decoupling Network to achieve decoupling of latent variable subspaces to learn exclusive features of different angles in head pose estimation problem for the first time.
• We design a feature decoupling module to explicitly highlight discriminative features meanwhile suppress less useful ones by recalibrating channel-wise responses of each pose angle.
• We introduce a cross-category center loss to constrain the distribution of latent variable subspaces so that we can obtain more compact and distinct subspaces for different angle categories.
• Our proposed method outperforms other state-of-the-art methods based on a single RGB image and behaves on par with methods based on multimodal input resources.

Related Work
In this section, we provide a brief survey of head pose estimation and softmax-based loss functions.

Head Pose Estimation
Human head pose estimation has been a widely studied task in computer vision during the past few years with various methods and different modal databases proposed. Traditionally, RGB based methods (Murphy-Chutorian and Trivedi 2008; Huang, Shao, and Wechsler 1998) often use rotationspecific facial features to estimate head pose which is fragile due to various illumination conditions, expressions and occlusions. Some approaches based on depth images improve the robustness of estimation by registering a morphable face model to the depth images and combining several optimization methods (Meyer et al. 2015) or the ensembling of discriminative random regression forests (Fanelli et al. 2011).
In recent years, with the progress in RGB based facial analysis using convolutional neural networks (CNN), there are two main categories in existing methods for head pose estimation.
One is landmark-based which benefits from accurate facial landmark detectors (Bulat and Tzimiropoulos 2017;Zhu et al. 2016). Given 2D facial landmarks, we can obtain the head pose using algorithms such as CASSC (Gao et al. 2003). But such two-stage methods are indirect for head pose estimation. Zhu et al. (Zhu et al. 2016) directly fit a 3D face model using CNN and head pose is produced in the 3D fitting process.
The other is landmark-free, as landmark-based methods are affected by the accuracy of facial landmark detectors to a large extent and suffer from severe performance degradation in landmark invisible conditions as well as incur unnecessary computation. Closely related to other facial analysis tasks, head pose estimation is often performed in a multi-task learning framework. KEPLER (Kumar, Alavi, and Chellappa 2017) uses H-CNN to capture structured global and local features for accurate keypoint detection and provide head pose as a by-product. Hyperface (Ranjan, Patel, and Chellappa 2017) fuses the intermediate layers to perform face detection, landmarks localization, pose estimation and gender recognition simultaneously under a multi-task learning framework and demonstrates the synergy among these tasks. However, jointly learning with other tasks lacks the targeted research on head pose estimation problem. Methods such as (Ahn, Park, and Kweon 2014;Chang et al. 2017) exploit CNN architecture for head pose estimation in a regression manner. Drouard et al. (Drouard et al. 2017) propose to mix linear regressions with partiallylatent output based on CNN. In (Ruiz, Chong, and Rehg 2018;; Wang, Chen, and Zhou 2019), head pose estimation networks are trained under the joint supervision of classification loss and regression loss. Recently, FSA-Net (Yang et al. 2019) which employs the soft stagewise regression scheme and adopts a fine-grained structure aggregation outperforms the state-of-the-art methods on head pose estimation. In addition to single RGB images, Gu et al. (Gu et al. 2017) propose to use a recurrent neural network (RNN) for dynamic facial analysis in videos which is also landmark-free.

Softmax-based Loss Functions
In classification problems, a widely adopted pipeline which accepts class labels as supervision is softmax function followed by cross-entropy-loss to supervise the training process of the network. Based on softmax, advanced loss functions are proposed in face recognition. Schroff et al. (Schroff, Kalenichenko, and Philbin 2015) demonstrate the effectiveness of metric learning in face recognition by introduce the triplet loss. Wen et al. (Wen et al. 2016) propose the center loss which assigns embedding centers for each class and pulls the deep features closer to corresponding centers. Recently, some approaches take a deeper look into the feature spaces distribution. Zhao et al. (Zhao, Xu, and Cheng 2019) employs an exclusive regularization to enlarge the angle between different classes. Duan et al. (Duan, Lu, and Zhou 2019) propose an equidistributed constraint in loss function to uniform the distribution of deep face features in feature space. Different from the above methods, our proposed loss function is cross-category as features distribute in different subspaces. Apart from face recognition, DEPICT (Ghasedi Dizaji et al. 2017) introduces Kullback-Leibler (KL) divergence to decrease the distance between prediction and target distribution in clustering problems based on softmax.

Method
In this section, we first formulate our problem. Then we elaborate the proposed Feature Decoupling Network for head pose estimation.

Given a set of training images
is a pose vector whose elements correspond to the angles of yaw, pitch, and roll respectively. Our goal is to model a mapping function φ by minimizing the mean absolute error (MAE) between the predictions and labels, is the output pose predictions by the model.

The Proposed Feature Decoupling Network
Overview of FDN Let z and a = [a yaw , a pitch , a roll ] denote the latent variable subspace and the pose angles respectively. Then, the pose prediction process can be viewed from the perspective of conditional probability, i.e., p(a yaw , a pitch , a roll | z). Given z, a yaw , a pitch , and a roll are blocked from each other. Hence, previous works predict all three pose angles based on the identical feature provided by z. In contrast, we propose to customize latent variable subspace for each pose angle. Afterwards, the prediction process turns into p(a yaw , a pitch , a roll | z yaw , z pitch , z roll ), where z yaw , z pitch , and z roll are latent variable subspaces for yaw, pitch, and roll respectively. Given z j , j ∈ {yaw, pitch, roll}, a is blocked from z, which means a is dependent on z j , not z. Therefore, the latent variable subspaces can be decoupled by appropriate z j . As shown in Figure 1, the proposed FDN takes a single RGB image as input and pass it to the backbone CNN. The acquired feature map is then fed into the proposed FD module which we will describe in the next subsection. The output of branches are the final latent variables for predictions of different angles. The whole network can be trained in an end-to-end fashion. Feature Decoupling Module This module takes the acquired feature map as input and performs the decoupling of latent variable subspaces, as shown in Figure 1. Inspired by Squeeze-and-Excitation blocks (Hu, Shen, and Sun 2018) we implement our FD module with three channel attention blocks. It is a basic of CNN that features of different layers aim to encode different-level information. Pose information is encoded in high-layer features as low-layer features contain more detail information such as edges and texture. By adding FD module to the output feature map of backbone CNN, our FDN will not suffer from the problem of background clutter and semantic ambiguity which appears in low-level features. In FD module, we achieve the decoupling by adaptively recalibrating the channel-wise responses of each angle branch using parametric channel attention mechanism with a bottleneck layer consisting of two fully connected (FC) layers around the nonlinear function. Since channel dependencies are implicitly embedded in learned filters, our FD module explicitly captures discriminative features for each angle by performing angle-dependent feature re-weighting to select the more informative channel features while suppressing less useful ones and updating module parameters with angle-dependent losses. Head pose estimation can be seen as a regression problem naturally. Previous works (Ruiz, Chong, and Rehg 2018;Wang, Chen, and Zhou 2019) show that the combined utilization of classification and regression supervision can further enhance the model performance. Thus we also follow this way to construct a mixed loss with two losses. As shown in Figure 2, the decoupled feature is then fed into an FC layer followed with a softmax function to obtain bins probability prediction, and the classification target is a 1D Gaussian with the mean centered at the ground-truth class and a small variance as consecutive discrete targets have explicit semantics in this case. We use Kullback-Leibler (KL) divergence between target and prediction distributions to compute classification loss. Then, the angle prediction is obtained by computing expectation of the bins output, followed with a Mean Squared Error (MSE) loss used as regression loss. The final mix loss for each angle branch is the following, where q j is the output classification probability, y j is the angle ground-truth, G(·) denotes 1D Gaussian of the class target,ŷ j = q j ·b is the final angle prediction, b is the indexes of bins, j ∈ {yaw, pitch, roll} denotes each angle branch, and λ is a hyper parameter to trade off the classification loss term and the MSE loss term.

Cross-category Center Loss
We further propose a crosscategory center (CCC) loss to achieve the intra-class compactness and inter-category separability of latent variable subspaces at the same time. Similar to (Wen et al. 2016), the center loss part of CCC loss for each angle branch is defined as, is the embedding center of the y i th class which is updated during training phase, m is the size of mini-batch and j denotes each angle branch.
The above part shortens the distance between latent variables of the same discrete angle ground-truth to ensure the intra-class compactness according to different angle categories. It is a fact that latent variables of different angles should distribute in decoupled subspaces. However, this is omitted in the center loss. To alleviate this shortcoming, we further define the decoupling loss part, where j, j , j ∈ {yaw, pitch, roll} and j = j = j , s(j, j ) = c j −c j 2 and s(j, j ) = c j −c j 2 denote the cross-category correlation distance,c j is the average of latent variable centers from each angle category and adding one in denominator is to prevent from results overflow. The proposed cross-category center (CCC) loss consists of the above two parts and can be formulated as, where α is a hyper parameter to trade off the two parts.
Backbone Architecture and Optimization Taking feature extraction capability and model size into account, we construct our backbone network based on inverted residual blocks proposed in (Sandler et al. 2018). To be more specific, except for the initial convolution layer with 32 filters and the last one with 640 filters, our backbone network consists of 9 inverted residual blocks described in detail in Table  1 where we follow the notations in (Sandler et al. 2018). The total loss for each angle branch can be formulated as, where j ∈ {yaw, pitch, roll} denotes each angle branch. We update embedding centers with the following equation (Wen et al. 2016), where jk denotes the kth center of the jth angle branch, δ(condition) = 1 if the condition is true and δ(condition) = 0 otherwise. Algorithm 1 details the training process of proposed method.

Algorithm 1 Feature Decoupling Network (FDN) Training
Require: Training data {x (i) }, training labels {y (i) }. The trade-off parameters λ, α, the learning rate μ 1 for update network parameters, the learning rate μ 2 for update embedding centers and the number of iteration T . Ensure: The network parameters θ C in backbone CNN, θ yaw , θ pitch , and θ roll in FD module. 1: Initialize θ c , θ yaw , θ pitch and θ roll . Update the θ c via back-propagation. 11: end for 12: return θ C , θ yaw , θ pitch and θ roll .

Experiments
In this section, our proposed FDN is systemically evaluated against several state-of-the-art methods. Both quantitative results and qualitative results are reported.

Implementation Details
All the images are cropped around the face to include the whole head. After being randomly cropped to 224 × 224, the images are normalized by ImageNet mean and standard deviation. The trade-off parameters λ, α are set to 2.5 and 0.01 respectively in all experiments. SGD optimizer is used to update centers with the learning rate 5 × 10 −4 and Adam optimizer is used to update the network parameters with the learning rate 1 × 10 −4 . Batch size is set to 16, and the network is trained for 100 epochs in total. All experiments are carried out based on Pytorch. Our method bins angles range from −99 • to 99 • , and we discard images with angles outside of this range following (Yang et al. 2019).

Datasets and Protocols
The 300W-LP dataset (Zhu et al. 2016) is a large synthetic dataset which was derived from 300W dataset (Sagonas et al. 2013). Zhu et al. re-annotated a collection of popular inthe-wild facial 2D landmark datasets by fitting the 3D dense   (Ruiz, Chong, and Rehg 2018) 95.9 6.47 6.56 5.44 6.16 5.17 6.98 3.39 5.18 Hybrid Classification (Wang, Chen, and Zhou 2019) 96.7 4.82 6.23 5.14 5. Our experiments are conducted on these datasets following two widely used protocols described below. In protocol 1, the 300W-LP dataset is only used for training and the trained models are evaluated on both of the AFLW2000 and the BIWI dataset. When testing on the AFLW2000 dataset, we retrieve the ground-truth landmarks to loosely crop the faces and when testing on the BIWI dataset, we employ dlib for face detection and do not using tracking. In protocol 2, we split videos in the BIWI dataset in a ratio of 7:3 for training and testing respectively following (Yang et al. 2019). Face bounding boxes in the BIWI dataset are detected by MTCNN face detector ).

Comparison with the State of the Art
Results with protocol 1. Table 2 shows the comparison with state-of-the-art methods, including landmark-based and landmark-free methods on two benchmark datasets. Compared with landmark-based methods (Kazemi and Sullivan 2014;Bulat and Tzimiropoulos 2017;Zhu et al. 2016), our FDN is landmark-free, thus the prediction error is not affected by landmark detection process and the model size can be very compact. In addition, benefited from the proposed feature decoupling strategy and the carefully designed model structure, our FDN is more accurate and lightweight than landmark-free methods (Ruiz, Chong, and Rehg 2018;Wang, Chen, and Zhou 2019). On the challenging AFLW2000 dataset, our FDN outperforms previous state-of-the-art methods such as FSA-Net (Yang et al. 2019) by 12.8% on MAE, which further exhibits the superiority of our FDN. It is the first time that a method proposes to learn customized features for each angle branch respectively and provides impressive performance improvements.  (Martin, Van De Camp, and Stiefelhagen 2014) --3.6 2.5 2.6 2.9 VGG16+RNN (Gu et al. 2017) ->500 3.14 3.48 2.60 3.07 Results with protocol 2. We also compare our approach with other methods that input with different modalities such as RGB, RGB-Depth, and RGB-Time while our FDN only uses a single RGB image on the BIWI dataset in Table 3. Our method outperforms the state-of-the-art RGB-based methods such as FSA-Net (Yang et al. 2019) by 8.6%. In addition, our method is more compact than VGG16 (Gu et al. 2017) and more concise than FSA-Net. DeepHeadPose (Mukherjee and Robertson 2015) estimates head pose in multi-modal RGB-D videos by combining classification and regression model. Martin (Martin, Van De Camp, and Stiefelhagen 2014) builds and registers a 3D head model to estimate head pose from depth images. VGG16+RNN (Gu et al. 2017) uses an end-to-end network containing a CNN and an RNN for head pose estimation from consecutive video frames. Compared to these multi-modal methods, our method only uses pixel intensity information and narrows the performance gap between RGB based and multi-modal inputs based methods.
Visualization Figure 3 shows a few results of our FDN compared with the previous state-of-the-art method, i.e., FSA-Net. It can be seen that our FDN is more robust for various poses and lighting. In addition, Ruiz et al. also try to predict three angles separately in Hopenet but using hybrid features for various angles. We visualize the Class Activation Map (  based on more discriminative local regions meanwhile most global features are fully utilized for all three angles and the boundary between local and global regions output by our FDN is more stable. It proves that our FDN learns more distinct subspace for each angle. In Figure 5, we show that by using 1D Gaussian, our FDN can not only reduce evident  misclassification but also provide more accurate pose prediction. In Figure 6, we visualize feature distributions using the t-SNE (Maaten and Hinton 2008), which shows that our FDN achieves the decoupling of feature subspaces across angle categories meanwhile maintains the intra-class compactness for each bin.

Ablation Study
In this subsection, we conduct extensive ablation studies to further demonstrate the effectiveness of our Feature Decoupling Network (FDN). Table 4 shows the experimental results on the AFLW2000 dataset and the BIWI dataset. In order to fairly compare, We adopt the same backbone network architecture as described in Table 1 across all three sets of experiments. The baseline model is in the 1st row which uses 3 FC layers to estimate three angles separately from the hybrid feature. Compared to the baseline model, adding our FD module to the hybrid feature and training the network with L mix loss (2nd row) improves the performance by 1.9% and 4.9% on the AFLW2000 and the BIWI dataset respectively, which indicates that our FD module learns to customize discriminative features for each angle branch and is beneficial to estimate the angles. As shown in the 3rd row, adding our L CCC loss to the model with FD module further enhance the performance by 3.0% on the AFLW2000 dataset and 4.6% on the BIWI dataset as the proposed L CCC loss promotes the decoupling of latent variable subspaces with different angles on the basis of FD module. Overall, our FDN outperforms the baseline model by a large margin, i.e., 5.0% and 9.4% on the AFLW2000 and the BIWI dataset, respectively. Furthermore, we evaluate the influence of different backbone structures in Table 5. We use ResNet18 with residual blocks for comparison. It shows that using inverted residual blocks is beneficial to the performance as well as the model size. However, the performance margin brought by our proposed method is still considerable. As shown in Table 4, our proposed FDN outperforms the baseline model by 9.4% on the BIWI dataset, which proves the effectiveness of our method.

Conclusion
In this paper, a novel three-branch network architecture, called Feature Decoupling Network (FDN), is proposed for landmark-free head pose estimation from a single RGB image. Different from previous works, We propose to decouple hybrid features and customize exclusive latent variable subspace of each angle by our proposed feature decoupling (FD) module and cross-category center (CCC) loss. In the FD module, we explicitly select discriminative features and suppress less useful ones of each angle by adaptively recalibrating its channel-wise responses. The CCC loss further improves the performance by encouraging more compact and distinct latent variable subspaces of different angle categories. Extensive experiments on the 300W-LP, the AFLW2000 and the BIWI datasets show the superiority of our FDN which results in the state-of-the-art performance on these datasets.