{'fraction_non_alphanumeric': 0.02837960372735516, 'fraction_numerical': 0.007832847799664306, 'mean_word_length': 4.683552631578947, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Facial Analysis and Performance Driven Animation through Disentangling Identity and Expression

We present techniques for improving performance driven facial animation, emotion recognition, and facial key-point or landmark prediction using learned identity invariant representations. Established approaches to these problems can work well if sufficient examples and labels for a particular identity are available and factors of variation are highly controlled. However, labeled examples of facial expressions, emotions and key-points for new individuals are difficult and costly to obtain. In this paper we improve the ability of techniques to generalize to new and unseen individuals by explicitly modeling previously seen variations related to identity and expression. We use a weakly-supervised approach in which identity labels are used to learn the different factors of variation linked to identity separately from factors related to expression. We show how probabilistic modeling of these sources of variation allows one to learn identity-invariant representations for expressions which can then be used to identity-normalize various procedures for facial expression analysis and animation control. We also show how to extend the widely used techniques of active appearance models and constrained local models through replacing the underlying point distribution models which are typically constructed using principal component analysis with identity-expression factorized representations. We present a wide variety of experiments in which we consistently improve performance on emotion recognition, markerless performance-driven facial animation and facial key-point tracking.


Introduction
One of the primary sources of variation in facial images is identity. Although this is an obvious statement, many approaches to vision tasks other than facial recognition do not directly account for the interaction between identity-related variation and other sources. However, many facial image datasets are subdivided by subject identity and this provides additional information that is often unused. This paper deals with the natural question of how to effectively use identity information in order to improve tasks other than identity recognition. In particular, our primary motivations, applications of interest and goals are to develop methods for facial expression analysis and performance driven facial animation that are less identity dependant. * Joint first authors.
Recently there has been work on facial recognition in which identity is separated from other sources of variation in 2D image data in a fully probabilistic way [37]. In this model, the factors are assumed to be additive and independent. This procedure can be interpreted as a probabilistic version of Canonical Correlation Analysis (CCA) presented by Bach [2], or as a standard factor analysis with a particular structure in the factors. In this paper, we investigate and extend the use of this probabilistic approach to separate sources of variation, but unlike prior work which focuses on inferences about identity, we focus on facial expression analysis and facial animation tasks. This includes performance-driven animation, emotion recognition, and key-point tracking. Our goal is to use learned representations so as to create automated techniques for expression analysis and animation that better generalize across identities. We show here how disentangling factors of variation related to identity can indeed yield improved results. In many cases we show how to use the learned representations as input to discriminative classification methods.
In our experimental work, we apply this learning technique to a wide variety of different input types including: raw pixels, key-points and the pixels of warped images. We evaluate our approach by predicting: standard emotion labels, facial action units, and 'bone' positions or animation sliders which are widely used in computer animation. We go on to show how it is indeed possible to improve the facial keypoint prediction performance of active appearance models (AAMs) on unseen identities as well as increase the performance of constrained local models (CLMs) through our identity-expression factorization extensions to these widely used techniques. Our evaluation tasks and a comparison of the types of data being used as input for each experiment are summarized in Table 1.
The rest of this manuscript is structured as follows: In Section 2, we discuss the facial expression analysis, performance driven animation and keypoint tracking applications that serve as the ultimate goals of our work in more detail and review relevant previous work. The various methods we present here build in particular on the work of Prince et al. [37] in which a linear Gaussian probabilistic model was proposed to explicitly separate factors of variation due to identity versus expression. In section 3, we present this model as a way to disentangle factors of variation arising from identity and expression variation. While Prince et al. used this type of identity-expression analysis to make inferences about identity, our work here focuses on how disentangling such factors can be used to make inferences about facial expression. Indeed, as discussed above, facial expression analysis and the applications of detailed facial expression analysis to computer animation is the motivating goal of our work here.
The first set of contributions of our work are presented in section 3.2. These contributions consist of a wide variety of novel techniques for using learned identityexpression representations for common goals related to expression analysis and computer animation. We provide novel techniques and experiments in which we predict: emotion labels, facial action units, facial keypoints, and animation control points as summarized in Table 1. Of particular note is the fact that we propose a novel formulation for identity normalizing facial images which we use for facial action unit recognition, emotion recognition and animation control. We find that using this identity-normalized representation leads to improved results across this wide variety of the expression analysis tasks.
We also go on to extend the underlying identity and expression analysis technique in two important directions, providing two additional technical contributions. First, in Section 4 we show that identity-expression analysis can be used in place of the principal component analysis (PCA) technique that is widely used in active appearance models (AAMs). This procedure yields what we call an identity-expression factorized AAM, or IE-AAM. We present the modifications that are necessary to integrate this approach into the AAM framework of [34]. Our experiments show that IE-AAMs can increase the performance of PCA-AAMs dramatically when no training data is available for a given subject. We also found that the use of IE-AAMs eliminated the convergence errors observed with PCA-AAMs. Secondly, in Section 5 we show that constrained local models (CLMs) can also be reformulated, extended and improved through using an underlying identity-expression analysis model. Our reformulation also provides a novel energy function and minimization formulation for CLMs in general.

Our Applications of Interest and Relevant Prior Work
Performance-driven facial animation. Performance-driven animation is the task of controlling a facial animation via images of a performer. This is a common process in the entertainment industry. In many cases, this problem is generally handled by the placement of special markers on the performer [52]. However, here we are interested in developing markerless motion capture techniques in which we employ machine learning methods and minimize manual intervention.
Many marker-less facial expression analysis methods rely either on tracking points using optical flow [16] or fitting Active Appearance Models [30]. Morphable models in [6], [36], [40] have also been investigated. More recently, 3D data and reconstruction is used to fit directly to the performer [51], [54]. These methods, however, often require additional data, for example, multiview stereo [53], [4] or structured light [7], [51]. Sandbach et al. provide a thorough review of 3D expression recognition techniques in [38]. In the end, these methods usually work by providing dense correspondences which then require a re-targeting step.
However, a simpler and often used approach in industry does not rely on markers and simply uses the input video of a facial performance. The idea is to use a direct 2D to 3D mapping based on regressing image features [21] to 3D model parameters. This method works well but is insufficiently automatic. Each video is mapped to 3D model parameters, possibly with interpolation between frames.
Key-point based representations often require both data and training time that compares unfavorably to this simpler approach given the re-targeting step. The primary benefit of key-point based representations appears to be a degree of natural identity-invariance. We provide experiments on performance driven animation, in which we predict bone positions using the well known Japanese Female Facial Expression (JAFFE) Database [33] as input as well as an experiment using professional helmet camera based video used for high quality, real world animations.
Emotion recognition. There is a large amount of overlap in the objectives for performance-driven facial animation and the objectives of detailed emotion recognition. Automatic emotion recognition has largely focused on the facial action coding system or FACS [15] as an auxiliary task [46]. In our work here we shall predict action unit (AU) values that have been annotated for the well known extended Cohn Kanade database [32]. In some cases, a FACs representation may be a useful intermediate representation for animation control.
Emotions can also be encoded via the simpler and more intuitive labels for well known emotion types of "Anger," "Disgust," "Fear," "Happiness," "Sadness," "Surprise". We evaluate the performance of recognizing the annotations of these emotions for JAFFE as well as CK+ in which "Contempt" is also included. These labels correspond to Paul Ekman's famous set of six basic emotions plus "Contempt" which is a part of his extended list of emotions.
Many existing approaches for emotion recognition also require significant preprocessing for contrast normalization and alignment, which helps to alleviate pose variation [43]. Indeed, an intermediary task that has often been used as a preprocessing phase to produce emotion recognition systems is key point detection and tracking [42], [3], [45], [9]. Tracked key-points themselves can also be used to control animations directly or indirectly via warping operations on images as we explore in our experiments on professional facial performance video obtained via helmet cameras.
AAMs and CLMs usually suffer from a degree of identity-dependence. That is, a model trained on a sample of subjects does not necessarily perform well on an unseen subject. AAMs in particular suffer greatly from this effect, performing much better when samples of an individual are used for both training and testing.
In this paper, we investigate the problem when no additional information about a new subject is available, as is the case in many common scenarios. As such, we evaluate our method using the AAM as a test algorithm, because it performs poorly in this setting but performs well otherwise.
View-based approaches [35], and multi-stage solutions [44], [31] address this issue by using multiple subspaces for each identity. Gaussian mixture models, [20] indirectly deal with identity variation by learning clusters of training data. Gross et al. described reduced fitting robustness of AAMs on unseen subjects [22], suggesting simultaneous appearance and shape fitting improve generalization. Our method approaches identity variation directly and probabilistically, learning both factors of variation simultaneously. Some other notable recent work from Jeni et al. has extended CLMs to account for 3D shape. Their work has demonstrated extremely high performance on the CK+ expression classification and action unit detection evaluations. We compare our approach with their method in Section 3.2.
Other work has used a structured max-margin approach to keypoint placement [56] yielding state of the art results for facial keypoint placement. We compare with this approach in Section 5.
Deep learning has recently emerged as a method capable of yielding state of the art results for keypoint placement [25,41]. Deep networks are however known to overfit when data set sizes are small, which is the case for many of the problems that we are addressing in our work here (ex. markerless facial performance capture imagery from helmet mounted cameras).
Other factorization techniques and our approach. Multi-linear and bilinear analysis of facial images [19], [48], [50] can model the interaction of different kinds of variation. However, with these models, a full image tensor is often needed, which can be difficult to obtain. Some approaches overcome this restriction by treating the required tensor labels as missing [12]. However, this leads to discarding data.
The problem we address in this paper is that of learning an expression representation which leverages identity information, then using the technique to improve a wide variety of expression and emotion related tasks. This is similar in spirit to the expression synthesis approaches used by Du and Lin [13] and by Zhou and Lin [55], which generate identity-independent expression factors.
Our approach generalizes this work as a framework for unsupervised learning of expression factors, building in particular on the identity-expression factorization technique of Prince et al. [37]. In the next section we present their probabilistic approach to this task, then go on to show how show how this formulation can be used to create an extremely wide variety of new techniques leading to improved performance for expression recognition and facial action recognition tasks. Subsequently, in Sections 4 and 5 we go on to extend the widely used formulations of AAMs and CLMs so as to use identity-expression factorized representations. In both cases we show that keypoint localization performance is improved over the traditional techniques which use principal component analysis as their underlying point distribution models.

A Model for Disentangling Identity and Expression
Face image datasets have a rich divesity due to many contributing factors of variation such as age, illumination, identity, pose, facial expression and emotion.
In this literature, we consider two main sources of variation. The primary variation is due to the identity factors and the second source of variation is due to the facial expressions as well as some degree of pose deviation.
A graphical model of this approach is shown in Figure 1, where each face image x ij ∈ X is generated by w i , representing the identity i, and v ij , representing the j th expression of identity i. The set of observed data is X = {x ij } i=1,...,I ; j=1,...,J i which contains all images of I identities, where for each each identity i, a total of J i expressions exist. The observed data can be pixels, vectors containing keypoint locations or a concatenation of both. We use the notation x ij to denote the j th image of the i th identity in the dataset.
Figure 1: Graphical model of facial data generation. x ij is generated from p(x ij | v ij , w i ), after sampling w i from an identity and v ij from an expressionrelated distribution respectively. Each x ij is generated by sampling w i and v ij from Gaussian distributions corresponding to identity i and its j th expression distributions p(w i ) and p(v ij ), and then combining these by sampling an image x ij according to p(x ij | w i , v ij ).
We use zero-mean independent Gaussian distributions for p(w i ) and p(v ij ).
The observation x ij is then sampled from a multivariate Gaussian conditional distribution parameterized by the mean µ, matrices F, G and diagonal covariance Σ, such that This corresponds to a conditional distribution given the variable w i , which is identical for all images of a unique identity, and the variable v ij , which varies across different expressions of a particular identity. The loading matrices F and G correspond respectively to identity and expression and are shared across all observations. The joint probability can be written as where X is the entire observable dataset and W = {w 1 , w 2 , . . . , w I } is the set of all latent identity representations and V = {v i j} i=1,...,I ; j=1,...,J i is the set of all latent expression representations for all identities.

Learning
The Expectation Maximization (EM) approach is used to learn the parameters of the model θ = {F, G, Σ, µ, λ, ρ}. The goal is to maximize the joint distribution by alternatively maximizing and taking expectations where the expectation is taken with respect to the posterior distribution and then the parameters in θ are updated by maximizing the log of the joint distribution p(W, V, X). Since the priors in Eqs.
(1) and (2) and the likelihood distribution in Eq. (3) are all Gaussian, the resulting joint distribution is also Gaussian.
For a given identity i, the set of observed variables {x ij } j=1,...,J i , can be written as a single feature vector Similarly the factors can be combined into a single loading matrix where the number of rows and columns in A i are J i and J i + 1. The latent space representation corresponding to identity i becomes Having w i ∈ R K and v ij ∈ R L , d i becomes a vector of dimensionality K + L × J i . Since d i is composed of two sets of vectors with zero-mean Gaussian distributions, it is also distributed as a zero mean Gaussian: Φ i = diag (λ 1 , . . . , λ K , ρ 1 1 , . . . , ρ 1 L , . . . , ρ Ji 1 , . . . , ρ Ji L ) , where the term Φ i is a diagonal covariance matrix, in which the first K elements of the diagonal are extracted from the diagonal elements of the covariance matrix λI. The diagonal of Φ i is then composed of J i blocks of L elements that are extracted from the diagonal of covariance matrix ρI repeated J i times. Intuitively the first set of K elements represent identity variations, while each of the J i blocks represent L factors encoding expression variations within each image.
Given this construction, the probability for identity i can be rewritten as a Gaus- The posterior probability of d i is also Gaussian with moments In the E-step, the mean and covariance matrices of the posterior distribution This is due to the fact that the parameters of the model should be updated with respect to all of the data in the training set and having done the E-step, we have already got the same identity representation for all expression observations from the same person. Therefore, disentangling d i into a set of d ij can reserve that information while at the same time it encodes a simpler latent representation for updating the parameters of the joint distribution. We define Given this notation, the conditional distribution can now be written as We can further simplify this notation by settingd ij = , which in turn gives the following conditional and prior distributions withΦ being equal toΦ where two zero vectors with row or column size of K + L are concatenated with Φ and another zero element to build a square matrix of size K + L + 1. The joint distribution then gets equal to with X representing the whole set of observed data and D representing the entire set of latent variables. This simplification is preferred since the variables F, G, µ are mutually dependent. Updating each one requires the other two. In the new notation, all of them can be updated simultaneously by updatingB. Maximizing with respect toB, Σ,Φ, give the following updates Note that the parameters are updated with respect to all expressions of all identities. As for inference at test time, the procedure for determining the optimal d ij vector is straight-forward, using the following posterior distribution:

Facial Expression Analysis, Animation Control Methods and Experiments
In the following sub-sections, we build upon the model above to create techniques for emotion recognition and performance-driven animation and evaluate the methods using multiple datasets. We first present emotion recognition methods and results, including the task of facial action unit classification. Then, we move on to performance-driven animation techniques and experiments.
A summary of these experiments is shown in Table 1. Images for a single test subject, left out from training, shown here with predicted labels from our method. The data is shown in two sets of rows. The first row in each set is the original input data, the second a rendering of the mesh with corresponding bone positions predicted by the model.

Emotion Recognition with JAFFE
We first run a set of experiments on the constrained JAFFE dataset [33]. The JAFFE dataset contains ten identities with varying expression across seven emotions containing 213 images in total. The frontal facial images are roughly aligned but we use a funneling algorithm [26] after face detection to correct small pose variation.
We then use an ellipse, manually specified, to mask background variation and reduce dimensionality, and then divide the face into three rectangular regions roughly corresponding to the mouth, eyes and ears. However, unlike previous works, such as [55], we do not manually locate facial landmarks. Then we run the algorithm for each region individually, learning a composite space in which facial expressions are treated as a linear combinations. The expression space weights for each region are learned separately and then concatenated to form a single input vector consisting of the weights v ij .
We then predict seven emotional states -anger, disgust, fear, happiness, neutral, sadness and surprise -using the learned representation v ij . We train a SVM RBF classifier for each of seven emotional states on the in-sample subject images, using a single image from each of the emotions of the left-out subject as a validation set to learn the slack and kernel bandwidth parameters. We treat the classification at test time as a one-against-all prediction. We compare our results against PCA performed on the same input data as our model, using 30 and 100 dimensions. The final rates are shown in Table 2 as the average accuracy across identities.
The overall prediction accuracy rate for the emotion prediction for an unseen identity is 72.17% using a 30 dimensional expression space. A recent result on the JAFFE set by Cheng et al. [8], in experiments with left-out subjects, obtained a mean accuracy of 55.24% using Gaussian process classification. Because their pre-processing is not identical and they do not report standard errors, we do not include it in Table 2.

Emotion Recognition and Facial Action Unit Detection with CK+
Certainly, the JAFFE data exhibits far less variation than is usually present in real-world data. In this section, we present more detailed experiments using the Procrustes analysis using affine transformation of the landmark positions to determine a mean shape and register the point locations to the mean by estimating the least square best 2D transformation. We then run PCA on the difference between the Procrustes aligned points and the original points to determine the similarity components. We add these to the AAM shape parameters to model rigid motion.
Finally, a piece-wise affine warp is applied to normalize the shape of each facial image to the base shape recovered from the Procrustes analysis to obtain warped images.
Using this approach, we generate two feature sets. The first is point locations after Procrustes analysis. The second set contains shape-normalized images, which are converted to vectors. We use leave-one-subject-out cross-validation us- For our identity-normalization experiments, we use the point-locations and shape-normalized images as inputs to our factor analysis. In this case, we use 100 and 30 dimensions for the identity and expression parameter vectors respectively.
In order to avoid overfitting, we increase the training data size for this unsupervised step from 1186 to 2588 by ensuring that each identity has at least 20 images, by sampling uniformly from intermediate frames. We do not use any testing subject images during training our models, including during the factor analysis in order to maintain fairness. The results are shown in Table 3, which shows an increase in average AUC for each of the feature-sets individually. In a majority of AU classes, the AUC score increased, while there are no large reductions in AUC after identity normalization. For both the SPTS and CAPP scores, identity normalization showed significant increases in performance for AU-7, Lid-Tightener and AU-15, Lip Corner Depresser. AU-7 is associated with an eye-narrowing or squinting action, and AU-15. We show examples from this dataset of these two AU's to show that resulting normalization leads to representations that can be seen as roughly identical.
In both cases, the expression is well reproduced in the identity-normalization procedure. The increase in performance from using both feature-sets is apparent again, which is greater than .86 AUC for every AU and well over 90% for most of them. Our comparison shows that the simple step of identity normalization improves performance on average. It appears to work especially well for those AU's that are easy, in some sense, to capture between identities.
We also place our work in further context with the method recently proposed by Jeni et al. [28] for shape based action unit recognition. They give an average AUC result of 89.13%. Our shape, texture and combined average AU results were 92.8%, 94.4% and 96.1%, respectively.
Emotion Detection. The baseline approach can also be used for emotion recognition. Again, we recreate their approach and use the identity-normalization step as an unsupervised learning procedure. Because of this, we can use more training examples than those that are labeled, making this procedure semi-supervised. Again, the SVM's and logistic regressors are learned using leave-one-subject-out, that is, over a total of 123 trials.
However, after combining scores from the SVM's using a logistic regression technique, the improvements are quite impressive. As in [32], results after the calibration of SVM scores indicate that the point data and image data capture different kinds of information. Combining these features yields impressive gains in performance. "Anger," especially improves using our method. In particular, only "Happiness" recognition is reduced using our approach, but the reduction in hit rate is modest. Overall, this method gave a 95.21% average accuracy compared to the result of 83.27% reported in [32]. Comparison of confusion matrices of emotion detection for the combined landmark (SPTS) and shape-normalised image (CAPP) features before and after identity normalisation is shown in Table 4.
More recently Jeni et al. [28] have proposed a 3D shape model and explored two normalization methods for the CK+ expression recognition evaluation. They report that their action unit 0 or AU0 normalization procedure yields an average class accuracy of 96%, while their personal mean shape procedure yielded an average class accuracy of 94.8%. Our average accuracy above is given as the per example average whereas these results were computed on a per-class basis. Our average per-class accuracy was 93.1%.

Animation Control Experiments with JAFFE
We again begin using the JAFFE dataset. Each image, after face detection and alignment, is labeled using a common 3D face mesh fitted with 27 bones created by an artist, as can be seen in Figure 2. Each bone is then positioned by an artist to a maximal and minimal position along a fixed path. The position along the path is specified by a real value in [−1, 1] as a fraction of the distance between the midpoint and the extreme values.
We use MSE for evaluation purposes on the animation experiment, which corresponds to bone parameter recovery. In this case, we use linear regression as the predictive algorithm. The MSE reported is the average error in bone position for all test images. For each trial, we leave one identity out from the trial, and compute both the MSE error and a prediction for the facial action. We compare our approach to PCA, using 30 and 100 dimensions. The results are summarized in Table 2. For the experiment labeled "none," the experiment denotes no unsupervised pre-processing step -the input is the raw image data. As it is difficult to gauge the quality of the predictions from MSE alone, the predictions for a test subject using our method are illustrated in Figure 2, showing that the method does recover the facial actions produced by the subject quite well. The small standard error of the MSE indicates that, in general, the procedure is capable of predicting facial actions across all unseen subjects.

Animation Control Experiments with Markerless Motion Capture
We now return to the challenging real-world problem of high quality facial animation control using video obtained from helmet cameras. This technique was used in the well known film Avatar and this data comes from Ubisoft, the company responsible for the brands Assassin's Creed and FarCry among others. FarCry 3 uses these technologies extensively. We obtained 16 videos of motion capture data without marker data. These videos are produced using infra-red cameras.
Examples are shown in Figure 3. As is common in motion capture data, faces are captured using a helmet-mounted camera which reduces pose variation. The camera position is relatively fixed. However, this data presents new challenges due to variation stemming from the varied appearance of the actors. To compensate for appearance variation due to facial structure, an Active Appearance Model is applied to the data as shown in Figure 3, using 66 fiducial points. The resulting points derived from the model are used to warp each video frame to a common coordinate structure, removing pixels from outside the convex hull. These were used as inputs to our model, for a dataset totaling 3122 frames. Example outputs can also be seen in Figure 3b.
For our evaluations, we were given professional results of a bone-based model   Table 2. We first show results using no preprocessing except for face detection using the Viola-Jones detector, cropped to a size 1.5 time the size of the detector to account for the distortion evident in the images.
The results shown in Table 2 are per bone, and clearly some data processing must be used (None* refers to the images after shape normalization). Best results were not obtained by applying PCA to the original data. The remaining experiments used shape-normalised data as inputs. As can be seen PCA improves the results, but the best results are achieved using our model. The classifier used was a simple linear regression, a more powerful discriminative classifier may be more effective.

Identity-Expression Active Appearance Models (IE-AAMs)
For many of the experiments in the preceding section a key step is in the application of key-point detection algorithms, specifically AAMs. The AAM has been shown to work very well for subjects on which it has been trained while generalization to unseen identities has proven more difficult, due to the increased complexity of the shape model [22]. Since labeling data for new identities can be labor intensive we would like to determine if AAM performance can be increased for an unseen identity using an identity-expression factorization extension to the traditional AAM formulation.
The traditional formulation on an AAM contains a PCA model of the joint point and appearance data. This lends itself to a linear Gaussian interpretation which leads us to investigate whether key-point localization itself is improved using identity normalization. In this section we develop a novel identity-expression factorized AAM and investigate the utility of identity normalization for key point detection.

Inverse Compositional AAMs
The AAM fitting procedure we adapt is the inverse-compositional method as described by Matthews and Baker in [34]. We briefly review this method here.
The modified objective in AAM fitting is to minimize the error between a template image, T and a set of warped images I j , j = 1, 2, ..., J, which generally belong to the same identity. The modified objective is to minimize the squared error between the set of image over all image locations x = (x, y), by estimating parameters p j of a warp function W , or We change notation here to make the following more clear. The warp function, W , can be interpreted as a map which determines a new location for any point x, i.e. , x = W (x, p j ). For AAM models, this is typically a piecewise affine transform based on the triangulation of a shape s, which are represented by (x, y) point locations.
We shall quickly review how this optimization is performed with the classical AAM inverse compositional mapping approach, then replace the standard approach with our identity and expression factorization method and see the impact on the optimization procedure. In the standard AAM, p j is the vector of weights for a set of eigen-shapes. The shape s is parameterized by a PCA model such that each s j is explained by the mean shape s 0 and a weighted combination of eigen-vectors P s with weight parameters p j , such that s j = s 0 + P s p j .
The inverse compositional approach is to optimize this objective by iteratively building up a series of warps to recover an optimal warp W (x, p j ). Somewhat confusingly, both the template, T and the image I are warped. T is always warped from its initial shape s 0 . I, however, is warped to the current estimate I(W (x, p j )). Once the optimal warp at the iteration is determined, ∆p j , the current warp parameters p j and ∆p j are "composed" in order to update the current warp parameters. When ∆p j is close to zeros, or if the warp is not changing much, the algorithm has converged. The current warp is formed by inverting the parameters of ∆p j (by negating) and applying the warp defined by these updated parameters to a warp composed of all previous warps. The modified objective at each iteration is then Using a first order Taylor expansion, we have So that ∆p j can be given as for an individual p j , where H, the Hessian, can be written as The algorithm consists of pre-computing ∇T , ∂W ∂v j , H v , H w and inverses at the mean shape for efficiency. These are used along with Eq. (27) to compute the change at (p j = 0) required to minimize the error between the template and the current warped target. To adapt this algorithm for our purposes requires few changes.

A Novel Identity-Expression AAM (IE-AAM) Formulation
In the AAM inverse compositional method, one must determine W (x, p k ) = To compute the parameters of W (x, p k ) one then computes the corresponding changes to the current mesh vertex locations ∆s = (∆x 1 , ∆y 1 , . . . , ∆x v , ∆y v ) T , which are computed by composing the current warp.
The new parameters at time k are then given by p k j = P −1 s (ŝ j ), whereŝ j = s k−1 j + ∆s j − s 0 . Now, in the case of our identity and expression decomposition model, we replace the PCA model with a factorized model with the form s j = s 0 + Fw + Gv j . The parameters are the vectors w and v j belonging to the identity and the j th expression of that identity. Note that compared to the notation used in Section 3.1, we have omitted the index i since we are dealing with the same identity. In our identity and appearance decomposition approach, we now have an objective of the form, Defining z j = [w T , v T j ] T , Taylor series expansion yields where ∆z j = [∆w T , ∆v T j ] T . As identity and expression are assumed to be independent, this can be computed separately, We then compute the warp composition as in [34], However, computing the parameters w k and v k j from ∆s j is no longer a simple matrix multiplication, as was the case when we updated p j above, because the loading matrices of the identity expression model F and G are not orthonormal and we use the sequence of images. Therefore, we use the expectation of Eq. (23).
That is, after computing ∆s j for each warp, let Then, with Ψ constructed as a diagonal matrix with Σ along the diagonal repeated J times, A i as defined in Eq. (7), and Φ constructed as a diagonal matrix, by merging the diagonal of λI with J blocks of diagonal of ρI we can get with Φ and A being equal to Φ = diag (λ 1 , . . . , λ K , ρ 1 1 , . . . , ρ 1 L , . . . , ρ J 1 , . . . , ρ J L ) , from which we can then compute the current warp W (x, w k , v k j ). Texture variation is handled by alternating between training the model for texture variation and estimating the texture independently. Alternating between the two parameter optimization yielded the best results, although more efficient methods can be applied.

Keypoint Localization Experiments with CK+ and IE-AAMs
Since the CK+ dataset is also supplied with facial landmark annotations, we experimented using the Active Appearance Model extension described in previous sections. We used the same training set as described in the emotion recognition In order to achieve good results using this package, however, we used some pre-processing which improved ICAAM results: we first applied the OpenCV Viola-Jones frontal face-detector to remove large-scale pose variation and then aligned the images using a Procrustes analysis. The variation after Procustes alignment was modeled using PCA with 3 components and added to the shape projection matrix in ICAAM in order to model pose. In our case, we added the pose parameters to the non-identity component matrix G and then ortho-normalization was applied, which follows Lucey et al. among others [32]. ICAAM was set to use the default 98% of shape variation, while our method used 200 components for identity and 50 for expression. Test images were initialized using the face-detector used for training.  The results are summarized in Figure 4. In 4a we show the total cumulative distribution of error in all tested points for four methods. This analysis is widely used to evaluate keypoint techniques [34]. can be a good practical strategy for fitting an AAM with a small amount of labeling effort. The subject-specific AAM uses the project-out approach, [34] and the inverse-compositional method. As expected, this method performs well, with less than 22.9% of predicted points being greater than 5% of the inter-ocular distance (calculated as the two farthest points on the eyes) from the ground truth.
We also compute the average inter-ocular distance error for each subject. In 4b, we show the cumulative distribution of average subject error, which evaluates how well methods perform within each subject trial. The subject-trained AAM suffers from convergence errors, i.e. dramatic failures where all or many keypoints completely fall off the face, which cause the average error for each subject to increase. In fact, the average error of all points for the subject-trained AAM is 23.3% of inter-ocular distance, while the average error of our method is 5.94%.
Our method uses joint optimization on a sequence of frames adding stability to the point localization. Because of this, our method does not have these kinds of convergence errors, as shown in Figure 4b, Other methods, such as the one of Van der Maaten and Hendriks report 4.69 pixel error using mixture modeling on the CK+ data [47]. They also report 0.23% convergence error rate at that level of accuracy. More crucially, however, test subjects in their testing regimen were not strictly prohibited from the training set.
We are not aware of any AAM experiments using a large number of disparate subjects and testing on left-out subjects that report lower error. In any case, we are motivated more by the possibility of improving methods by using identity information, and believe that identity-normalization might easily be applied to any AAM-method. However, we do note that we see 0% convergence error, which we believe is due to the inference over multiple images of the same identity.
These results strongly suggest that identity is an important source of information and there are measurable benefits to modeling such information explicitly.
The average error for the CK+ set using the ICAAM code with face detection initialization is 24.36, with a standard deviation of 11.10, whereas our model yields an average pixel error of 7.15, with a standard deviation of 6.57. Both the bias and the variance is minimized simply by extricating the source of variation. The change to the algorithm, as discussed in previous sections, is relatively small. The result is significantly improved.

Identity-Expression Constrained Local Models (IE-CLMs)
The term Constrained Local Models (CLMs) has evolved from the original work of Cristinacce and Cootes [11], which can be viewed as a particular instance of a CLM [39]. Nowadays the term CLM has come to refer to a number of methods which involve finding the landmarks of an image I through assigning a cost Q to candidate landmark positions {x 1 , ..., x n } ∈ x on the image and the parameters of the model p. The corresponding objective function can be written as: where D i encodes the image dependent suitability measure for the i th landmark being located at position x i in the image. The R term can be interpreted as a regularization term that encodes preferences for certain spatial configurations of landmark positions 1 . This set-up leads to what some refer to as a deformable model fitting problem. Further, it is common to use a linear approximation for how the shape of non-rigid objects deform and a common variant of such a modeling technique is known as a point distribution model or PDM [10]. The PDM of [10] models non-rigid shape variations linearly and composes them with a global rigid transformation such that the 2D location of the PDM's i th landmark is given by: where the PDM parameters are defined by p = {s, R, t, q}. These parameters consist of a global scale s, rotation R, and translation t (forming a similarity transformation), as well as a set of parameters encoded in vector q which capture the global non-rigid deformations though a sub-matrix Φ i of a larger matrix Φ capturing variations using basis vectors. Using a probabilistic notation, the probability over the landmark position x i can be represented as a normal distribution with mean sR(µ i + Φ i q) + t and a covariance of σ 2 , such that Consequently, the distribution over the whole set of landmarks x gets the form of a normal distribution which corresponds to where µ and Φ represent respectively a concatenation of the terms µ i and Φ i for i ∈ {1, . . . , n}. The Covariance σ 2 I is a diagonal matrix having the values in its diagonal being repeated n times. Eq. (39) gives a principled probabilistic representation for the regularization term R in Eq. (36). It is common to place a uniform prior on the parameters of the similarity transformation, while the model for non-rigid motion is typically represented using principal component analysis.

A Deeper Probabilistic View of PDMs
Note that in this formulation the variable z represents the set of all landmarks in a transformed space where they are represented in a standardized coordinate space.
That is different from their representation in x where each landmark is undergone an affine transformation and therefore can be observed in more volatile regions image. This difference guides us to a better representation in the z space to formulate the distribution of the key-points. Therefore, we transform the landmarks in each image from the x space to z space through a set of affine transformation, whose parameters are trained to be robust against translation, rotation, and scaling transformations. From this point forward, we discuss our model over the landmarks in the z space.

Probabilistic PCA based CLMs
Some of our previous work [24] has formulated a CLM in the following way; for a given image I, we wish to combine the outputs of the local classifiers for the keypoints with a spatial model of global keypoint configurations. The local classifiers acts as discriminative predictors for the keypoints, while the spatial model provides a prior on how the keypoints are distributed.
where z = [z 1 , z 2 , · · · z n ] T gives the coordinates of n candidate locations for the keypoints whose energy is measured. The enegy function E is composed of two where U p is a matrix of the p principle eingenvectors of the keypoints in z space, Λ p is a diagonal matrix whose diagonal is composed of eigenvalues λ 1 , . . . , λ p , knowing that eigenvalue λ j corresponds to the eigenvector in column j of the matrix U p , and R is an arbitrary orthogonal matrix. The term σ 2 is the average of the remaining eigenvalues which explain the least significant direction of variation in data. Note that the first and second terms in Eq. (42) correspond respectively to the first and second terms of Eq. (36).
To minimize E we perform a search over the candidate keypoint locations z.
This is done by iterating over the keypoints, where in each iteration all key-points are fixed except one. The optimum value for that keypoint is found through a comprehnsive search over the energy of the local response map grid locations.
The iterations continue until all keypoints converge.

Identity-Expression Factorized CLMs
Once the parameters of the identity expression model are learned using the EM procedure, as explained in Section (3.1), this model is used in conjunction with the local response classifiers derived from the previously trained SVMs. Combining the energy of the local classifier with the energy term of the Identity Expression Factorized model, we minimize the energy of the following function: where w represents a set of candidate keypoints in all test images belonging to the same identity whose energy is measured. Note that as in the previous section

Keypoint Localization Experiments with the MultiPIE Dataset and IE-CLMs
The CMU MultiPie face dataset [23] captures a subset of expressions, such as surprise, smile, neutral, squint, disgust, and scream, of 337 identities. The images are taken in four sessions and in each session a subset of expressions is captured.
Therefore, not all expressions are registered per identity. The availability of multiple expressions per identity provides the right dataset to evaluate our identity expression model. We perform the same experiment as in [56] and compare our model with other models on the frontal face images with 68 keypoints. Following the same split of the dataset as in [56], the first 300 images are used for training and the next 300 images are used for testing. Figure 7 shows the main direc-   This level of performance is on par with the state of the art methods of [56] and [24]. Figures 8 and 9 illustrate our model's performance on different expressions of two sample identities in the test-set. Our model exploits the fact that all these images have the same identity and thereupon, the same identity representation is used when the model searches for the optimal keypoints. Figure 6 shows some sample keypoint localization by our model on different identities in the test set, which includes data of different expression, ethnicity, age, gender, illumination, and face masks such as trimming style and eye-glasses usage.
In terms of running time, the PPCA-CLM model takes 32 seconds for the local response image generation and 6 seconds for the optimization procedure.
However, we have use none of the obvious acceleration techniques that could be applied to speed the local response map generation procedure, such as GPU accel-

Conclusions and Future Work
We have shown that identity-expression disentanglement and various forms of identity normalization can be used to improve the performance of supervised learning approaches to performance-driven animation and expression recognition tasks. Detailed facial expression analysis techniques are often limited by the need for subject-specific models, which limits the quantity of labeled data that can be brought to bear. In our work here, we have detailed how approaches that work well for a single subject can be adapted and their performance improved for unseen identities. Our approach is based on disentangling identity and expression with data that was been weakly labeled according to identity. The increase in performance across many tasks suggests that using simpler tools for both performancedriven animation and automatic expression recognition may be sufficient if identity labels can be used as an additional source of information.
In future work it might be interesting to extend this type of approach to be able to deal with more complex data, including data exhibiting greater pose variation. For example an explicit 3D shape model such as the formulation proposed for CLMs in the work of Jeni et al. [28] could be combined with the identityexpression factorization approach used here.
We have shown how the identity-expression factorization approach can be integrated into a CLM for keypoint localization. We believe there are a number of possible directions that could lead to further improvements. Firstly, we believe that replacing the binary SVM classifier with a multi-class SVM might yield increased performance with minimal changes required to the underlying model formulation. However, once formulated in this way, it would also be more straightforward to train the model to make structured predictions using a fully discriminative training procedure, for example, along the lines of [56]. Another direction is to replace the Iterated Conditional Modes (ICM) energy minimization step with a temperature based annealing procedure. The ICM algorithm is known to be quite fast at finding local minima, but it may be the case that better solutions could be found with a stochastic annealing procedure.
The use of multi-scale intensity response information in conjunction with alternative optimization techniques could be a promising direction for future research. It is also likely possible to speed up response image generation and energy minimization when multiple images are processed per identity through various means.
Finally, the local SVM classifiers and hand engineered features that we have used here could be replaced with a convolutional neural network (CNN), such as the architecture proposed in [25]. A CNN approach was not feasible in the context of our primary goal of animation control at the time this work was performed since CNNs typically require a considerable amount of training data to avoid overfitting and our face camera dataset is quite small compared to the data sets that have been used where CNNs have yielded state of the art results. However, for other application settings and datasets research combining identity-expression disentanglement techniques with CNNs may be a promising direction of future research.    Table 4: Comparison of confusion matrices of emotion detection for the combined landmark (SPTS) and shape-normalised image (CAPP) features before and after identity normalisation. The average accuracy for all predicted emotions using the state of the art method in [32] is 83.27% (top table), using our method yields 95.21% (bottom table), a substantial improvement of 11.9%.