{'fraction_non_alphanumeric': 0.036602226696097216, 'fraction_numerical': 0.011885414898003736, 'mean_word_length': 4.817412163115564, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Mutual Information Regularized Identity-aware Facial ExpressionRecognition in Compressed Video

How to extract effective expression representations that invariant to the identity-specific attributes is a long-lasting problem for facial expression recognition (FER). Most of the previous methods process the RGB images of a sequence, while we argue that the off-the-shelf and valuable expression-related muscle movement is already embedded in the compression format. In this paper, we target to explore the inter-subject variations eliminated facial expression representation in the compressed video domain. In the up to two orders of magnitude compressed domain, we can explicitly infer the expression from the residual frames and possibly extract identity factors from the I frame with a pre-trained face recognition network. By enforcing the marginal independence of them, the expression feature is expected to be purer for the expression and be robust to identity shifts. Specifically, we propose a novel collaborative min-min game for mutual information (MI) minimization in latent space. We do not need the identity label or multiple expression samples from the same person for identity elimination. Moreover, when the apex frame is annotated in the dataset, the complementary constraint can be further added to regularize the feature-level game. In testing, only the compressed residual frames are required to achieve expression prediction. Our solution can achieve comparable or better performance than the recent decoded image-based methods on the typical FER benchmarks with about 3 times faster inference.


Introduction
The video modality is increasingly important in many computer vision applications [1,2]. Considering the natural dynamic property of the human face expression [3], many works propose to explore spatio-temporal features of facial expression recognition (FER) from the videos. Recently, deep neural networks (DNN) have achieved significant progress for image-based facial expression recognition [4], while the processing of expression video is still challenging.
Although the multi-frame series can inherit richer information and the temporal-correlation between consecutive frames can typically be useful for FER, the video also introduced a lot of redundancy. The signal-to-noise ratio (SNR) in FER videos is exceptionally low due to the slight muscle activity [5,4].
The typical spatio-temporal FER DNNs explore the spatial and temporal clues in a series of frames to extract the facial expression-related information [5]. The recursive neural network (RNN), 3D convolutional, and non-local networks are the most frequently used network backbones [6,7]. Unfortunately, using DNN to process many consecutive frames can be computationally costly. It can be hard to scalable for the long videos [8,4]. Furthermore, for some poral dependence. This suggests the temporal cues can be hard to explore in the low SNR FER videos with the conventional solutions [9,10].
We argue that the compressed domain can be suitable for the FER task for four reasons. 1) the consecutive frames in video modality have many uninformative and repeating patterns, which may drown the "interesting" and "true" signal [11]. With the standard video compression algorithms, the compression ratio can usually be hundred times [12]. Manipulating on the compressed domain can significantly reduce the cost of computation and memory. 2) the typical compression methods (e.g., MPEG-4, H.264, and HEVC) break the video to the I frame (intracoded frames) with the first image, and follows several P frames (predictive frames) which encoded as the "change" or "movement" [13], as shown in Figure. 1. The fundamental of expression is the action of the face muscle.. Many FER systems, in reality, are based on the action unit framework [14].
As a result, compressed P frames will inherit off-theshelf but useful expression-related factors, and their pattern is substantially simpler than raw images. 3) our compressed domain exploration can also be effective since it focuses on the "true" signals rather than processing the repeatedly nearduplicates [15]. 4) Since the to-be-processed data is transmitted in compressed format, the decoding procedure is not needed in the real-world mission. Furthermore, the FER task has long suffered from the high inter-subject variation caused by identity discrepancies in facial attributes [16,8]. The learned features may capture more identity-related information than expression-related information, and are not purely related to the FER task. Noticing that the P frames may contain the relative location of face key points, which can be related to the identity [17]. Metric learning is a standard approach for extracting identity from the expression representation [16,8,4]. Inspired by the adversarial disentanglement, [18,19] propose to render the identity removed face, which is inspired by adversarial disentanglement (GAN). These researchs, on the other hand, concentrate on image-based FER. [3] extend the metric learning [8] for video data by explicitly substitute the image with the video features, without taking into account the video's characteristics. Furthermore, these methods necessitate the use of the identity label and multiple expressions of the same person, which significantly restricts their applicability to the in-the-wild FER task [20].
In this paper, we target to exploit the identity information from the I frame using a pre-trained face recognition network, e.g., FaceNet [21]. Their identity embeddings are remarkably reliable, since they achieve high accuracy over millions of identities [22], and robust to a broad range of nuisance factors such as expression, pose, illumination and occlusion variations.
Using the identity feature as the anchor, we can explicitly enforce the marginal independence of our identity and expression feature. Instead of the complicated adversarial training [23] for disentanglement, we adopt the mutual information (MI) as the statistical measure of the independence of these two representations [24]. The MI of two random variables can usually be intractable to directly and precisely measure in a high-dimensional space [25]. Recently, some of the works illustrate that mutual information can be differentiable approximated [26]. We propose to minimize the differentiable MI measure as the objective. Practically, it can be a latent space min-min game of an encoder-discriminator framework, which follows a collaborative fashion rather than adversarial competition. We note that GAN is notorious for its unstable model collapse [27], while our MI regularization is concise and efficient.
This work extends our previous work [28] in the following significant ways: • A novel expression and identity disentanglement framework based on practical mutual information minimization, which follows a min-min game with the joint and marginal distribution sampling.
• We demonstrate the generality of our framework in more FER dataset, i.e., MMI, carry out all experiments using the novel MIC framework.
• Moreover, the systematical cross-dataset evaluation, sensitivity analysis, and identity feature extraction analysis are provided.
The main contributions of this paper are summarized as follows: • We propose to inference expression from the residual frames, which explores the off-the-shelf yet valuable expression related muscle movement in the up to two orders of magnitude compressed domain.
• Targeting for the identity-aware video-based FER, the independence of expression and identity representations from P frames and I frame are enforced with the differentiable MI measure.
• The separability of expression and identity representations is maximized by a min-min game with the joint and marginal distribution sampling, which does not rely on the unstable adversarial game to achieve identity elimination.
We evidenced its effectiveness on several video-based FER benchmarks with much faster inference. The promising performance evidenced its generality and scalability.

Related Works
Video-based FER has been thoroughly researched, since facial expression is a natural and universal means for human communication [5,29]. Considering that the expression is essentially a dynamic action which should take minute muscle movements through time into account [30]. Traditionally, the handcrafted features are utilized to represent the spatio-temporal cues and for FER. Frame aggregation and spatiotemporal FER networks are being developed in parallel with the exponential growth of deep learning. Frame aggregation approaches may make use of image-based FER networks by conducting frame-wise aggregation at the decisionlevel [9] or feature-level [10].
The essential temporal correlation, on the other hand, is not investigated. Instead, the spatio-temporal FER networks use sequential frames to utilize both the spatial/textural and temporal information [31]. Furthermore, cascaded networks suggest integrating CNN-learned perceptual vision representations with RNNs for variable-length video [32,33]. Moreover, by using a non-local network for video processing, the number of potential connections and the corresponding computation costs grow exponentially to the number of frames. The 3D convolutional has shared kernel-weights along the time axis, which has also been widely used for video-based FER [34,35].
However, recent studies have only looked at the image domain, and the spatiotemporal FER does not outperform aggregation methods substantially [5]. To the best of our knowledge, this is the first effort to investigate the compressed video FER, which is orthogonal to these advantages and can be conveniently added to each other. Video compression convert the digital video into a specific format that is suitable for recording and distribution of this video [12]. Conventionally, the H.264/MPEG-4 and the Advanced Video Coding are the typical algorithms [11]. Typically, the video sequence is divided into several Group Of Pictures (GOP) by the video codecs. In a GOP, there is an I frame and follow by several P frames. Specifically, the I frame is a self-contained RGB frame with full visual representation, while the P frame is the inter frames that hold motion vectors and residuals w.r.t. the previous frame [13]. The motion vector can be used as an alternative to the optical flow [36,37], which needs to decode the RGB images.
The recent action recognition method [15] proposes to aggregate I frame, residuals, and motion vectors in the compressed domain, without the RGB image decoding. Although FER shares some similarities with action recognition, the movement range and use of I frame can be vastly different. In action recognition [15], the I frame is directly used to predict the action and combine with the result of P frames. In contrast, the I frame in FER usually be a neutral face (different from the video label). Furthermore, the low-resolution motion vector can not well encode the expression.
Mutual-information has a long history in unsupervised learning. The infomax principle [25], as prescribed for neural networks, advocates maximizing the MI between network input and output. This can be the fundamental of many ICA algorithms, which can be nonlinear [38] but are often hard to adapt for use with deep networks. Recently, some of the works proposed to achieve unsupervised learning with MI. [39] proposes a generative adversarial network to minimize MI with positive and negative samples for Independent Component Analysis (ICA). It introduces a strategy to draw samples from the joint distribution and the product of marginal distributions and proposed to train an encoder and a discriminator to minimize the Jansen-Shannon divergence. Moreover, it has recently been shown that the GAN framework can be extended not only to maximize or minimize MI but also to explicitly compute it using the Mutual Information Neural Estimation (MINE) proposed in [26]. In [40], the DeepInfo-Max (DIM) is proposed to learn the representations based on both local and global information. In [41], Deep Graph Infomax (DGI) extends this approach to graph-structured data. Inspired by these works, we are targeting to utilize the MI as the justified measure of independence, and minimize it directly as our disentanglement objective. Eliminating identity can benefit to extract more "pure" expression feature [16,8]. We note that previous identity-aware FER methods usually explicitly require the identity labels of FER datasets to sample the triplets [16,8,3], while the identity label is not common in FER. In contrast, our solution does not relies on the identity label of FER samples, but utilizes the easily available face recognition dataset.
The typical solution for FER is metric learning. Our MI regularization is also related to the triplet loss [21], which maximizes the Euclidean or cosine distance between two identities. With the development of GAN, adversarial training also can be utilized for disentanglement [23,24]. Instead, we consider the mutual information to be a more meaningful divergence to capture complex non-linear relationships, between the identity and expression representations. Besides, the identity label is required in these methods [16,8,3]. We can also choose adversarial training [23,24] as a baseline to achieve identity elimination in our framework. We note that the adversarial game is notorious for its unstable model collapse [27], while our solution follows a collaborative way.
Several adversarial disentanglement works demonstrate that simply separate the input may result in the extracted feature has no meaningful information [24,23,42,43]. The reconstruction of input can explicitly enforce the disentan-gled factors to be complementary to each other. However, reconstructing the video can be hugely underconstrained.

Methodology
We propose to develop an efficient video-based FER framework that operates directly on the compressed domain. The overall framework is shown in Figure. 2, which is consisted of four core modules. The pre-trained identity branch and FER branch (frame embedding network , aggregation module, and Classifier) work on the I frame and undecoded P frames, respectively. The dependence of identity and expression representations is then measured using a differentiable mutual information regularization module. Furthermore, when the apex frame is annotated, the complementary constraint can be applied to stabilize the early stage training.

Modeling Compressed Representations
To illustrate the format of the input video, we choose the MPEG-4 as an example [12]. The compressed domain has two typical frames, i.e., I frame and P frames. Specifically, the I frame ∈ ℝ ℎ× ×3 is a complete RGB image. We use ℎ and to denote its height and width, respectively. Besides, the P frame at time ∈ ℝ ℎ× ×3 can be reconstructed with the stored offsets, called residual errors Δ ∈ ℝ ℎ× ×3 and motion vectors  ∈ ℝ ℎ× ×2 .
Noticing that the motion vectors  has a much lower resolution, since their values within the same macroblock are identical. Considering the micro-movements of facial expression in each frame, the low resolution  usually not helpful for the FER. For P frame reconstruction = −1 − + Δ , where index all the pixels and 0 = . Then,  and Δ are processed by discrete cosine transform and entropyencoded.
The majority of compression algorithms are programmed primarily to minimize file size, and the encoded format can vary greatly from RGB images in terms of statistical and structural properties. As a consequence, a specially built processing network is needed to manage the compressed format. Considering the structure of residual images Δ are much simpler than the decoded images, it is possible to utilize simpler and faster CNNs ∶ ℝ ℎ× ×3 → ℝ 512 to extract the feature of each frame [6,44,1]. Practically, we follow the CNN in the typical CNN-LSTM FER structure [44,6,1], but with fewer layers to explore the information in Δ . Noticing that is shared for all frames, and only needs to store one in processing. Besides, most existing action recognition methods with compressed video [15] independently concatenate the paired Δ and  at each time step and predict an action score of each P-frame. The temporal cues and their development patterns are important for the FER task [5]. We simply choose the LSTM in [44] to model the sequential development of residual frames and summarize the information to an expression feature . Since our LSTM is applied to 512-dim features, the computation burden is largely smaller than work on the raw images. Noticing that more advanced RNN, 3D ( ,̂ , ) ).
For the I frame with raw image format, we simply use the FaceNet [21] pre-trained on millions of identities [22] as our identity feature extractor ∶ ℝ ℎ× ×3 → , where ∈ ℝ 1024 denotes the identity feature. We note that and do not need to have the same dimension for MI Regularization. Several datasets' FER videos begin with a neutral expression, which may help with identity recognition.

MI regularization
To eliminate the identity-related factors in our FER representation, we propose to utilize the identity feature from pre-trained face recognizer as anchor, and explicitly inspect the information w.r.t.
in . Achieving the disentanglement of different factors requires two major objectives, i.e., 1) each factor has its specific information, and 2) does not incorporate the information of the other factors [23,24]. For example, can achieve 1) using the conventional C.E. loss minimization w.r.t. the expression label , and is from the pre-trained identity extractor, which inherently has identity information. However, how to explicitly measure the dependency between these factors and minimize this metric to achieve the latter objective can be challenging.
Actually, the mutual information (MI) is the exact metric to measure the amount of information obtained about one random variable through observing another random variable.
where and are the random variables follows the distribution  and  respectively. ℙ indicates the joint prob- ability distribution of ( , ), ℙ = ∫  ℙ and ℙ = ∫  ℙ are the marginals. ℙ ⊗ ℙ is the product of the marginals.
MI minimization explicitly enforces the joint distribution to be equal to the product of marginals, which leads to the statistical independence of two vectors. Instead, the MI maximization can result in two vectors have the same information, and the MI is simply equal to the entropy of a variable.
We propose to utilize the mutual information neural estimator (MINE) [26] to provide the unbiased estimation of MI on independent and identically distributed (i.i.d.) samples. It is linearly scalable w.r.t. dimensionality and sample size, by leveraging a gradient descent over neural network ∶  ×  → ℝ. MINE proposes to approximate MI by exploiting a lower bound based on the Donsker-Varadhan representation of the Kullback-Leibler divergence. Therefore, the neural information measure can be formulated as Given the distribution ℙ,P denotes the empirical distribution associated to i.i.d. samples. Since the supremum is taken over all functions of , the two expectations are finite. Then, the MI in Eq. (2) can be estimated as follows: Besides, we leverage Monte-Carlo integration to avoid computing the integrals to compute ( ; ) as wherê is sampled from the marginal distribution. Note that ( , ) are sampled from the joint distribution ℙ . Then, we can evaluate the bias corrected gradients (e.g., moving average) [26].
The estimated ( ; ) is used as the supervision to update the FER branch. By utilizing MI regularization, the adversarial discriminator [46,23,24] is no longer needed in our new framework, which makes the balance of each module easier. Note that we need the additional neural network to measure the MI, but it is collaboratively trained with the FER branch to maximize the discrepancy between the two features. Essentially, we are playing a min-min game instead of a min-max game. Therefore, it is easier to stabilize the training (compared to adversarial training). Besides, MI is a symmetric measure, while the conditional entropy ( | ) = ( ) − ( ; ) optimized in conventional disentanglement works [23,24] is asymmetric and essentially we should calculate both ( | ) and ( | ) as supervision signal [23,24].
To maximize the discrepancy of and , we can also apply the adversarial disentanglement solutions [23,24]. Nevertheless, with the above-mentioned limitations, such methods can be hard to optimize and lead to inferior performance.

Complementary constraint
Many FER datasets follow a well-defined collection protocol, which usually starts from the neutral face and then develops to an expression. Specifically, the video in CK+ [47,14] consists of a sequence that shifts from the neutral expression to an apex facial expression. The last frame usually is the apex frame, which has the most strong expression intensity. Actually, the image-based FER methods select the last three frames to construct their training and testing datasets. Similarly, in MMI [48], the video frames usually start from the neutral face and develop to the apex around the middle of the video, and returning back to the neutral at the end of the video. Noticing that the apex frame (i.e., last frame in CK+ or middle frame in MMI) can clearly incorporate both the identity and expression information. Therefore, we are possible to utilize the apex frame as a reference of reconstruction, and simply apply the  2 loss.
wherê = ( , ). The complementary restriction is not necessary for our system since the FER loss is heavily weighted in the FER branch. It requires to maintain sufficient information w.r.t. expression and not easy to have nothing meaningful. However, the complementary constraint does helpful for the convergence in the early stage. When the apex is annotated, we only need to decode the apex frame in the decoded image domain at the start of a few training epochs.

Overall objectives
We have three to be minimized objectives, i.e., crossentropy loss, mutual information and  1 loss, which works collaboratively to update each module. The expression classification is the main task of the FER. We choose the typical cross-entropy loss  = − ∑ =1 log( ( ) ) to ensure contains sufficient expression information and finally have a good performance on -class expression classification. We use and ( ) indicate the ℎ class probability of the label and classifier softmax predictions respectively. Since the FER branch can be updated with all of the losses, we assign the balance parameter ∈ [0, 1] and ∈ [0, 1] to mutual information and  1 loss minimization objectives, respectively. Specifically, we update our and LSTM modules with For the MI estimator , we update it with ( ; ) . Moreover, the decoder module is updated with  1 . The detailed training flow is shown in Algorithm 1. We note that only the FER branch, i.e., , LSTM and Cls, is used for testing.
Since we are using the neural network for mutual information estimation, it is scalable, flexible, and completely trainable via back-propagation. Moreover, the decoder for reconstruction is used for complementary constraints. In contrast, FLF [23,24] uses a discriminator and a decoder for adversarial disentanglement and complementary constraint. Considering the in MIC and discriminator in FLF [23,24] has a similar structure, there is no significant difference for the network complexity. The mutual information calculation with Eq. 4 has the complexity of ( ), where is the number of sampled data. The linear complexity can be simple for implementation. Conventionally, the fast computation of MI is limited to discrete variables [49]. For the continuous random variables, its complexity is quadratic to the number of samples, which is not desired for a loss function.

Description of the datasets
benchmark [5,45]. The video is collected in a restricted environment, in which the participate subjects are facing the recorder with an empty background. The video in CK+ consists of a sequence that shifts from the neutral expression to an apex facial expression. The last frame usually is the apex frame, which has the most strong expression intensity. The expression included in this dataset is anger, contempt, disgust, fear, happiness, sadness, and surprise. There are 327 facial expression videos collected from 118 subjects. Following the previous works, we use subject independent 10folds cross-validation [5,3]. Many FER datasets follow a well-defined collection protocol, which usually starts from the neutral face and develops to an expression. Specifically, the image-based FER methods select the last three frames to construct their training and testing datasets. In Fig. 3, we show the compressed and decoded frames in CK+ dataset. Similarly, in MMI [48], the video frames usually start from the neutral face and develop to the apex around the middle of the video, and returning back to the neutral at the end of the video. Noticing that the apex frame (i.e., last frame in CK+ or middle frame in MMI) can clearly incorporate both the identity and expression information. Therefore, we are possible to utilize the apex frame as a reference of reconstruction, and simply apply the  1 loss.   Table 2 Comparison of the identity eliminating on CK+ dataset w.r.t. the identity recognition accuracy using , and the mutual information between and .

MMI Dataset [48]
consist of a total of 326 facial expression videos from 32 participants. There are 213 labeled videos with the expression label angry, disgust, fear, happy, sad, and surprise. The video frames start from the neutral face. Then the expression is developed to the apex in the middle of video, and returning back to the neutral at the end of the video. In our experiments, we follow the previous works to use subject independent 10-folds cross-validation [5,3]. In Fig. 4, we show the compressed and decoded frames in the MMI dataset. AFEW Dataset [20] is more close to the uncontrolled realworld environment. It is consists of video clips of movies [56]. The video in AFEW has a spontaneous facial expression. The AFEW has seven expressions: anger, disgust, fear, happiness, sadness, surprise, and neutral. Following the evaluation protocol in EmotiW [57], there are training, validation, and testing sets. Since its testing label is not available, we follow the previous work to use the validation set for comparison [44]. Noticing that the validation set is not used in the training stage for the parameter or hyperparameter tuning. In Fig. 5, we show the compressed and decoded frames in the AFEW dataset.

Implementation details
We preprocess video frames and augment the data according to [44,6,1] for fair comparison. For these three datasets, the videos only have one GOP and do not need to segment the video. We utilize the Pytorch deep learning platform for our framework. In the training stage, on all datasets, we set the batch size to 48. All of the modules use the Adam optimizer with momentum 0.9, and a weight decay of 1e-5 for 100 training epochs. On the CK+ and MMI datasets, the learning rate is initialized to 1e-1, and be modified to 1e-2 for the 30 ℎ epoch. For the AFEW dataset, we initialize the learning rate to 1e-4, and modify it to 8e-6 for the 30 ℎ epoch and 1e-7 for the 60 ℎ epochs.
All of our training/testings use an NVIDIA Titan X GPU. We note that the calculation of the accumulated residuals to recover the apex frame is measured on Intel E5-2698 v4 CPUs, but we do not need this operation in testing. For the testing speed, we measure the average frame per second (fps) according to the average running time, which is the sum of the data pre-processing time and the FER branch forward pass time.
Practically, our , LSTM and Cls follow the CNN-LSTM structure in [6,44,1] for fair comparison. Considering the relatively simpler residual data, we use part of convolutional layer for (i.e., C1, C2 and F4 layers as in [6,44,1]) and a fully connected layer for Cls (i.e., ℝ 512 → ℝ 6 ℝ 7 ). FAN [7] shows the ResNet [58] can be a powerful backbone. To further demonstrate the generality of our framework, we use the first five convolutional layers (i.e., before the second residual block) and the first fully connected layer in ResNet18 as our feature extractor backbone. We denote this ResNet backbone as ResNet6.
For the CK+ and MMI dataset, we choose the last or middle frame as the apex reference image, respectively. Since the complementary constraint is only used to stabilize the initial training of disentanglement, we uniformly decrease from 1 to 0 until the 30 ℎ epoch. Practically, we use the grid search to find the optimal and set it to 0.1, 0.1, 0.2 on CK+, MMI, and AFEW datasets, respectively.

Evaluation and ablation study
Results on CK+ dataset.
The 10-fold cross-validation performance of our proposed method is shown in Table 1. For a fair comparison, the image-based experiment settings are not incorporated in the tables. Besides, only the state-of-the-art (SOTA) accuracy obtained by the single-models (non-ensemble model) is listed.
Based on the mode variational LSTM [44], our proposed MIC achieves the SOTA result without the facial landmarks, 3D face models, or optical flow. It worth noticing that the much simpler CNN encoding network makes more residual frames that can be processed parallel than [44]. Moreover, our efficiency also benefits from avoiding the decompress of  Table 3 Experimental results on MMI dataset. Note that in order to make the comparison fair, we do not consider image-based and 3D geometry based experiment setting and models [8,16,4].
the video. Since the videos are typically stored and transmitted with the compressed version, and the residuals are off-the-shelf. As a result, the proposed compressed domain MIC can speed up the testing about 3 times over [44], and achieve better accuracy. Besides, [3] is a typical metric-learning-based identity removing method. Our solution can significantly outperform it with respect to both speed and accuracy. Actually, the sampling of tuplets usually makes the training not scalable [4], while our identity eliminating scheme is concise and effective.
When we remove some modules from our framework, the performances have different degrees of decline. We use -MI and -̂ to denote the MIC without MI regularizer or complementary constraint, respectively. The performance drop of MIC is significant when we remove the MI regularization module, which further evidenced that the identity can be a notorious factor for FER. The result also implies that the identity can be well encoded by the face recognition network and the disentanglement with mutual information is feasible. Compared with using the conventional adversarial training based disentanglement [23,24] as an alternative (i.e., MIC-MI+Adv), our MI regularizer is easy to train and can converge 1.8 times faster in training.
We can also follow the action recognition method [60] to concatenate the motion vector and residual as the input, and denote as MIC+ . However, we do not achieve significant performance on all datasets, but the inference speed in testing can be slower. This may be related to the coarse resolution of the motion vector can not well describe the finegrain muscle movement of the face. More appealingly, the performance of our MIC can be further improved With the ResNet backbone. With the simplified 6-layer ResNet, MIC outperforms the FAN [7] w.r.t. both accuracy and processing speed. Aggregation-based methods do not explore the temporal cues and can be computational costly to compare all possible image pairs within a set [2]. We note that the ResNet18 used in FAN [7] is sequentially pre-trained on the additional MSCeleb-1M face recognition dataset and FER+ expression dataset.
The confusion matrix of our proposed MIC method on the CK+ is reported in Figure 7 (left). The accuracy for the expression of class happiness, disgust, anger, surprise, and contempt are almost perfect.
In Table 2, we investigate the identity eliminating performance. The first metric is following [23] to use recognize identity with . Besides, we can directly use mutual information as the metric of independence. We can see that the residual frame itself can incorporate much less identity information than the decoded images as in [44], while it is still possible to detect identity with the facial contours. The MI regularization can explicitly remove the identity factors and outperforms the adversarial training [23].

Results on MMI dataset.
The evaluation results on the MMI dataset are shown in Table 3. The performance is also consistent with the CK+ dataset, which evidenced its effectiveness and generality. All of our methods achieve comparable performance to the landmarkbased STOA methods. It is more promising that MIC can be significantly better than the methods without the landmarks. Our MIC is efficient, since we explore the correlation in video frames in the compressed domain.
In Figure 6, we give a comparison of accuracy w.r.t. each emotion among five MIC baselines, and the confusion matrix of our MIC is reported in Figure 7 (middle). There is a good performance for the expression class of fear, happiness, sadness, and surprise. In contrast, the accuracy of expression class anger and disgust is relatively limited. Especially, there is a high degree of confusion between anger and disguise. This may be related to the subtle movements between these expressions are relatively in the residual frames.

Results on AFEW dataset.
The evaluation of the proposed MIC on the AFEW dataset is shown in Table 4. We note that only the SOTA accuracy obtained by the single-models (non-ensemble model) are listed for a fair comparison. Besides, the audio modality in AFEW can be used to boost the recognition performance [61,62,63]. We note that we only focus on the image compression in this paper, and the audio/video data are stored in separate tracks, but the additional modality can also potentially to be added on our framework following the multimodal methods [61].
With the simplified mode variational LSTM-based [44] Table 4 Experimental results on AFEW dataset. *Additional FER+ dataset is used. † Optical flow is used. † †Additional body language dataset is used.
backbone, the exploration in the compressed domain can achieve comparable or even better recognition performance. More promisingly, our MIC can also achieve real-time processing for the uncontrolled environment, which evidenced its generality. We note that the typical time resolution in FER is 24fps [6].
In addition, we note that the complementary constraint requires the apex frame in training, which is not applicable for the AFEW dataset. We do not apply the reconstruction loss in the AFEW task. Although the requirement of the apex frame imposes some limitations on the training, it does not affect the generality of the testing or implementation of the trained model with CK+ and MMI.
Some of the works propose to improve the image-based FER networks and combine the frame-wise scores for videobased FER [64,7]. The image-based FER methods [64] achieves high performance, but [64] uses a very deep network DenseNet-161 and pretrains it on the private Situ dataset. Moreover, [64] utilize the sophisticated post-processing. Actually, an intuition of a statistic-based solution is to avoid LSTM and speed up the processing. However, with the super deep and complicated structure, their processing can be much slower than our solution. [62]     cues of the videos. Moreover, [52,51,65] also propose to modify the LSTM model for the spatial-temporal modeling. However, all of the above solutions are applied to the decoded space, which requires decoding processing and needs to handle much more complicated data. With the 6-layer ResNet backbone as an expression feature extractor, the performance of our MIC model can be further improved without an additional FER dataset for pre-training. Overall, the proposed MIC can improve the testing speed by a large margin and can achieve the SOTA accuracy as the previous models.   Table 9 Facial expression recognition with the extracted identity feature from I frame.

Identity feature extraction
We adopt the pre-trained face recognizer FaceNet [21] to extract the identity factor from the I frame. The feature embedded with the convolutional layers and the first fully connected layer can be robust to a broad range of nuisance factors such as expression, pose, illumination, and occlusion variations, since it achieves high accuracy over millions of identities [22]. To check the expression information in the extracted identity feature , we use the feature for expression classification as [23]. The results are shown in 9. We note that achieve zero FER accuracy with does not mean has no information about expression. Instead, approaching the chance probability, i.e., uniform distribution w.r.t. expression classes, indicates the expression is well disentangled from the identity feature with FaceNet.

Sensitivity analysis
We use and to balance the MI and complementary constraint terms and choose the best value with grid searching. In Tab. 10, we provide the sensitivity analysis of using different in three datasets. We can see that we can achieve the best performance on CK+ and MMI with = 0.1. For the AFEW dataset, the performance is relatively stable for from 0.15 to 0.25. We simply use 0.2 for all of our MIC models on the AFEW dataset.
is used to balance the complementary constraint term, which can be helpful for stabilizing the training. In Tab. 11, both the fixed and linear changing are compared. Decreasing from 1 to 0 for 30 or 50 epochs can usually achieve the best performance.

Cross-database validation
Since the subjects are different across CK+ and MMI datasets, the cross-dataset evaluation can be used to evidence if the trained model is affected by identity [16]. In Tab. 5, the FER model trained CK+ training sets in the previous experiment is directly implemented to the MMI testing set. We can see that our MIC can outperform the other methods with the same backbone. In Tab. 6, we use MMI as training data and test on CK+ dataset. The proposed MIC outperforms the other methods with the same backbone consistently.
We note that FAN [7] uses ResNet18 as a feature extractor and sequentially pre-trained on MSCeleb-1M face recognition dataset and FER+ expression dataset.
In Tab. 7 and Tab. 8, we use the training set of AFEW for training and test on MMI and CK+, respectively. Considering the large domain shift between AFEW and MMI/CK+, there is a significant performance drop. We note that the proposed method can also achieve better performance than its backbones [44,7].

Critical discussion and future work
The proposed MIC framework has demonstrated its effectiveness w.r.t. accuracy and testing speed. However, MIC highly relies on the separation of I and P frames in compressed video. Although MPEG-4, H.264, and HEVC formats are widely used, some of the compression solutions do not follow the motion compensation with I and P frames. For example, the YUV format compresses the video by considering the different changes of brightness and chromaticity. Moreover, frame loss can be common in real-world video transfer. How will the frame loss affect the FER performance is underexplored.
Identity can be the most challenging variation for FER [8], while the FER performance can also be affected by pose and illuminations. It can be promising to take the other variations into account.
The recent work [55] proposes to utilize the additional unlabeled face dataset to boost the performance, which can be a powerful means for alleviating the scale issue of videobased FER.
For the large domain gap, e.g., AFEW to MMI/CK+, the domain adaptation methods [67,68] can be used to achieve better cross-dataset performance.

Conclusion
In this paper, we target to explore the facial expression cues directly on the compressed video domain. We are motivated by our practical observation that facial muscle movements can be well encoded in the residual frames, which can be informative and free of cost. Besides, the video compression can reduce the repeating boring patterns in the videos, which rendering the representation to be robust. The increased relevance and reduced complexity or redundancy in FER videos make computation much more effective. We extract the identity and expression factor from the I frame and P frame, respectively, and explicitly enforce their independence with concise and effective mutual information regularization. When the apex frame label is available in training, the complementary constraint can further stabilize the training. In three video-based FER benchmarks, our MIC can improve the performance without the additional identity, face model, or facial landmarks labels. The processing speed of the test stage is promising for real-time FER. Moreover, our mutual information regularization can potentially   be a good alternative to adversarial training [23] for many disentanglement tasks.