{'fraction_non_alphanumeric': 0.030750968033690398, 'fraction_numerical': 0.011644668042169525, 'mean_word_length': 4.918701906992305, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MERANet: Facial Micro-Expression Recognition using 3D Residual Attention Network

Micro-expression has emerged as a promising modality in affective computing due to its high objectivity in emotion detection. Despite the higher recognition accuracy provided by the deep learning models, there are still significant scope for improvements in micro-expression recognition techniques. The presence of micro-expressions in small-local regions of the face, as well as the limited size of available databases, continue to limit the accuracy in recognizing micro-expressions. In this work, we propose a facial micro-expression recognition model using 3D residual attention network named MERANet to tackle such challenges. The proposed model takes advantage of spatial-temporal attention and channel attention together, to learn deeper fine-grained subtle features for classification of emotions. Further, the proposed model encompasses both spatial and temporal information simultaneously using the 3D kernels and residual connections. Moreover, the channel features and spatio-temporal features are re-calibrated using the channel and spatio-temporal attentions, respectively in each residual module. Our attention mechanism enables the model to learn to focus on different facial areas of interest. The experiments are conducted on benchmark facial micro-expression datasets. A superior performance is observed as compared to the state-of-the-art for facial micro-expression recognition on benchmark data.


INTRODUCTION
In recent years, facial expression recognition using deep learning techniques have gained lot of popularity among the research community because of its potential applications across various fields, such as psychology, marketing, security, etc. However, recognizing true human emotions from facial expressions often becomes unreliable due to the performer's deliberate control on the expressions, depending on the social circumstances. Hence, micro-expression recognition has evolved as an attempt to capture the true emotion of a performer [3]. Micro-expressions are involuntarily exposed short term expressions that appear during an active suppression of facial expression to hide the true emotion. Micro-expressions are very short spanned (fraction of a second; usually around 0.5 seconds), which makes the task of capturing the relevant features for recognizing emotion as challenging [4].
Efforts have been made to apply Convolutional Neural Networks (CNNs) for micro-expression recognition [25,34]. Although CNN models are able to automatically capture useful features from the arXiv:2012.04581v2 [cs.CV] 23 Jan 2022 videos, but they are often unable to capture the minute textural changes on various parts of the face, observed during microexpressions. To capture the minute textural information, facial landmarks are used for feature extraction [11]. However, even the temporal features play an important role in recognising facial microexpressions.
A few efforts have also been made in capturing temporal cues from optical flow, for recognizing micro-expressions [5,17,18]. However, features based on optical flow are frequently hampered by unnecessary motion information from background pixels. Another line of thought for capturing temporal features suggests applying a Long Short Term Memory (LSTM) on the spatial features [12][13][14]. A few 3D CNN based models can also be found in the literature, to capture temporal cues [19,30]. The existing approaches of applying temporal features often fails to capture the minute texture information from micro-expression videos, where the expression continues for the very short span of time. Attention based models often show efficacy in extracting the minute texture information [15,24]. However, attention mechanism is still not properly explored for micro-expression recognition task. Wang et al. proposed a spatial attention model for micro-expression recognition [36]. Wang et al. proposed an attention mechanism for recognizing micro-expression using just the apex frame from the micro-expression video. The model completely ignores the temporal information between the frames. Whereas, the proposed model considers both spatial and temporal information simultaneously by applying 3D Conv operations while recalibrating features using attention mechanism and residual connections to learn more fine grained information. To capture the temporal information from the short span of video, where the micro-expression takes place, we propose a 3D channel and spatio-temporal attentions based model called as MERANet. Due to the utilization of the 3D Spatial attention and Channel attention, the proposed model is able to ignore the background information while focusing on the region of interest precisely which is consistent across the frames of the video and responsible for the given micro expression. Fig. 1 represents the overall idea of the proposed MERANet model. The main contributions of this paper are as follows: • A 3D residual attention is introduced in this paper for recognizing subtle features related to micro-expressions by attention guided re-calibration of features. • In order to prioritize the important channels and spatiotemporal features, the channel attention and spatio-temporal attention blocks are used as part of the residual module. • The increase in number of parameters and FLOPs in the proposed model is negligible as compared to the vanilla model. • Extensive experiments are performed to show the impact of the proposed 3D residual attention model on micro-expression recognition. We show the saliency map to visualize the effect of the proposed approach in extracting features from faces during expression.
Next we provide a survey of literature for facial micro-expression recognition.

LITERATURE REVIEW
The proposed method aims to recognize the micro-expressions using attention models. This section provides a discussion on the literature related to micro-expression recognition, followed by a discussion on how attention models are applied to solve various problems in Computer Vision.

Facial Micro-Expression Recognition
The facial micro-expression recognition methods can be categorized into two categories: handcrafted feature based methods and deep learning based methods.
2.1.1 Handcrafted Feature based Methods. The traditional approaches for micro-expression recognition were mostly focused on superficial attributes involving complex calculations. Polikovsky [28] proposed a three step algorithm that divides the facial region into twelve parts to obtain facial cubes. They computed 3D orientation histogram descriptor to capture correlations across the frames for each facial cube. They classified the expression based on the normalized descriptor vectors. Shreve et al. [32] utilized the facial strain caused by the non-rigid motion of facial muscles during an expression. The magnitude of strain was found using optical flow over different regions of the face. Pfister et al. [27] used temporal interpolation model to address variable video length problem. They introduced spatiotemporal local texture descriptors to generate feature vector and finally, various kinds of classifiers have been tried to classify the micro-expression. Huang et al. [9] proposed a spatiotemporal completed local quantized pattern that extracts sign, magnitude and orientation components from an expression and constructed a codebook for each component that holds more dynamic pattern representations. Liu et al. [20] proposed a method that generates a compact feature vector by finding main directional mean optical flow using optical flow field over different regions of interest of the face. The final feature vector is fed into an SVM to classify micro-expressions. Zhao et al. [44] proposed a computationally compact feature using Local Binary Pattern -Three Orthogonal Planes (LBP-TOP) which can efficiently extract co-occurring features from neighboring points. Lu et al. [22] proposed a fusion based feature that finds differentials of optical flow over horizontal and vertical components. The fused feature vector is fed into an SVM classifier for classification.

Deep Learning based Methods.
In most of the computer vision applications, the deep learning techniques outperform the handcrafted feature based methods due to its increased feature learning capability from the data [16]. Patel et al. [25] proposed a deep learning model for micro-expression recognition, where they used transfer learning to initialize the CNN with weights of a CNN trained on largely available macro-expression data to avoid overfitting. Takalkar implemented a variety of data augmentation techniques to generate synthetic data, resembling different environments to deal with lack of training data. Finally a CNN is applied to classify micro-expressions [34]. Qiuyu et al. [18] proposed a three step algorithm where a multi-task CNN was used to find the facial landmarks. Deep CNN was used to calibrate the optical flow features from different facial regions. Finally, an SVM was employed to classify the micro-expressions. Liong et al. [5] collected the optical flow feature for each micro-expression and used it as input to a CNN for classification. Li et al. [17] proposed a 3D flow based CNN that uses gray scale images along with components of optical flow in different directions as input to classify micro-expressions. Wang et al. [36] proposed an image based residual model that uses microattention units and transfer learning to classify micro-expressions. The model fails to consider temporal information across the frames, which is important for micro-expression videos. Several proposals can be found in the literature [12][13][14] that used two step architecture where, a CNN is applied on each frame of the micro-expression video to generate spatial features and then a Long Short Term Memory (LSTM) based Recurrent Neural Network (RNN) is used to learn the temporal inter-dependencies between different spatial features from previous step. Reddy et al. [30] proposed a 3D CNN based model that tries to capture both spatial and temporal information in a micro-expression video simultaneous using 3D kernels. Liong et al. [19] proposed a shallow triple stream 3D CNN that uses optical strain, horizontal and vertical flows from the onset and apex frames to classify micro-expressions.
The existing methods focus neither on the minute spatial changes on face regions during micro-expressions nor on the temporal cues, as the micro-expressions last for a small span of time. A spatial attention mechanism can help in capturing the minute spatial changes in facial regions during micro-expression. In addition to that, a temporal attention can help in focusing on the smaller span of the micro-expression.

Attention Mechanism in Literature
The process of human perceptual analysis is based on the attention mechanism [24]. Human vision obtains the target area within the scene, that needs to be focused on, and pays more attention on the target area to obtain more detailed information about the scene that helps to capture the visual structure of the target better [15]. Attention mechanism is used to find the region of interest within an image and to highlight the representation of the region of interest [10]. The two main aspects of attention mechanism in deep learning are (a) to obtain the meaningful channels for the respective input feature and (b) to pay attention to the most informative channels to obtain the salient locations.
Recently, several studies attempted to apply attention mechanism to enhance the performance of CNNs in a range of visual tasks (especially the tasks where objects play vital roles), such as image classification, image localization and video understanding [1,33]. Wang et al. [37] proposed a Residual Attention Network (RAN) which uses a trunk-and-mask module to incorporate an attention mechanism. By re-weighting the feature map, the network not only performs well but is also robust to input noise. Hu et al. [8] introduced a Squeeze-and-Excitation module to exploit the inter-channel relationship. Adding to this, Woo et al. [38] introduced spatial attention to further improve the image classification performance. In [26], Peng et al. designed an attention branch that helps to spotlight finger micro-gesture and reduces the noise introduced from the background and wrist. The spatial attention has improved the recognition accuracy by 3%. However the attention block proposed in the study of [26], increased the parameters by about 2 . Similarly, for micro-expression recognition, an attention mechanism helps the model to focus on the face and important facial regions, and suppress the irrelevant facial areas and background.
The existing attention based mechanisms have not yet utilized the 3D channel and spatio-temporal attentions, which are important in the context of facial micro-expression recognition. In this work, we design a novel and effective 3D residual attention mechanism to simultaneously learn fine and subtle features along spatial, temporal and channel dimensions. The proposed attention modules, including channel attention and spatio-temporal attention, add just a few parameters to the base model, so that the increase in computation overhead is negligible. Next we illustrate the proposed method for micro-expression recognition.

PROPOSED MERANET MODEL FOR FACIAL MICRO-EXPRESSION RECOGNITION
Given a facial micro-expression video, our objective is to obtain a good representation in terms of their emotional score. We propose a deep learning model, named MERANet, for micro-expression recognition using 3D residual attention network. The proposed residual attention (RA) network is constructed by stacking up multiple RA blocks. The RA block comprises of two major sub-modules: (a) A Spatio-Temporal Attention module to extract the spatio-temporal importance map and (b) A Channel Attention module to extract more discriminative features by prioritizing the important channels.

Spatio-Temporal Attention Block
We compute the spatio-temporal attention map by utilizing the spatial-temporal relationship among the features. The spatio-temporal attention focuses on where we need to pay more attention in the input feature volume. In order to compute the spatio-temporal attention without increasing the model parameters, we first apply average-pooling and max-pooling operations along the channel axis and aggregate them by concatenating to generate an efficient feature descriptor F ∈ R 2× ′ × ′ × ′ , where ′ , ′ and ′ represent the temporal dimension, width, and height respectively, of the generated feature descriptor. The pooling operation is employed along the channel axis as it is effective in highlighting the informative regions [42]. The average pooling across the channels is defined as, and the max pooling across the channels is defined as, where corresponds to the input feature volume and ′ is the number of channels in the input feature ( ).
In order to limit the module complexity, a single convolution operation is performed on the feature descriptor to produce a more refined feature map followed by a sigmoid activation function. The final generated attention importance map (A ) learns the importance score for each spatial-temporal location. The A is defined as, where F is given as, and is the concatenated features of average pooling and max pooling across the channel dimension, A ∈ R 1× ′ × ′ × ′ is the generated attention importance map, denotes the sigmoid function and 5×5×5 corresponds to a convolution operation with the kernel of size 5 with same padding. Note that all the values in the importance map are in the range 0 to 1, thus mimicking the probability of being important. Basically, A encodes that which spatio-temporal information to emphasize or suppress. The block diagram of the spatio-temporal attention module is shown in Fig. 2. Element-wise multiplication is employed between the input feature volume and the spatio-temporal importance map (A ) to re-weight each pixel value and obtain the refined feature map. The re-calibrated feature activation ( ) is defined as, where and are the ℎ channel of input and re-calibrated output using spatio-temporal attention and ⊗ denotes the elementwise multiplication.

Channel Attention Block
The spatio-temporal attention captures the weight for each spatiotemporal feature volume. However, it is not able to distinguish the weight among the different channels of feature volume. Thus, we also propose a channel attention block in the residual framework. The channel features in the CNNs are exploited extensively to extract more discriminative features. We compute a channel attention map by utilizing the inter-channel relationship of features. The channel attention map focuses on finding the meaningful channels for the respective input feature volume. Integrating a channel attention map helps to improve the learning ability of the model by re-calibrating the channels of the input feature volume with increased or decreased weight. In order to capture the channel attention effectively for a feature volume (F ′ ∈ R ′ × ′ × ′ × ′ ), first we squeeze the spatial and temporal dimensions of the input feature volume by applying average pooling and max pooling and then we combine the squeezed features by concatenating them. The average pooling over spatial and temporal dimensions is given as, (6) and the max pooling over spatial and temporal dimensions are given as, where is the number of channels.
The channel descriptors F ℎ and F ℎ are then fed to a shared sub-network to fully capture the inter-dependencies. We propose two design choices for the shared sub-network. One variant is with the convolutional layers and other with the multi-layer perceptron layers (MLP). Our experimental results show that the shared network with convolutions is better than the MLP. The detailed analysis will follow in experimental results in Section 4. The shared convolutional network consists of a convolution layer with the channel reduction mechanism (through 1 × 1 × 1 convolution) to reduce the computation overhead. The ReLU activation is used to introduce the non-linearity followed by a dimension increasing layer to match the input channel dimension (through 1 × 1 × 1 convolution).
The final channel attention map (A ℎ ) is computed as, where and correspond to the Sigmoid and ReLU activation functions, respectively, and are the convolution operations in the shared network with (.) ∈ R ′ ×1×1×1 and (.) ∈ R ′ ×1×1×1 , is the channel reduction ratio in the shared sub-network, and A ℎ ∈ R ′ ×1×1×1 . The reduction ratio( ) is set to 16 in this paper.
Note that all the values in the channel importance map are in the range 0 to 1. Thus, it represents the probability of being important for each channel. The block diagram of the channel attention module is shown in Fig. 3.
Channel-wise multiplication is employed between the input feature volume and the channel importance map (A ) to obtain the re-calibrated channels of the output feature map. The re-calibrated feature activation ( ′ ℎ ) is computed as, ′ ℎ ( , , ℎ, ) = ′ ( , , ℎ, ) ⊗ A ℎ ( , 1, 1, 1),  Here '+' stands for element wise addition, 'X' stands for element wise multiplication and the output corresponds to a recalibrated residual feature map that goes through channel and spatial attention sequentially.

Residual Attention Module
In this paper, we propose a 3D RA module by utilizing the spatiotemporal and channel attention blocks. In contrast to usual residual block, we introduce the attention blocks into the residual module. The residual attention module comprises of two 3D convolution operations, channel attention, spatial-temporal attention, residual connection followed by an activation function. For simplicity, we ignore the batch-norm and activation layers in the block diagram.
The block diagram for the proposed 3D RA module is shown in Fig. 4. Suppose I ∈ R × × × is the input feature volume to the proposed residual attention module and I ∈ R ′ × ′ × ′ × ′ is the generated intermediate feature volume after two convolution blocks. We apply the channel attention operation over I to generate the channel attention map A ℎ as defined in Eq. (8) which uses the average pooling and max pooling over spatial and temporal dimensions as given in Eq. (6) and (7), respectively. The re-calibrated intermediate feature after channel attention is given using Eq. (9) as, Next, we apply the spatio-temporal attention operation over I ℎ to generate the spatio-temporal attention map A as defined in Eq.
(3) which uses the average pooling and max pooling across channel dimension as given in Eq. (1) and (2), respectively. The re-calibrated intermediate feature after spatio-temporal attention is given using Eq. (5) as, The channel attention excites the important channels separately, whereas the spatio-temporal attention excites the important spatiotemporal features. The last step in the proposed residual module is to add the generated re-calibrated features with the original input of this block. The final output of the proposed residual module where is the ReLU activation function and is a down-sampling function to change the dimension of input from × × × to ′ × ′ × ′ × ′ . The down-sampling function is used in the residual path to match the dimensions of the features I and I. (a) If the dimensions of the I and I are same then the down-sampling will be a simple identity mapping with no additional parameters added. (b) If the dimensions of the I and I are different then the down-sampling function will use 1 × 1 × 1 convolution with the stride 2 to match the dimensions.

Network Architecture
The residual network (ResNet) and its variants are one of the most successful architectures for image classification [7] [39] [43]. A basic residual module consists of two convolutional layers in a sequence. The first convolutional layer is followed by a batch normalization and a ReLU activation, whereas the second convolution layer is followed by a batch normalization. The residual module adds the output of the second convolution layer to the input which is followed by a ReLU activation function. The ResNet uses the residual connections that enforces the network to learn the difference of transformation instead of complete transformation. It facilitates better gradient flow during the back-propagation and helps in the optimization of the network. Thus, facilitating the training of very deep networks feasible. Due to the residual property, the ResNet abates the vanishing gradient problem. We also generate the proposed MERANet architecture by following the ResNet architecture in this work. The MERANet-18 is constructed by stacking up the multiple 3D residual attention modules. We use the identity connections and zero padding in the residual paths of the residual attention module to avoid increasing the number of parameters of the model. The 3D residual attention modules are used as the basic building blocks of the proposed MERANet model. We add a stem network to get the reasonable size feature map similar to the ResNet. We also employ a fully connected layer at the end to classify the emotion score. The detailed architecture is summarized in supplementary.

EXPERIMENTAL SETTINGS
In this section, first we provide a brief description of four benchmark micro-expression datasets used for the experiments, followed by the pre-processing details and the training settings.

Micro-Expression Datasets Used
We have used four benchmark micro-expression datasets to evaluate the proposed model, namely CASMEI, CASMEII, CAS(ME) 2 and SAMM. CASME I Dataset [41]: CASME I dataset is divided into two parts: CASME A and CASME B, which are constructed under two different conditions. CASME A contains images with 1280 × 720 resolution under natural light settings. Whereas, CASME B contains images with resolution of 640 × 480 under the artificial light settings. Both settings have a temporal resolution of 60 fps. Altogether, CASME dataset contains 195 micro-expressions belonging to seven different classes. For this experiment, we have chosen three classes, including disgust (34 samples), repression (36 samples), and tense (45 samples) based on the higher number of samples in these classes. CASME II Dataset [40]: CASME II is a spontaneous micro-expressions dataset built under well-controlled laboratory environment. The samples have a higher face resolution at 280 × 340 pixels and a relatively high temporal resolution (200 fps). The database has five micro-expression categories. The number of samples for each category is distributed unequally (there are 60 samples for disgust and only 7 samples for sadness). Since the data is not uniformly distributed across the classes to obtain consistent results, we have used three classes, including disgust (63 samples), happy (32 samples), and surprise (28 samples) based on the higher number of samples in these classes. CAS(ME) 2 Dataset [29]: CAS(ME) 2 dataset comprises of both macro-expressions and micro-expressions. This dataset is constructed by showing 9 elicitation videos to 22 participants under artificial light settings. The elicitation videos contain three emotions (happy, angry, and disgust) evoking videos. CAS(ME) 2 contains the samples with 640 × 480 resolution at 30 frames per second. In our experiment, we have only used micro-expressions of CAS(ME) 2 dataset. The used dataset contains three expressions, namely happy (18 samples), angry (22 samples), and disgust (17 samples).
SAMM Dataset [2]: SAMM dataset comprises of 159 samples with 2040 × 1088 resolution at 200 frames per second. The database has seven micro-expression categories. The number of samples for each category is distributed unequally (there are 57 samples for anger and just 6 samples for sadness). Since the data is not evenly distributed across the classes to obtain consistent results, we have used the following three classes: anger (57 samples), happiness (26 samples), and other (26 samples). We have not included rest of the 4 classes as the number of samples in each of these classes is too less (≤ 15) to train a deep learning system.

Pre-Processing
In the experiments, first, we select the apex frame in the video corresponding to the micro-expression in order to generate a training sample. As the data is captured at different frame rates per second, we have assigned temporal depth for SAMM as 64, CASMEI and CASMEII as 32 and CAS(ME) 2 as 16. Also, we have considered samples having atleast 16 frames for CASMEI and CASMEII in this experimentation. A 16-frame clip is generated around the selected apex frame. If the video is shorter than 16 frames, then we replicate the edge frames as many times as necessary. The temporal dimension is selected based on the average number of frames available in a video clip. We spatially resize the sample to 112 × 112 . The size of each sample is 3 channels × 16 frames × 112 pixels × 112 pixels, and each sample is horizontally flipped with 0.5 probability. Mean subtraction is applied for each sample by subtracting the mean values from the sample for each channel. Each sample is normalized by the standard deviation.

Training Settings
We have performed 75%-25% train-test split for CASME I and CASME II and 80%-20% train-test split for CAS(ME) 2 due to less samples. Among the splits 80% of data is used as a training set and the rest is used as a validation set. Stratified sampling is used for the validation split to maintain consistency across the results. The train-validation split-up is performed once and then the same training and validation sets are used for all the experiments.
The weights of both convolutional and fully-connected layers are initialized with the Xavier initialization proposed in [6]. We set the parameter to random values uniformly drawn from [− , ], where is defined as, = √︃ 6 ( + ) where and are the size of input and output channels, respectively. All biases are initialized to 0. For batch normalization layers, weights are initialized to 1.
The optimizer proposed in [23] is used for training. We use Categorical Cross Entropy as the loss function. Each model is trained for 100 epochs with a batch size of 8. The learning rate is initialized to 0.1. Learning rate adjustment is one of the crucial steps while training. We use the cosine annealing strategy proposed in [21]. The cosine annealing decreases the learning rate from the initial value to 0 by following the cosine function. The intuition behind using annealing is that it helps to traverse quickly from the initial parameters to a range of good parameter values with a smaller learning rate. Thus, we can explore the deeper and narrower parts of the loss function, which potentially improves the training progress by avoiding the divergence.
We avoid the overfitting problem in two ways, 1) by using the Horizontal flip data augmentation and 2) by incorporating the warm-up scheduling along with the cosine annealing for ADAM Table 1: The performance of the proposed MERANet model in terms of the facial micro-expression recognition accuracy as compared to the existing state-of-the-art models. Note that the results of existing methods are taken from the respective papers. The HCM and DLM denote the hand crafted method and deep learning method, respectively.
Method CASME I CASME II CAS(ME) 2 Year Method Type STCLQP [9] 57. 31 [23], which helps significantly in the convergence. Next we discuss the results obtained by experimenting on the proposed method.

RESULTS AND DISCUSSION
In this section, first we present the experimental results comparison, then the impact of kernel size on spatio-temporal attention module and the visualization of impact of attention through saliency maps.

Experimental Results
Experiments are conducted by training both the 3D residual network and the proposed 3D residual attention network models. The results are compared in terms of the facial micro-expression recognition accuracy. The performance of the proposed residual attention network is compared with the state-of-the-art hand-crafted methods (HCM) and deep learning methods (DLM) as shown in Table  1. The results are computed over three benchmark facial microexpression datasets, including CASME I, CASME II, and CAS(ME) 2 in Table 1. Note that the results of competing methods are taken from the corresponding papers. Table 1 depicts that the proposed MERANet model outperforms the existing hand crafted and deep learning models with a significant margin. As expected, the channel attention and the spatio-temporal attention blocks help the residual network to learn better micro-level spatio-temporal features leading to the enhanced decision making for facial micro-expression recognition. The total number of parameters for the original 3D ResNet model is 3, 31, 67, 811 which, for the proposed 3D MERANet model, is increased to 3, 35, 47, 552. The proposed model uses 1.72 GFLOPs whereas original model uses 1.61 GFLOPs. The number of parameters increased for the proposed model by 3, 79, 741, and FLOPs increased by 0.11G, which is quite small compared to the original 3D ResNet model. So, the overall overhead added by the proposed attention module is negligible in terms of both parameters and computation. Table 2 represents the results in terms of the Accuracy, Precision, Recall and F1 measures for four benchmark datasets, including , , ( ) 2 , and . Promising results are obtained using the proposed MERANet model for microexpression recognition. The quantitative results provided in Table  1 and Table 2 show the efficacy of the proposed attention modules. Table 3 shows the confusion matrix for the proposed method on the SAMM dataset.

Impact of Shared Network in Channel Attention
We now compare the efficacy of shared CNN and shared MLP in channel attention. For this study, we use the ( ) 2 dataset. Inorder to fully capture the interrelationship, the channel descriptor of average pooling F ℎ and max pooling F ℎ are fed to a shared

Impact of Kernel Size on Spatio-Temporal Attention
The performance of spatial-temporal attention block is mainly dependant on the applied convolution operation. In general, convolution layer performance depends on the size of the filter used. The kernel size plays a prominent role in extracting the subtle features. So, we have experimented with different kernel sizes, i.e., 3, 5, and 7 to find the optimal kernel size that boosts the model learning capability. The validation curve is shown in Fig. 6 having the plots corresponding to each kernel size. It is observed that the smaller kernel size (i.e., small receptive field) doesn't help the model to learn better. In contrast, higher kernel size ( = 7) increases the computation head, yet it fails to generalize better. From Fig. 6, we also infer that a reasonable receptive field is needed for deciding important regions. In the comparison of different convolution kernel sizes, we find the kernel size of 5 helps the model to learn better micro-expression representation.

Impact of Attention Through Saliency Map
In this subsection, we empirically show the effectiveness of our proposed model. For this study, we use all the three datasets. In order to further understand the behavior of the proposed model, we have used the feature visualization technique proposed in [31]. These feature visualizations help us to find which spatial locations trigger the classification outcome, thus providing the insights about the model's understanding of the emotions. The feature visualizations are given in Fig. 7. According to [4], an emotion depicting disgust on a subject's face has facial muscle movements centered around the nose of the subject. Similarly, a happy emotion has facial muscle movements centered around the lip region. From Fig. 7, we infer that the MERANet focuses on the face regions completely ignoring the background. Furthermore, it gives more weight to facial regions corresponding to an emotion. The saliency maps of 3D ResNet model seem to be very arbitrary by considering the noisy regions in the decision making process. Whereas, the saliency maps for the proposed MERANet model clearly matches the intuition with the general procedure of finding facial expressions and more convincing.

CONCLUSION
We have introduced a 3D residual attention based MERANet model for micro-expression recognition. The proposed 3D residual attention module utilizes the benefits of channel attention as well as spatio-temporal attention. These attention blocks help the proposed model to focus over the spatio-temporal features leading to the facial micro-expression recognition. We have conducted the extensive experiments with the proposed model over four benchmark datasets and compared with the state-of-the-art models. The proposed MERANet model outperforms the existing models with a negligible increase in the number of parameters. We also found that the shared CNN is better suitable with channel attention. However, the kernel size of 5 is better suitable with spatio-temporal attention.
The saliency map of the proposed MERANet model is very close to the human perception of the emotions in terms of the facial features contribution towards the identification of the emotions.