Continuous Emotion Recognition with Spatiotemporal Convolutional Neural
  Networks
  Facial expressions are one of the most powerful ways for depicting specific
patterns in human behavior and describing human emotional state. Despite the
impressive advances of affective computing over the last decade, automatic
video-based systems for facial expression recognition still cannot handle
properly variations in facial expression among individuals as well as
cross-cultural and demographic aspects. Nevertheless, recognizing facial
expressions is a difficult task even for humans. In this paper, we investigate
the suitability of state-of-the-art deep learning architectures based on
convolutional neural networks (CNNs) for continuous emotion recognition using
long video sequences captured in-the-wild. This study focuses on deep learning
models that allow encoding spatiotemporal relations in videos considering a
complex and multi-dimensional emotion space, where values of valence and
arousal must be predicted. We have developed and evaluated convolutional
recurrent neural networks combining 2D-CNNs and long short term-memory units,
and inflated 3D-CNN models, which are built by inflating the weights of a
pre-trained 2D-CNN model during fine-tuning, using application-specific videos.
Experimental results on the challenging SEWA-DB dataset have shown that these
architectures can effectively be fine-tuned to encode the spatiotemporal
information from successive raw pixel images and achieve state-of-the-art
results on such a dataset.
