Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for
  In-The-Wild Affect Recognition
  In this paper, we present our submission to 3rd Affective Behavior Analysis
in-the-wild (ABAW) challenge. Learningcomplex interactions among multimodal
sequences is critical to recognise dimensional affect from in-the-wild
audiovisual data. Recurrence and attention are the two widely used sequence
modelling mechanisms in the literature. To clearly understand the performance
differences between recurrent and attention models in audiovisual affect
recognition, we present a comprehensive evaluation of fusion models based on
LSTM-RNNs, self-attention and cross-modal attention, trained for valence and
arousal estimation. Particularly, we study the impact of some key design
choices: the modelling complexity of CNN backbones that provide features to the
the temporal models, with and without end-to-end learning. We trained the
audiovisual affect recognition models on in-the-wild ABAW corpus by
systematically tuning the hyper-parameters involved in the network architecture
design and training optimisation. Our extensive evaluation of the audiovisual
fusion models shows that LSTM-RNNs can outperform the attention models when
coupled with low-complex CNN backbones and trained in an end-to-end fashion,
implying that attention models may not necessarily be the optimal choice for
continuous-time multimodal emotion recognition.
