LMR-CBT: Learning Modality-fused Representations with CB-Transformer for
  Multimodal Emotion Recognition from Unaligned Multimodal Sequences
  Learning modality-fused representations and processing unaligned multimodal
sequences are meaningful and challenging in multimodal emotion recognition.
Existing approaches use directional pairwise attention or a message hub to fuse
language, visual, and audio modalities. However, those approaches introduce
information redundancy when fusing features and are inefficient without
considering the complementarity of modalities. In this paper, we propose an
efficient neural network to learn modality-fused representations with
CB-Transformer (LMR-CBT) for multimodal emotion recognition from unaligned
multimodal sequences. Specifically, we first perform feature extraction for the
three modalities respectively to obtain the local structure of the sequences.
Then, we design a novel transformer with cross-modal blocks (CB-Transformer)
that enables complementary learning of different modalities, mainly divided
into local temporal learning,cross-modal feature fusion and global
self-attention representations. In addition, we splice the fused features with
the original features to classify the emotions of the sequences. Finally, we
conduct word-aligned and unaligned experiments on three challenging datasets,
IEMOCAP, CMU-MOSI, and CMU-MOSEI. The experimental results show the superiority
and efficiency of our proposed method in both settings. Compared with the
mainstream methods, our approach reaches the state-of-the-art with a minimum
number of parameters.
