ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale
  Fusion of Locally Descriptors
  Fusion technique is a key research topic in multimodal sentiment analysis.
The recent attention-based fusion demonstrates advances over simple
operation-based fusion. However, these fusion works adopt single-scale, i.e.,
token-level or utterance-level, unimodal representation. Such single-scale
fusion is suboptimal because that different modality should be aligned with
different granularities. This paper proposes a fusion model named ScaleVLAD to
gather multi-Scale representation from text, video, and audio with shared
Vectors of Locally Aggregated Descriptors to improve unaligned multimodal
sentiment analysis. These shared vectors can be regarded as shared topics to
align different modalities. In addition, we propose a self-supervised shifted
clustering loss to keep the fused feature differentiation among samples. The
backbones are three Transformer encoders corresponding to three modalities, and
the aggregated features generated from the fusion module are feed to a
Transformer plus a full connection to finish task predictions. Experiments on
three popular sentiment analysis benchmarks, IEMOCAP, MOSI, and MOSEI,
demonstrate significant gains over baselines.
