Which is Making the Contribution: Modulating Unimodal and Cross-modal
  Dynamics for Multimodal Sentiment Analysis
  Multimodal sentiment analysis (MSA) draws increasing attention with the
availability of multimodal data. The boost in performance of MSA models is
mainly hindered by two problems. On the one hand, recent MSA works mostly focus
on learning cross-modal dynamics, but neglect to explore an optimal solution
for unimodal networks, which determines the lower limit of MSA models. On the
other hand, noisy information hidden in each modality interferes the learning
of correct cross-modal dynamics. To address the above-mentioned problems, we
propose a novel MSA framework \textbf{M}odulation \textbf{M}odel for
\textbf{M}ultimodal \textbf{S}entiment \textbf{A}nalysis ({$ M^3SA $}) to
identify the contribution of modalities and reduce the impact of noisy
information, so as to better learn unimodal and cross-modal dynamics.
Specifically, modulation loss is designed to modulate the loss contribution
based on the confidence of individual modalities in each utterance, so as to
explore an optimal update solution for each unimodal network. Besides, contrary
to most existing works which fail to explicitly filter out noisy information,
we devise a modality filter module to identify and filter out modality noise
for the learning of correct cross-modal embedding. Extensive experiments on
publicly datasets demonstrate that our approach achieves state-of-the-art
performance.
