Found in Translation: Learning Robust Joint Representations by Cyclic
  Translations Between Modalities
  Multimodal sentiment analysis is a core research area that studies speaker
sentiment expressed from the language, visual, and acoustic modalities. The
central challenge in multimodal learning involves inferring joint
representations that can process and relate information from these modalities.
However, existing work learns joint representations by requiring all modalities
as input and as a result, the learned representations may be sensitive to noisy
or missing modalities at test time. With the recent success of sequence to
sequence (Seq2Seq) models in machine translation, there is an opportunity to
explore new ways of learning joint representations that may not require all
input modalities at test time. In this paper, we propose a method to learn
robust joint representations by translating between modalities. Our method is
based on the key insight that translation from a source to a target modality
provides a method of learning joint representations using only the source
modality as input. We augment modality translations with a cycle consistency
loss to ensure that our joint representations retain maximal information from
all modalities. Once our translation model is trained with paired multimodal
data, we only need data from the source modality at test time for final
sentiment prediction. This ensures that our model remains robust from
perturbations or missing information in the other modalities. We train our
model with a coupled translation-prediction objective and it achieves new
state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI,
ICT-MMMO, and YouTube. Additional experiments show that our model learns
increasingly discriminative joint representations with more input modalities
while maintaining robustness to missing or perturbed modalities.
