Multi-modal embeddings using multi-task learning for emotion recognition
  General embeddings like word2vec, GloVe and ELMo have shown a lot of success
in natural language tasks. The embeddings are typically extracted from models
that are built on general tasks such as skip-gram models and natural language
generation. In this paper, we extend the work from natural language
understanding to multi-modal architectures that use audio, visual and textual
information for machine learning tasks. The embeddings in our network are
extracted using the encoder of a transformer model trained using multi-task
training. We use person identification and automatic speech recognition as the
tasks in our embedding generation framework. We tune and evaluate the
embeddings on the downstream task of emotion recognition and demonstrate that
on the CMU-MOSEI dataset, the embeddings can be used to improve over previous
state of the art results.
