GMC -- Geometric Multimodal Contrastive Representation Learning
  Learning representations of multimodal data that are both informative and
robust to missing modalities at test time remains a challenging problem due to
the inherent heterogeneity of data obtained from different channels. To address
it, we present a novel Geometric Multimodal Contrastive (GMC) representation
learning method comprised of two main components: i) a two-level architecture
consisting of modality-specific base encoder, allowing to process an arbitrary
number of modalities to an intermediate representation of fixed dimensionality,
and a shared projection head, mapping the intermediate representations to a
latent representation space; ii) a multimodal contrastive loss function that
encourages the geometric alignment of the learned representations. We
experimentally demonstrate that GMC representations are semantically rich and
achieve state-of-the-art performance with missing modality information on three
different learning problems including prediction and reinforcement learning
tasks.
