Sparse Fusion for Multimodal Transformers
  Multimodal classification is a core task in human-centric machine learning.
We observe that information is highly complementary across modalities, thus
unimodal information can be drastically sparsified prior to multimodal fusion
without loss of accuracy. To this end, we present Sparse Fusion Transformers
(SFT), a novel multimodal fusion method for transformers that performs
comparably to existing state-of-the-art methods while having greatly reduced
memory footprint and computation cost. Key to our idea is a sparse-pooling
block that reduces unimodal token sets prior to cross-modality modeling.
Evaluations are conducted on multiple multimodal benchmark datasets for a wide
range of classification tasks. State-of-the-art performance is obtained on
multiple benchmarks under similar experiment conditions, while reporting up to
six-fold reduction in computational cost and memory requirements. Extensive
ablation studies showcase our benefits of combining sparsification and
multimodal learning over naive approaches. This paves the way for enabling
multimodal learning on low-resource devices.
