StreaMulT: Streaming Multimodal Transformer for Heterogeneous and
  Arbitrary Long Sequential Data
  This paper tackles the problem of processing and combining efficiently
arbitrary long data streams, coming from different modalities with different
acquisition frequencies. Common applications can be, for instance, long-time
industrial or real-life systems monitoring from multimodal heterogeneous data
(sensor data, monitoring report, images, etc.). To tackle this problem, we
propose StreaMulT, a Streaming Multimodal Transformer, relying on cross-modal
attention and an augmented memory bank to process arbitrary long input
sequences at training time and run in a streaming way at inference. StreaMulT
reproduces state-of-the-art results on CMU-MOSEI dataset, while being able to
deal with much longer inputs than other models such as previous Multimodal
Transformer.
