MMTM: Multimodal Transfer Module for CNN Fusion
  In late fusion, each modality is processed in a separate unimodal
Convolutional Neural Network (CNN) stream and the scores of each modality are
fused at the end. Due to its simplicity late fusion is still the predominant
approach in many state-of-the-art multimodal applications. In this paper, we
present a simple neural network module for leveraging the knowledge from
multiple modalities in convolutional neural networks. The propose unit, named
Multimodal Transfer Module (MMTM), can be added at different levels of the
feature hierarchy, enabling slow modality fusion. Using squeeze and excitation
operations, MMTM utilizes the knowledge of multiple modalities to recalibrate
the channel-wise features in each CNN stream. Despite other intermediate fusion
methods, the proposed module could be used for feature modality fusion in
convolution layers with different spatial dimensions. Another advantage of the
proposed method is that it could be added among unimodal branches with minimum
changes in the their network architectures, allowing each branch to be
initialized with existing pretrained weights. Experimental results show that
our framework improves the recognition accuracy of well-known multimodal
networks. We demonstrate state-of-the-art or competitive performance on four
datasets that span the task domains of dynamic hand gesture recognition, speech
enhancement, and action recognition with RGB and body joints.
