Deep Multimodal Fusion by Channel Exchanging
  Deep multimodal fusion by using multiple sources of data for classification
or regression has exhibited a clear advantage over the unimodal counterpart on
various applications. Yet, current methods including aggregation-based and
alignment-based fusion are still inadequate in balancing the trade-off between
inter-modal fusion and intra-modal processing, incurring a bottleneck of
performance improvement. To this end, this paper proposes
Channel-Exchanging-Network (CEN), a parameter-free multimodal fusion framework
that dynamically exchanges channels between sub-networks of different
modalities. Specifically, the channel exchanging process is self-guided by
individual channel importance that is measured by the magnitude of
Batch-Normalization (BN) scaling factor during training. The validity of such
exchanging process is also guaranteed by sharing convolutional filters yet
keeping separate BN layers across modalities, which, as an add-on benefit,
allows our multimodal architecture to be almost as compact as a unimodal
network. Extensive experiments on semantic segmentation via RGB-D data and
image translation through multi-domain input verify the effectiveness of our
CEN compared to current state-of-the-art methods. Detailed ablation studies
have also been carried out, which provably affirm the advantage of each
component we propose. Our code is available at https://github.com/yikaiw/CEN.
