Interpretable multimodal fusion networks reveal mechanisms of brain
  cognition
  Multimodal fusion benefits disease diagnosis by providing a more
comprehensive perspective. Developing algorithms is challenging due to data
heterogeneity and the complex within- and between-modality associations.
Deep-network-based data-fusion models have been developed to capture the
complex associations and the performance in diagnosis has been improved
accordingly. Moving beyond diagnosis prediction, evaluation of disease
mechanisms is critically important for biomedical research. Deep-network-based
data-fusion models, however, are difficult to interpret, bringing about
difficulties for studying biological mechanisms. In this work, we develop an
interpretable multimodal fusion model, namely gCAM-CCL, which can perform
automated diagnosis and result interpretation simultaneously. The gCAM-CCL
model can generate interpretable activation maps, which quantify pixel-level
contributions of the input features. This is achieved by combining intermediate
feature maps using gradient-based weights. Moreover, the estimated activation
maps are class-specific, and the captured cross-data associations are
interest/label related, which further facilitates class-specific analysis and
biological mechanism analysis. We validate the gCAM-CCL model on a brain
imaging-genetic study, and show gCAM-CCL's performed well for both
classification and mechanism analysis. Mechanism analysis suggests that during
task-fMRI scans, several object recognition related regions of interests (ROIs)
are first activated and then several downstream encoding ROIs get involved.
Results also suggest that the higher cognition performing group may have
stronger neurotransmission signaling while the lower cognition performing group
may have problem in brain/neuron development, resulting from genetic
variations.
