M2Net: Multi-modal Multi-channel Network for Overall Survival Time
  Prediction of Brain Tumor Patients
  Early and accurate prediction of overall survival (OS) time can help to
obtain better treatment planning for brain tumor patients. Although many OS
time prediction methods have been developed and obtain promising results, there
are still several issues. First, conventional prediction methods rely on
radiomic features at the local lesion area of a magnetic resonance (MR) volume,
which may not represent the full image or model complex tumor patterns. Second,
different types of scanners (i.e., multi-modal data) are sensitive to different
brain regions, which makes it challenging to effectively exploit the
complementary information across multiple modalities and also preserve the
modality-specific properties. Third, existing methods focus on prediction
models, ignoring complex data-to-label relationships. To address the above
issues, we propose an end-to-end OS time prediction model; namely, Multi-modal
Multi-channel Network (M2Net). Specifically, we first project the 3D MR volume
onto 2D images in different directions, which reduces computational costs,
while preserving important information and enabling pre-trained models to be
transferred from other tasks. Then, we use a modality-specific network to
extract implicit and high-level features from different MR scans. A multi-modal
shared network is built to fuse these features using a bilinear pooling model,
exploiting their correlations to provide complementary information. Finally, we
integrate the outputs from each modality-specific network and the multi-modal
shared network to generate the final prediction result. Experimental results
demonstrate the superiority of our M2Net model over other methods.
