RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for
  Adaptive Radiotherapy
  The current cancer treatment practice collects multimodal data, such as
radiology images, histopathology slides, genomics and clinical data. The
importance of these data sources taken individually has fostered the recent
raise of radiomics and pathomics, i.e. the extraction of quantitative features
from radiology and histopathology images routinely collected to predict
clinical outcomes or to guide clinical decisions using artificial intelligence
algorithms. Nevertheless, how to combine them into a single multimodal
framework is still an open issue. In this work we therefore develop a
multimodal late fusion approach that combines hand-crafted features computed
from radiomics, pathomics and clinical data to predict radiation therapy
treatment outcomes for non-small-cell lung cancer patients. Within this
context, we investigate eight different late fusion rules (i.e. product,
maximum, minimum, mean, decision template, Dempster-Shafer, majority voting,
and confidence rule) and two patient-wise aggregation rules leveraging the
richness of information given by computer tomography images and whole-slide
scans. The experiments in leave-one-patient-out cross-validation on an in-house
cohort of 33 patients show that the proposed multimodal paradigm with an AUC
equal to $90.9\%$ outperforms each unimodal approach, suggesting that data
integration can advance precision medicine. As a further contribution, we also
compare the hand-crafted representations with features automatically computed
by deep networks, and the late fusion paradigm with early fusion, another
popular multimodal approach. In both cases, the experiments show that the
proposed multimodal approach provides the best results.
