Multi stain graph fusion for multimodal integration in pathology
  In pathology, tissue samples are assessed using multiple staining techniques
to enhance contrast in unique histologic features. In this paper, we introduce
a multimodal CNN-GNN based graph fusion approach that leverages complementary
information from multiple non-registered histopathology images to predict
pathologic scores. We demonstrate this approach in nonalcoholic steatohepatitis
(NASH) by predicting CRN fibrosis stage and NAFLD Activity Score (NAS). Primary
assessment of NASH typically requires liver biopsy evaluation on two
histological stains: Trichrome (TC) and hematoxylin and eosin (H&E). Our
multimodal approach learns to extract complementary information from TC and H&E
graphs corresponding to each stain while simultaneously learning an optimal
policy to combine this information. We report up to 20% improvement in
predicting fibrosis stage and NAS component grades over single-stain modeling
approaches, measured by computing linearly weighted Cohen's kappa between
machine-derived vs. pathologist consensus scores. Broadly, this paper
demonstrates the value of leveraging diverse pathology images for improved
ML-powered histologic assessment.
