Multimodal Generalized Zero Shot Learning for Gleason Grading using
  Self-Supervised Learning
  Gleason grading from histopathology images is essential for accurate prostate
cancer (PCa) diagnosis. Since such images are obtained after invasive tissue
resection quick diagnosis is challenging under the existing paradigm. We
propose a method to predict Gleason grades from magnetic resonance (MR) images
which are non-interventional and easily acquired. We solve the problem in a
generalized zero-shot learning (GZSL) setting since we may not access training
images of every disease grade. Synthetic MRI feature vectors of unseen grades
(classes) are generated by exploiting Gleason grades' ordered nature through a
conditional variational autoencoder (CVAE) incorporating self-supervised
learning. Corresponding histopathology features are generated using cycle GANs,
and combined with MR features to predict Gleason grades of test images.
Experimental results show our method outperforms competing feature generating
approaches for GZSL, and comes close to performance of fully supervised
methods.
