Towards Explainable End-to-End Prostate Cancer Relapse Prediction from
  H&E Images Combining Self-Attention Multiple Instance Learning with a
  Recurrent Neural Network
  Clinical decision support for histopathology image data mainly focuses on
strongly supervised annotations, which offers intuitive interpretability, but
is bound by expert performance. Here, we propose an explainable cancer relapse
prediction network (eCaReNet) and show that end-to-end learning without strong
annotations offers state-of-the-art performance while interpretability can be
included through an attention mechanism. On the use case of prostate cancer
survival prediction, using 14,479 images and only relapse times as annotations,
we reach a cumulative dynamic AUC of 0.78 on a validation set, being on par
with an expert pathologist (and an AUC of 0.77 on a separate test set). Our
model is well-calibrated and outputs survival curves as well as a risk score
and group per patient. Making use of the attention weights of a multiple
instance learning layer, we show that malignant patches have a higher influence
on the prediction than benign patches, thus offering an intuitive
interpretation of the prediction. Our code is available at
www.github.com/imsb-uke/ecarenet.
