Multimodal Multitask Representation Learning for Pathology Biobank
  Metadata Prediction
  Metadata are general characteristics of the data in a well-curated and
condensed format, and have been proven to be useful for decision making,
knowledge discovery, and also heterogeneous data organization of biobank. Among
all data types in the biobank, pathology is the key component of the biobank
and also serves as the gold standard of diagnosis. To maximize the utility of
biobank and allow the rapid progress of biomedical science, it is essential to
organize the data with well-populated pathology metadata. However, manual
annotation of such information is tedious and time-consuming. In the study, we
develop a multimodal multitask learning framework to predict four major
slide-level metadata of pathology images. The framework learns generalizable
representations across tissue slides, pathology reports, and case-level
structured data. We demonstrate improved performance across all four tasks with
the proposed method compared to a single modal single task baseline on two test
sets, one external test set from a distinct data source (TCGA) and one internal
held-out test set (TTH). In the test sets, the performance improvements on the
averaged area under receiver operating characteristic curve across the four
tasks are 16.48% and 9.05% on TCGA and TTH, respectively. Such pathology
metadata prediction system may be adopted to mitigate the effort of expert
annotation and ultimately accelerate the data-driven research by better
utilization of the pathology biobank.
