Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation
  Medical report generation, which aims to automatically generate a long and
coherent report of a given medical image, has been receiving growing research
interests. Existing approaches mainly adopt a supervised manner and heavily
rely on coupled image-report pairs. However, in the medical domain, building a
large-scale image-report paired dataset is both time-consuming and expensive.
To relax the dependency on paired data, we propose an unsupervised model
Knowledge Graph Auto-Encoder (KGAE) which accepts independent sets of images
and reports in training. KGAE consists of a pre-constructed knowledge graph, a
knowledge-driven encoder and a knowledge-driven decoder. The knowledge graph
works as the shared latent space to bridge the visual and textual domains; The
knowledge-driven encoder projects medical images and reports to the
corresponding coordinates in this latent space and the knowledge-driven decoder
generates a medical report given a coordinate in this space. Since the
knowledge-driven encoder and decoder can be trained with independent sets of
images and reports, KGAE is unsupervised. The experiments show that the
unsupervised KGAE generates desirable medical reports without using any
image-report training pairs. Moreover, KGAE can also work in both
semi-supervised and supervised settings, and accept paired images and reports
in training. By further fine-tuning with image-report pairs, KGAE consistently
outperforms the current state-of-the-art models on two datasets.
