EMIXER: End-to-end Multimodal X-ray Generation via Self-supervision
  Deep generative models have enabled the automated synthesis of high-quality
data for diverse applications. However, the most effective generative models
are specialized to data from a single domain (e.g., images or text). Real-world
applications such as healthcare require multi-modal data from multiple domains
(e.g., both images and corresponding text), which are difficult to acquire due
to limited availability and privacy concerns and are much harder to synthesize.
To tackle this joint synthesis challenge, we propose an End-to-end MultImodal
X-ray genERative model (EMIXER) for jointly synthesizing x-ray images and
corresponding free-text reports, all conditional on diagnosis labels. EMIXER is
an conditional generative adversarial model by 1) generating an image based on
a label, 2) encoding the image to a hidden embedding, 3) producing the
corresponding text via a hierarchical decoder from the image embedding, and 4)
a joint discriminator for assessing both the image and the corresponding text.
EMIXER also enables self-supervision to leverage vast amount of unlabeled data.
Extensive experiments with real X-ray reports data illustrate how data
augmentation using synthesized multimodal samples can improve the performance
of a variety of supervised tasks including COVID-19 X-ray classification with
very limited samples. The quality of generated images and reports are also
confirmed by radiologists. We quantitatively show that EMIXER generated
synthetic datasets can augment X-ray image classification, report generation
models to achieve 5.94% and 6.9% improvement on models trained only on real
data samples. Taken together, our results highlight the promise of state of
generative models to advance clinical machine learning.
