Unsupervised Multimodal Representation Learning across Medical Images
  and Reports
  Joint embeddings between medical imaging modalities and associated radiology
reports have the potential to offer significant benefits to the clinical
community, ranging from cross-domain retrieval to conditional generation of
reports to the broader goals of multimodal representation learning. In this
work, we establish baseline joint embedding results measured via both local and
global retrieval methods on the soon to be released MIMIC-CXR dataset
consisting of both chest X-ray images and the associated radiology reports. We
examine both supervised and unsupervised methods on this task and show that for
document retrieval tasks with the learned representations, only a limited
amount of supervision is needed to yield results comparable to those of
fully-supervised methods.
