Joint Learning of Localized Representations from Medical Images and
  Reports
  Contrastive learning has proven effective for pre-training image models on
unlabeled data with promising results for tasks such as medical image
classification. Using paired text and images (such as radiological reports and
images) during pre-training improved the results even further. Still, most
existing methods target image classification as downstream tasks and may not be
optimal for localized tasks like semantic segmentation or object detection. We
therefore propose Localized representation learning from Vision and Text
(LoVT), to our best knowledge, the first text-supervised pre-training method
that targets localized medical imaging tasks. Our method combines
instance-level image-report contrastive learning with local contrastive
learning on image region and report sentence representations. We evaluate LoVT
and commonly used pre-training methods on a novel evaluation framework
consisting of 18 localized tasks on chest X-rays from five public datasets.
While there is no single best method, LoVT performs best on 11 out of the 18
studied tasks making it the preferred method of choice for localized tasks.
