MMLN: Leveraging Domain Knowledge for Multimodal Diagnosis
  Recent studies show that deep learning models achieve good performance on
medical imaging tasks such as diagnosis prediction. Among the models,
multimodality has been an emerging trend, integrating different forms of data
such as chest X-ray (CXR) images and electronic medical records (EMRs).
However, most existing methods incorporate them in a model-free manner, which
lacks theoretical support and ignores the intrinsic relations between different
data sources. To address this problem, we propose a knowledge-driven and
data-driven framework for lung disease diagnosis. By incorporating domain
knowledge, machine learning models can reduce the dependence on labeled data
and improve interpretability. We formulate diagnosis rules according to
authoritative clinical medicine guidelines and learn the weights of rules from
text data. Finally, a multimodal fusion consisting of text and image data is
designed to infer the marginal probability of lung disease. We conduct
experiments on a real-world dataset collected from a hospital. The results show
that the proposed method outperforms the state-of-the-art multimodal baselines
in terms of accuracy and interpretability.
