Multi-Modal Causal Inference with Deep Structural Equation Models
  Accounting for the effects of confounders is one of the central challenges in
causal inference. Unstructured multi-modal data (images, time series, text)
contains valuable information about diverse types of confounders, yet it is
typically left unused by most existing methods. This paper seeks to develop
techniques that leverage this unstructured data within causal inference to
correct for additional confounders that may otherwise not be accounted for. We
formalize this task and we propose algorithms based on deep structural
equations that treat multi-modal unstructured data as proxy variables. We
empirically demonstrate on tasks in genomics and healthcare that unstructured
data can be used to correct for diverse sources of confounding, potentially
enabling the use of large amounts of data that were previously not used in
causal inference.
