Counterfactually Guided Off-policy Transfer in Clinical Settings
  Domain shift, encountered when using a trained model for a new patient
population, creates significant challenges for sequential decision making in
healthcare since the target domain may be both data-scarce and confounded. In
this paper, we propose a method for off-policy transfer by modeling the
underlying generative process with a causal mechanism. We use informative
priors from the source domain to augment counterfactual trajectories in the
target in a principled manner. We demonstrate how this addresses data-scarcity
in the presence of unobserved confounding. The causal parametrization of our
sampling procedure guarantees that counterfactual quantities can be estimated
from scarce observational target data, maintaining intuitive stability
properties. Policy learning in the target domain is further regularized via the
source policy through KL-divergence. Through evaluation on a simulated sepsis
treatment task, our counterfactual policy transfer procedure significantly
improves the performance of a learned treatment policy when assumptions of
"no-unobserved confounding" are relaxed.
