How and Why to Use Experimental Data to Evaluate Methods for
  Observational Causal Inference
  Methods that infer causal dependence from observational data are central to
many areas of science, including medicine, economics, and the social sciences.
A variety of theoretical properties of these methods have been proven, but
empirical evaluation remains a challenge, largely due to the lack of
observational data sets for which treatment effect is known. We describe and
analyze observational sampling from randomized controlled trials (OSRCT), a
method for evaluating causal inference methods using data from randomized
controlled trials (RCTs). This method can be used to create constructed
observational data sets with corresponding unbiased estimates of treatment
effect, substantially increasing the number of data sets available for
empirical evaluation of causal inference methods. We show that, in expectation,
OSRCT creates data sets that are equivalent to those produced by randomly
sampling from empirical data sets in which all potential outcomes are
available. We then perform a large-scale evaluation of seven causal inference
methods over 37 data sets, drawn from RCTs, as well as simulators, real-world
computational systems, and observational data sets augmented with a synthetic
response variable. We find notable performance differences when comparing
across data from different sources, demonstrating the importance of using data
from a variety of sources when evaluating any causal inference method.
