The Case for Evaluating Causal Models Using Interventional Measures and
  Empirical Data
  Causal inference is central to many areas of artificial intelligence,
including complex reasoning, planning, knowledge-base construction, robotics,
explanation, and fairness. An active community of researchers develops and
enhances algorithms that learn causal models from data, and this work has
produced a series of impressive technical advances. However, evaluation
techniques for causal modeling algorithms have remained somewhat primitive,
limiting what we can learn from experimental studies of algorithm performance,
constraining the types of algorithms and model representations that researchers
consider, and creating a gap between theory and practice. We argue for more
frequent use of evaluation techniques that examine interventional measures
rather than structural or observational measures, and that evaluate those
measures on empirical data rather than synthetic data. We survey the current
practice in evaluation and show that the techniques we recommend are rarely
used in practice. We show that such techniques are feasible and that data sets
are available to conduct such evaluations. We also show that these techniques
produce substantially different results than using structural measures and
synthetic data.
