Interpolation and Regularization for Causal Learning
  We study the problem of learning causal models from observational data
through the lens of interpolation and its counterpart -- regularization. A
large volume of recent theoretical, as well as empirical work, suggests that,
in highly complex model classes, interpolating estimators can have good
statistical generalization properties and can even be optimal for statistical
learning. Motivated by an analogy between statistical and causal learning
recently highlighted by Janzing (2019), we investigate whether interpolating
estimators can also learn good causal models. To this end, we consider a simple
linearly confounded model and derive precise asymptotics for the *causal risk*
of the min-norm interpolator and ridge-regularized regressors in the
high-dimensional regime. Under the principle of independent causal mechanisms,
a standard assumption in causal learning, we find that interpolators cannot be
optimal and causal learning requires stronger regularization than statistical
learning. This resolves a recent conjecture in Janzing (2019). Beyond this
assumption, we find a larger range of behavior that can be precisely
characterized with a new measure of *confounding strength*. If the confounding
strength is negative, causal learning requires weaker regularization than
statistical learning, interpolators can be optimal, and the optimal
regularization can even be negative. If the confounding strength is large, the
optimal regularization is infinite, and learning from observational data is
actively harmful.
