Empirical or Invariant Risk Minimization? A Sample Complexity
  Perspective
  Recently, invariant risk minimization (IRM) was proposed as a promising
solution to address out-of-distribution (OOD) generalization. However, it is
unclear when IRM should be preferred over the widely-employed empirical risk
minimization (ERM) framework. In this work, we analyze both these frameworks
from the perspective of sample complexity, thus taking a firm step towards
answering this important question. We find that depending on the type of data
generation mechanism, the two approaches might have very different finite
sample and asymptotic behavior. For example, in the covariate shift setting we
see that the two approaches not only arrive at the same asymptotic solution,
but also have similar finite sample behavior with no clear winner. For other
distribution shifts such as those involving confounders or anti-causal
variables, however, the two approaches arrive at different asymptotic solutions
where IRM is guaranteed to be close to the desired OOD solutions in the finite
sample regime, while ERM is biased even asymptotically. We further investigate
how different factors -- the number of environments, complexity of the model,
and IRM penalty weight -- impact the sample complexity of IRM in relation to
its distance from the OOD solutions
