Out-of-distribution Prediction with Invariant Risk Minimization: The
  Limitation and An Effective Fix
  This work considers the out-of-distribution (OOD) prediction problem where
(1)~the training data are from multiple domains and (2)~the test domain is
unseen in the training. DNNs fail in OOD prediction because they are prone to
pick up spurious correlations. Recently, Invariant Risk Minimization (IRM) is
proposed to address this issue. Its effectiveness has been demonstrated in the
colored MNIST experiment. Nevertheless, we find that the performance of IRM can
be dramatically degraded under \emph{strong $\Lambda$ spuriousness} -- when the
spurious correlation between the spurious features and the class label is
strong due to the strong causal influence of their common cause, the domain
label, on both of them (see Fig. 1). In this work, we try to answer the
questions: why does IRM fail in the aforementioned setting? Why does IRM work
for the original colored MNIST dataset? How can we fix this problem of IRM?
Then, we propose a simple and effective approach to fix the problem of IRM. We
combine IRM with conditional distribution matching to avoid a specific type of
spurious correlation under strong $\Lambda$ spuriousness. Empirically, we
design a series of semi synthetic datasets -- the colored MNIST plus, which
exposes the problems of IRM and demonstrates the efficacy of the proposed
method.
