On Invariance Penalties for Risk Minimization
  The Invariant Risk Minimization (IRM) principle was first proposed by
Arjovsky et al. [2019] to address the domain generalization problem by
leveraging data heterogeneity from differing experimental conditions.
Specifically, IRM seeks to find a data representation under which an optimal
classifier remains invariant across all domains. Despite the conceptual appeal
of IRM, the effectiveness of the originally proposed invariance penalty has
recently been brought into question. In particular, there exists
counterexamples for which that invariance penalty can be arbitrarily small for
non-invariant data representations. We propose an alternative invariance
penalty by revisiting the Gramian matrix of the data representation. We discuss
the role of its eigenvalues in the relationship between the risk and the
invariance penalty, and demonstrate that it is ill-conditioned for said
counterexamples. The proposed approach is guaranteed to recover an invariant
representation for linear settings under mild non-degeneracy conditions. Its
effectiveness is substantiated by experiments on DomainBed and
InvarianceUnitTest, two extensive test beds for domain generalization.
