Pareto Invariant Risk Minimization
  Despite the success of invariant risk minimization (IRM) in tackling the
Out-of-Distribution generalization problem, IRM can compromise the optimality
when applied in practice. The practical variants of IRM, e.g., IRMv1, have been
shown to have significant gaps with IRM and thus could fail to capture the
invariance even in simple problems. Moreover, the optimization procedure in
IRMv1 involves two intrinsically conflicting objectives, and often requires
careful tuning for the objective weights. To remedy the above issues, we
reformulate IRM as a multi-objective optimization problem, and propose a new
optimization scheme for IRM, called PAreto Invariant Risk Minimization (PAIR).
PAIR can adaptively adjust the optimization direction under the objective
conflicts. Furthermore, we show PAIR can empower the practical IRM variants to
overcome the barriers with the original IRM when provided with proper guidance.
We conduct experiments with ColoredMNIST to confirm our theory and the
effectiveness of PAIR.
