Neural Networks for Learning Counterfactual G-Invariances from Single
  Environments
  Despite -- or maybe because of -- their astonishing capacity to fit data,
neural networks are believed to have difficulties extrapolating beyond training
data distribution. This work shows that, for extrapolations based on finite
transformation groups, a model's inability to extrapolate is unrelated to its
capacity. Rather, the shortcoming is inherited from a learning hypothesis:
Examples not explicitly observed with infinitely many training examples have
underspecified outcomes in the learner's model. In order to endow neural
networks with the ability to extrapolate over group transformations, we
introduce a learning framework counterfactually-guided by the learning
hypothesis that any group invariance to (known) transformation groups is
mandatory even without evidence, unless the learner deems it inconsistent with
the training data. Unlike existing invariance-driven methods for
(counterfactual) extrapolations, this framework allows extrapolations from a
single environment. Finally, we introduce sequence and image extrapolation
tasks that validate our framework and showcase the shortcomings of traditional
approaches.
