Towards Out-Of-Distribution Generalization: A Survey
  Classic machine learning methods are built on the $i.i.d.$ assumption that
training and testing data are independent and identically distributed. However,
in real scenarios, the $i.i.d.$ assumption can hardly be satisfied, rendering
the sharp drop of classic machine learning algorithms' performances under
distributional shifts, which indicates the significance of investigating the
Out-of-Distribution generalization problem. Out-of-Distribution (OOD)
generalization problem addresses the challenging setting where the testing
distribution is unknown and different from the training. This paper serves as
the first effort to systematically and comprehensively discuss the OOD
generalization problem, from the definition, methodology, evaluation to the
implications and future directions. Firstly, we provide the formal definition
of the OOD generalization problem. Secondly, existing methods are categorized
into three parts based on their positions in the whole learning pipeline,
namely unsupervised representation learning, supervised model learning and
optimization, and typical methods for each category are discussed in detail. We
then demonstrate the theoretical connections of different categories, and
introduce the commonly used datasets and evaluation metrics. Finally, we
summarize the whole literature and raise some future directions for OOD
generalization problem. The summary of OOD generalization methods reviewed in
this survey can be found at http://out-of-distribution-generalization.com.
