Towards a theory of out-of-distribution learning
  What is learning? 20$^{st}$ century formalizations of learning theory --
which precipitated revolutions in artificial intelligence -- focus primarily on
$\mathit{in-distribution}$ learning, that is, learning under the assumption
that the training data are sampled from the same distribution as the evaluation
distribution. This assumption renders these theories inadequate for
characterizing 21$^{st}$ century real world data problems, which are typically
characterized by evaluation distributions that differ from the training data
distributions (referred to as out-of-distribution learning). We therefore make
a small change to existing formal definitions of learnability by relaxing that
assumption. We then introduce $\mathbf{learning\ efficiency}$ (LE) to quantify
the amount a learner is able to leverage data for a given problem, regardless
of whether it is an in- or out-of-distribution problem. We then define and
prove the relationship between generalized notions of learnability, and show
how this framework is sufficiently general to characterize transfer, multitask,
meta, continual, and lifelong learning. We hope this unification helps bridge
the gap between empirical practice and theoretical guidance in real world
problems. Finally, because biological learning continues to outperform machine
learning algorithms on certain OOD challenges, we discuss the limitations of
this framework vis-\'a-vis its ability to formalize biological learning,
suggesting multiple avenues for future research.
