The Inductive Bias of In-Context Learning: Rethinking Pretraining
  Example Design
  Pretraining Neural Language Models (NLMs) over a large corpus involves
chunking the text into training examples, which are contiguous text segments of
sizes processable by the neural architecture. We highlight a bias introduced by
this common practice: we prove that the pretrained NLM can model much stronger
dependencies between text segments that appeared in the same training example,
than it can between text segments that appeared in different training examples.
This intuitive result has a twofold role. First, it formalizes the motivation
behind a broad line of recent successful NLM training heuristics, proposed for
the pretraining and fine-tuning stages, which do not necessarily appear related
at first glance. Second, our result clearly indicates further improvements to
be made in NLM pretraining for the benefit of Natural Language Understanding
tasks. As an example, we propose "kNN-Pretraining": we show that including
semantically related non-neighboring sentences in the same pretraining example
yields improved sentence representations and open domain question answering
abilities. This theoretically motivated degree of freedom for pretraining
example design indicates new training schemes for self-improving
representations.
