Data Distributional Properties Drive Emergent In-Context Learning in
  Transformers
  Large transformer-based models are able to perform in-context few-shot
learning, without being explicitly trained for it. This observation raises the
question: what aspects of the training regime lead to this emergent behavior?
Here, we show that this behavior is driven by the distributions of the training
data itself. In-context learning emerges when the training data exhibits
particular distributional properties such as burstiness (items appear in
clusters rather than being uniformly distributed over time) and having large
numbers of rarely occurring classes. In-context learning also emerges more
strongly when item meanings or interpretations are dynamic rather than fixed.
These properties are exemplified by natural language, but are also inherent to
naturalistic data in a wide range of other domains. They also depart
significantly from the uniform, i.i.d. training distributions typically used
for standard supervised learning. In our initial experiments, we found that
in-context learning traded off against more conventional weight-based learning,
and models were unable to achieve both simultaneously. However, our later
experiments uncovered that the two modes of learning could co-exist in a single
model when it was trained on data following a skewed Zipfian distribution --
another common property of naturalistic data, including language. In further
experiments, we found that naturalistic data distributions were only able to
elicit in-context learning in transformers, and not in recurrent models. In
sum, our findings indicate how the transformer architecture works together with
particular properties of the training data to drive the intriguing emergent
in-context learning behaviour of large language models, and how future work
might encourage both in-context and in-weights learning in domains beyond
language.
