\section{Introduction}
\label{s:intro}


Sequence-to-sequence (seq2seq) learners~\citep{Sutskever2014} demonstrated remarkable performance in machine translation, story generation, and open-domain dialog~\citep{Sutskever2014,Fan2018,Adiwardana2020}. Yet, these models have been criticized for requiring a tremendous amount of data and being unable to generalize systematically~\citep{dupoux2018,loula2018,Lake2017,Bastings2018}.  In contrast, humans rely on their inductive biases to generalize from a limited amount of data~\citep{chomsky1965,Lake2019human}. Due to  the centrality of humans' biases in language learning, several works have studied inductive biases of seq2seq models and connected their poor generalization to the lack of the ``right'' biases \citep{Lake2017,Lake2019human}. 

In this work, we focus on studying inductive biases of seq2seq models. We start from an observation that, generally, multiple explanations can be consistent with a limited training set, each leading to different predictions on unseen data. A learner might prefer one type of explanations over another in a systematic way,
as a result of its \textit{inductive biases}~\citep{Ritter2017,Feinman2018}. 

To illustrate the setup we work in, consider a quiz-like question: if $f(3)$ maps to  $6$, what does $f(4)$ map to? The ``training'' example is consistent with the following answers: 6 $\left(f(x) \equiv 6\right)$; $7$ $\left(f(x) = x + 3\right)$; $8$ $\left(f(x) = 2 \cdot x\right)$; any number $z$, since we always can construct a function such that $f(3) = 6$ and $f(4) = z$. By analyzing the learner's output on this new input, we can infer its biases.

This example demonstrates how biases of learners are studied through the lenses of the poverty of the stimulus principle~\citep{chomsky1965,chomsky1980}: if nothing in the training data indicates that a learner should generalize in a certain way, but it does nonetheless, then this is due to the biases of the learner.
Inspired by the work of \cite{Zhang2019} in the image domain, we take this principle to the extreme and study biases of seq2seq learners in the regime of very few training examples, often as little as one. Under this setup, we propose four new synthetic tasks that probe seq2seq learners' preferences to memorization-, arithmetic-, hierarchical- and compositional-based ``reasoning''. 

Next, we connect to the ideas of Solomonoff's theory of induction~\citep{Solomonoff1964} and Minimal Description Length~\citep{Rissanen1978,Grunwald2004} and propose to use description length, under a learner, as a principled measure of its inductive biases.

Our experimental study\footnote{Code used in the experiments can be found at \url{https://github.com/facebookresearch/FIND}.} shows that the standard seq2seq learners have strikingly different inductive biases. We find that LSTM-based learners can learn non-trivial counting-, multiplication-, and addition-based rules from as little as one example. CNN-based seq2seq learners would prefer linear over hierarchical generalizations, while LSTM-based ones and Transformers would do just the opposite. 

When investigating the compositional reasoning, description length proved to be a sensitive measure.
Equipped with it, we found that CNN-, and, to a lesser degree, LSTM-based learners prefer compositional generalization over memorization when provided with enough composite examples. In turn, Transformers show a strong bias toward memorization.
{"config": "arxiv", "file": "2006.14953/intro.tex"}