\subsection{Training and evaluation}
\label{ss:trainingevaluation}

For all tasks, we follow the same training procedure. We train with Adam optimizer~\citep{kingma2014} for $3000$ epochs. The learning rate starts at $10^{-5}$ and increases for the first $1000$ warm-up updates till reaching $10^{-3}$. We include all available examples in a single batch. We use teacher forcing~\citep{Goodfellow2016} and set the dropout probability to 0.5. 
For each learner, we perform training and evaluation 100 times, changing random seeds.  
At generation, to calculate FPA, we select the next token greedily. We use the model implementations from fairseq~\citep{Ott2019fairseq}. 

As discussed in Section~\ref{ss:searching}, when calculating $L$, we use the training examples as the first transmitted block at $t=1$.
In \textit{Count-or-Memorization} and \textit{Add-or-Multiply} this block contains one example, in \textit{Hierarchical-or-Linear} it has 4 examples, and in \textit{Composition-or-Memorization} it has $N+M$ examples. Next, we transmit examples obtained from the candidate rules 
in a randomized order, by blocks of size 1, 1, 1, and 4 for \textit{Count-or-Memorization}, \textit{Add-or-Multiply}, \textit{Composition-or-Memorization}, and \textit{Hierarchical-or-Linear} respectively. 
At each step, the learner is re-trained from the same initialization, using the procedure and  hyper-parameters as discussed above. 

Our training setup is typical for seq2seq training. It does not include any additional pressure towards description length minimization. Description length is solely used as a measure of inductive biases.
{"config": "arxiv", "file": "2006.14953/trainingregime.tex"}