Capacity, Bandwidth, and Compositionality in Emergent Language Learning
  Many recent works have discussed the propensity, or lack thereof, for
emergent languages to exhibit properties of natural languages. A favorite in
the literature is learning compositionality. We note that most of those works
have focused on communicative bandwidth as being of primary importance. While
important, it is not the only contributing factor. In this paper, we
investigate the learning biases that affect the efficacy and compositionality
of emergent languages. Our foremost contribution is to explore how capacity of
a neural network impacts its ability to learn a compositional language. We
additionally introduce a set of evaluation metrics with which we analyze the
learned languages. Our hypothesis is that there should be a specific range of
model capacity and channel bandwidth that induces compositional structure in
the resulting language and consequently encourages systematic generalization.
While we empirically see evidence for the bottom of this range, we curiously do
not find evidence for the top part of the range and believe that this is an
open question for the community.
