Entropy Minimization In Emergent Languages
  There is growing interest in studying the languages that emerge when neural
agents are jointly trained to solve tasks requiring communication through a
discrete channel. We investigate here the information-theoretic complexity of
such languages, focusing on the basic two-agent, one-exchange setup. We find
that, under common training procedures, the emergent languages are subject to
an entropy minimization pressure that has also been detected in human language,
whereby the mutual information between the communicating agent's inputs and the
messages is minimized, within the range afforded by the need for successful
communication. That is, emergent languages are (nearly) as simple as the task
they are developed for allow them to be. This pressure is amplified as we
increase communication channel discreteness. Further, we observe that stronger
discrete-channel-driven entropy minimization leads to representations with
increased robustness to overfitting and adversarial attacks. We conclude by
discussing the implications of our findings for the study of natural and
artificial communication systems.
