Learning Nearly Decomposable Value Functions Via Communication
  Minimization
  Reinforcement learning encounters major challenges in multi-agent settings,
such as scalability and non-stationarity. Recently, value function
factorization learning emerges as a promising way to address these challenges
in collaborative multi-agent systems. However, existing methods have been
focusing on learning fully decentralized value functions, which are not
efficient for tasks requiring communication. To address this limitation, this
paper presents a novel framework for learning nearly decomposable Q-functions
(NDQ) via communication minimization, with which agents act on their own most
of the time but occasionally send messages to other agents in order for
effective coordination. This framework hybridizes value function factorization
learning and communication learning by introducing two information-theoretic
regularizers. These regularizers are maximizing mutual information between
agents' action selection and communication messages while minimizing the
entropy of messages between agents. We show how to optimize these regularizers
in a way that is easily integrated with existing value function factorization
methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit
micromanagement benchmark, our framework significantly outperforms baseline
methods and allows us to cut off more than $80\%$ of communication without
sacrificing the performance. The videos of our experiments are available at
https://sites.google.com/view/ndq.
