Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn
  an Adaptive Sparse Communication Graph
  The complexity of multiagent reinforcement learning (MARL) in multiagent
systems increases exponentially with respect to the agent number. This
scalability issue prevents MARL from being applied in large-scale multiagent
systems. However, one critical feature in MARL that is often neglected is that
the interactions between agents are quite sparse. Without exploiting this
sparsity structure, existing works aggregate information from all of the agents
and thus have a high sample complexity. To address this issue, we propose an
adaptive sparse attention mechanism by generalizing a sparsity-inducing
activation function. Then a sparse communication graph in MARL is learned by
graph neural networks based on this new attention mechanism. Through this
sparsity structure, the agents can communicate in an effective as well as
efficient way via only selectively attending to agents that matter the most and
thus the scale of the MARL problem is reduced with little optimality
compromised. Comparative results show that our algorithm can learn an
interpretable sparse structure and outperforms previous works by a significant
margin on applications involving a large-scale multiagent system.
