LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning
Cooperative multi-agent reinforcement learning (MARL) has made prominent
progress in recent years. For training efficiency and scalability, most of the
MARL algorithms make all agents share the same policy or value network.
However, in many complex multi-agent tasks, different agents are expected to
possess specific abilities to handle different subtasks. In those scenarios,
sharing parameters indiscriminately may lead to similar behavior across all
agents, which will limit the exploration efficiency and degrade the final
performance. To balance the training complexity and the diversity of agent
behavior, we propose a novel framework to learn dynamic subtask assignment
(LDSA) in cooperative MARL. Specifically, we first introduce a subtask encoder
to construct a vector representation for each subtask according to its
identity. To reasonably assign agents to different subtasks, we propose an
ability-based subtask selection strategy, which can dynamically group agents
with similar abilities into the same subtask. In this way, agents dealing with
the same subtask share their learning of specific abilities and different
subtasks correspond to different specific abilities. We further introduce two
regularizers to increase the representation difference between subtasks and
stabilize the training by discouraging agents from frequently changing
subtasks, respectively. Empirical results show that LDSA learns reasonable and
effective subtask assignment for better collaboration and significantly
improves the learning performance on the challenging StarCraft II
micromanagement benchmark.