HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with
  Dual Coordination Mechanism
  Multi-agent reinforcement learning often suffers from the exponentially large
action space caused by a large number of agents. This paper proposes a novel
value decomposition framework HAVEN based on hierarchical reinforcement
learning for the fully cooperative multi-agent problems. To address the
instability that arises from the concurrent optimization of high-level and
low-level policies and another concurrent optimization of agents, we introduce
the dual coordination mechanism of inter-layer strategies and inter-agent
strategies. HAVEN does not require domain knowledge and pretraining, and can be
applied to any value decomposition variant. Our method achieves desirable
results on different decentralized partially observable Markov decision process
domains and offers an efficient solution to partition the decision space
implicitly.
