Locality Matters: A Scalable Value Decomposition Approach for
  Cooperative Multi-Agent Reinforcement Learning
  Cooperative multi-agent reinforcement learning (MARL) faces significant
scalability issues due to state and action spaces that are exponentially large
in the number of agents. As environments grow in size, effective credit
assignment becomes increasingly harder and often results in infeasible learning
times. Still, in many real-world settings, there exist simplified underlying
dynamics that can be leveraged for more scalable solutions. In this work, we
exploit such locality structures effectively whilst maintaining global
cooperation. We propose a novel, value-based multi-agent algorithm called
LOMAQ, which incorporates local rewards in the Centralized Training
Decentralized Execution paradigm. Additionally, we provide a direct reward
decomposition method for finding these local rewards when only a global signal
is provided. We test our method empirically, showing it scales well compared to
other methods, significantly improving performance and convergence speed.
