A New Framework for Multi-Agent Reinforcement Learning -- Centralized
  Training and Exploration with Decentralized Execution via Policy Distillation
  Deep reinforcement learning (DRL) is a booming area of artificial
intelligence. Many practical applications of DRL naturally involve more than
one collaborative learners, making it important to study DRL in a multi-agent
context. Previous research showed that effective learning in complex
multi-agent systems demands for highly coordinated environment exploration
among all the participating agents. Many researchers attempted to cope with
this challenge through learning centralized value functions. However, the
common strategy for every agent to learn their local policies directly often
fail to nurture strong inter-agent collaboration and can be sample inefficient
whenever agents alter their communication channels. To address these issues, we
propose a new framework known as centralized training and exploration with
decentralized execution via policy distillation. Guided by this framework and
the maximum-entropy learning technique, we will first train agents' policies
with shared global component to foster coordinated and effective learning.
Locally executable policies will be derived subsequently from the trained
global policies via policy distillation. Experiments show that our new
framework and algorithm can achieve significantly better performance and higher
sample efficiency than a cutting-edge baseline on several multi-agent DRL
benchmarks.
