RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement
  Learning Agents
  Current value-based multi-agent reinforcement learning methods optimize
individual Q values to guide individuals' behaviours via centralized training
with decentralized execution (CTDE). However, such expected, i.e.,
risk-neutral, Q value is not sufficient even with CTDE due to the randomness of
rewards and the uncertainty in environments, which causes the failure of these
methods to train coordinating agents in complex environments. To address these
issues, we propose RMIX, a novel cooperative MARL method with the Conditional
Value at Risk (CVaR) measure over the learned distributions of individuals' Q
values. Specifically, we first learn the return distributions of individuals to
analytically calculate CVaR for decentralized execution. Then, to handle the
temporal nature of the stochastic outcomes during executions, we propose a
dynamic risk level predictor for risk level tuning. Finally, we optimize the
CVaR policies with CVaR values used to estimate the target in TD error during
centralized training and the CVaR values are used as auxiliary local rewards to
update the local distribution via Quantile Regression loss. Empirically, we
show that our method significantly outperforms state-of-the-art methods on
challenging StarCraft II tasks, demonstrating enhanced coordination and
improved sample efficiency.
