Collective eXplainable AI: Explaining Cooperative Strategies and Agent
  Contribution in Multiagent Reinforcement Learning with Shapley Values
  While Explainable Artificial Intelligence (XAI) is increasingly expanding
more areas of application, little has been applied to make deep Reinforcement
Learning (RL) more comprehensible. As RL becomes ubiquitous and used in
critical and general public applications, it is essential to develop methods
that make it better understood and more interpretable. This study proposes a
novel approach to explain cooperative strategies in multiagent RL using Shapley
values, a game theory concept used in XAI that successfully explains the
rationale behind decisions taken by Machine Learning algorithms. Through
testing common assumptions of this technique in two cooperation-centered
socially challenging multi-agent environments environments, this article argues
that Shapley values are a pertinent way to evaluate the contribution of players
in a cooperative multi-agent RL context. To palliate the high overhead of this
method, Shapley values are approximated using Monte Carlo sampling.
Experimental results on Multiagent Particle and Sequential Social Dilemmas show
that Shapley values succeed at estimating the contribution of each agent. These
results could have implications that go beyond games in economics, (e.g., for
non-discriminatory decision making, ethical and responsible AI-derived
decisions or policy making under fairness constraints). They also expose how
Shapley values only give general explanations about a model and cannot explain
a single run, episode nor justify precise actions taken by agents. Future work
should focus on addressing these critical aspects.
