Policy Evaluation and Seeking for Multi-Agent Reinforcement Learning via
  Best Response
  This paper introduces two metrics (cycle-based and memory-based metrics),
grounded on a dynamical game-theoretic solution concept called sink
equilibrium, for the evaluation, ranking, and computation of policies in
multi-agent learning. We adopt strict best response dynamics (SBRD) to model
selfish behaviors at a meta-level for multi-agent reinforcement learning. Our
approach can deal with dynamical cyclical behaviors (unlike approaches based on
Nash equilibria and Elo ratings), and is more compatible with single-agent
reinforcement learning than alpha-rank which relies on weakly better responses.
We first consider settings where the difference between largest and second
largest underlying metric has a known lower bound. With this knowledge we
propose a class of perturbed SBRD with the following property: only policies
with maximum metric are observed with nonzero probability for a broad class of
stochastic games with finite memory. We then consider settings where the lower
bound for the difference is unknown. For this setting, we propose a class of
perturbed SBRD such that the metrics of the policies observed with nonzero
probability differ from the optimal by any given tolerance. The proposed
perturbed SBRD addresses the opponent-induced non-stationarity by fixing the
strategies of others for the learning agent, and uses empirical game-theoretic
analysis to estimate payoffs for each strategy profile obtained due to the
perturbation.
