Individual-Level Inverse Reinforcement Learning for Mean Field Games
  The recent mean field game (MFG) formalism has enabled the application of
inverse reinforcement learning (IRL) methods in large-scale multi-agent
systems, with the goal of inferring reward signals that can explain
demonstrated behaviours of large populations. The existing IRL methods for MFGs
are built upon reducing an MFG to a Markov decision process (MDP) defined on
the collective behaviours and average rewards of the population. However, this
paper reveals that the reduction from MFG to MDP holds only for the fully
cooperative setting. This limitation invalidates existing IRL methods on MFGs
with non-cooperative environments. To measure more general behaviours in large
populations, we study the use of individual behaviours to infer ground-truth
reward functions for MFGs. We propose Mean Field IRL (MFIRL), the first
dedicated IRL framework for MFGs that can handle both cooperative and
non-cooperative environments. Based on this theoretically justified framework,
we develop a practical algorithm effective for MFGs with unknown dynamics. We
evaluate MFIRL on both cooperative and mixed cooperative-competitive scenarios
with many agents. Results demonstrate that MFIRL excels in reward recovery,
sample efficiency and robustness in the face of changing dynamics.
