Scalable Deep Reinforcement Learning Algorithms for Mean Field Games
  Mean Field Games (MFGs) have been introduced to efficiently approximate games
with very large populations of strategic agents. Recently, the question of
learning equilibria in MFGs has gained momentum, particularly using model-free
reinforcement learning (RL) methods. One limiting factor to further scale up
using RL is that existing algorithms to solve MFGs require the mixing of
approximated quantities such as strategies or $q$-values. This is non-trivial
in the case of non-linear function approximation that enjoy good generalization
properties, e.g. neural networks. We propose two methods to address this
shortcoming. The first one learns a mixed strategy from distillation of
historical data into a neural network and is applied to the Fictitious Play
algorithm. The second one is an online mixing method based on regularization
that does not require memorizing historical data or previous estimates. It is
used to extend Online Mirror Descent. We demonstrate numerically that these
methods efficiently enable the use of Deep RL algorithms to solve various MFGs.
In addition, we show that these methods outperform SotA baselines from the
literature.
