Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and
  Mean-Field Q-Learning
  We study infinite horizon discounted Mean Field Control (MFC) problems with
common noise through the lens of Mean Field Markov Decision Processes (MFMDP).
We allow the agents to use actions that are randomized not only at the
individual level but also at the level of the population. This common
randomization allows us to establish connections between both closed-loop and
open-loop policies for MFC and Markov policies for the MFMDP. In particular, we
show that there exists an optimal closed-loop policy for the original MFC.
Building on this framework and the notion of state-action value function, we
then propose reinforcement learning (RL) methods for such problems, by adapting
existing tabular and deep RL methods to the mean-field setting. The main
difficulty is the treatment of the population state, which is an input of the
policy and the value function. We provide convergence guarantees for tabular
algorithms based on discretizations of the simplex. Neural network based
algorithms are more suitable for continuous spaces and allow us to avoid
discretizing the mean field state space. Numerical examples are provided.
