Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint
  Concave Utility Reinforcement Learning (CURL) extends RL from linear to
concave utilities in the occupancy measure induced by the agent's policy. This
encompasses not only RL but also imitation learning and exploration, among
others. Yet, this more general paradigm invalidates the classical Bellman
equations, and calls for new algorithms. Mean-field Games (MFGs) are a
continuous approximation of many-agent RL. They consider the limit case of a
continuous distribution of identical agents, anonymous with symmetric
interests, and reduce the problem to the study of a single representative agent
in interaction with the full population. Our core contribution consists in
showing that CURL is a subclass of MFGs. We think this important to bridge
together both communities. It also allows to shed light on aspects of both
fields: we show the equivalence between concavity in CURL and monotonicity in
the associated MFG, between optimality conditions in CURL and Nash equilibrium
in MFG, or that Fictitious Play (FP) for this class of MFGs is simply
Frank-Wolfe, bringing the first convergence rate for discrete-time FP for MFGs.
We also experimentally demonstrate that, using algorithms recently introduced
for solving MFGs, we can address the CURL problem more efficiently.
