Reinforcement Learning in Non-Stationary Discrete-Time Linear-Quadratic
  Mean-Field Games
  In this paper, we study large population multi-agent reinforcement learning
(RL) in the context of discrete-time linear-quadratic mean-field games
(LQ-MFGs). Our setting differs from most existing work on RL for MFGs, in that
we consider a non-stationary MFG over an infinite horizon. We propose an
actor-critic algorithm to iteratively compute the mean-field equilibrium (MFE)
of the LQ-MFG. There are two primary challenges: i) the non-stationarity of the
MFG induces a linear-quadratic tracking problem, which requires solving a
backwards-in-time (non-causal) equation that cannot be solved by standard
(causal) RL algorithms; ii) Many RL algorithms assume that the states are
sampled from the stationary distribution of a Markov chain (MC), that is, the
chain is already mixed, an assumption that is not satisfied for real data
sources. We first identify that the mean-field trajectory follows linear
dynamics, allowing the problem to be reformulated as a linear quadratic
Gaussian problem. Under this reformulation, we propose an actor-critic
algorithm that allows samples to be drawn from an unmixed MC. Finite-sample
convergence guarantees for the algorithm are then provided. To characterize the
performance of our algorithm in multi-agent RL, we have developed an error
bound with respect to the Nash equilibrium of the finite-population game.
