Convergence of Policy Gradient for Entropy Regularized MDPs with Neural
  Network Approximation in the Mean-Field Regime
  We study the global convergence of policy gradient for infinite-horizon,
continuous state and action space, and entropy-regularized Markov decision
processes (MDPs). We consider a softmax policy with (one-hidden layer) neural
network approximation in a mean-field regime. Additional entropic
regularization in the associated mean-field probability measure is added, and
the corresponding gradient flow is studied in the 2-Wasserstein metric. We show
that the objective function is increasing along the gradient flow. Further, we
prove that if the regularization in terms of the mean-field measure is
sufficient, the gradient flow converges exponentially fast to the unique
stationary solution, which is the unique maximizer of the regularized MDP
objective. Lastly, we study the sensitivity of the value function along the
gradient flow with respect to regularization parameters and the initial
condition. Our results rely on the careful analysis of the non-linear
Fokker-Planck-Kolmogorov equation and extend the pioneering work of Mei et al.
2020 and Agarwal et al. 2020, which quantify the global convergence rate of
policy gradient for entropy-regularized MDPs in the tabular setting.
