On the Global Optimum Convergence of Momentum-based Policy Gradient
  Policy gradient (PG) methods are popular and efficient for large-scale
reinforcement learning due to their relative stability and incremental nature.
In recent years, the empirical success of PG methods has led to the development
of a theoretical foundation for these methods. In this work, we generalize this
line of research by studying the global convergence of stochastic PG methods
with momentum terms, which have been demonstrated to be efficient recipes for
improving PG methods. We study both the soft-max and the Fisher-non-degenerate
policy parametrizations, and show that adding a momentum improves the global
optimality sample complexity of vanilla PG methods by
$\tilde{\mathcal{O}}(\epsilon^{-1.5})$ and
$\tilde{\mathcal{O}}(\epsilon^{-1})$, respectively, where $\epsilon>0$ is the
target tolerance. Our work is the first one that obtains global convergence
results for the momentum-based PG methods. For the generic
Fisher-non-degenerate policy parametrizations, our result is the first
single-loop and finite-batch PG algorithm achieving $\tilde{O}(\epsilon^{-3})$
global optimality sample complexity. Finally, as a by-product, our methods also
provide general framework for analyzing the global convergence rates of
stochastic PG methods, which can be easily applied and extended to different PG
estimators.
