Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy
  Gradient Methods with Entropy Regularization
  Entropy regularization is an efficient technique for encouraging exploration
and preventing a premature convergence of (vanilla) policy gradient methods in
reinforcement learning (RL). However, the theoretical understanding of entropy
regularized RL algorithms has been limited. In this paper, we revisit the
classical entropy regularized policy gradient methods with the soft-max policy
parametrization, whose convergence has so far only been established assuming
access to exact gradient oracles. To go beyond this scenario, we propose the
first set of (nearly) unbiased stochastic policy gradient estimators with
trajectory-level entropy regularization, with one being an unbiased visitation
measure-based estimator and the other one being a nearly unbiased yet more
practical trajectory-based estimator. We prove that although the estimators
themselves are unbounded in general due to the additional logarithmic policy
rewards introduced by the entropy term, the variances are uniformly bounded. We
then propose a two-phase stochastic policy gradient (PG) algorithm that uses a
large batch size in the first phase to overcome the challenge of the stochastic
approximation due to the non-coercive landscape, and uses a small batch size in
the second phase by leveraging the curvature information around the optimal
policy. We establish a global optimality convergence result and a sample
complexity of $\widetilde{\mathcal{O}}(\frac{1}{\epsilon^2})$ for the proposed
algorithm. Our result is the first global convergence and sample complexity
results for the stochastic entropy-regularized vanilla PG method.
