Homotopic Policy Mirror Descent: Policy Convergence, Implicit
  Regularization, and Improved Sample Complexity
  We propose the homotopic policy mirror descent (HPMD) method for solving
discounted, infinite horizon MDPs with finite state and action space, and study
its convergence properties. We report several findings that seem to be new in
the literature of policy gradient methods: (1) HPMD exhibits global linear
convergence of the value optimality gap, and local superlinear convergence of
both the policy and optimality gap with order $\gamma^{-2}$. The superlinear
convergence takes effect after no more than $\mathcal{O}(\log(1/\Delta^*))$
number of iterations, where $\Delta^*$ is defined via a gap quantity associated
with the optimal state-action value function; (2) HPMD also exhibits
last-iterate convergence of the policy, with the limiting policy corresponding
exactly to the optimal policy with the maximal entropy for every state. No
regularization is added to the optimization objective and hence the second
observation arises solely as an algorithmic property of the homotopic policy
gradient method; (3) The last-iterate convergence of HPMD holds for a much
broader class of decomposable distance-generating functions, including the
$p$-th power of $\ell_p$-norm and the negative Tsallis entropy. As a byproduct
of the analysis, we also discover the finite-time exact convergence of HPMD
with these divergences, and show that HPMD continues converging to the limiting
policy even if the current policy is already optimal; (4) For the stochastic
HPMD method, we further demonstrate that a better than
$\tilde{\mathcal{O}}(|\mathcal{S}| |\mathcal{A}| / \epsilon^2)$ sample
complexity for small optimality gap $\epsilon$ holds with high probability,
when assuming a generative model for policy evaluation.
