Linear convergence of a policy gradient method for finite horizon
  continuous time stochastic control problems
  Despite its popularity in the reinforcement learning community, a provably
convergent policy gradient method for general continuous space-time stochastic
control problems has been elusive. This paper closes the gap by proposing a
proximal gradient algorithm for feedback controls of finite-time horizon
stochastic control problems. The state dynamics are continuous time nonlinear
diffusions with controlled drift and possibly degenerate noise, and the
objectives are nonconvex in the state and nonsmooth in the control. We prove
under suitable conditions that the algorithm converges linearly to a stationary
point of the control problem, and is stable with respect to policy updates by
approximate gradient steps. The convergence result justifies the recent
reinforcement learning heuristics that adding entropy regularization or a
fictitious discount factor to the optimization objective accelerates the
convergence of policy gradient methods. The proof exploits careful regularity
estimates of backward stochastic differential equations.
