On a Variance Reduction Correction of the Temporal Difference for Policy
  Evaluation in the Stochastic Continuous Setting
  This paper deals with solving continuous time, state and action optimization
problems in stochastic settings, using reinforcement learning algorithms, and
considers the policy evaluation process. We prove that standard learning
algorithms based on the discretized temporal difference are doomed to fail when
the time discretization tends to zero, because of the stochastic part. We
propose a variance-reduction correction of the temporal difference, leading to
new learning algorithms that are stable with respect to vanishing time steps.
This allows us to give theoretical guarantees of convergence of our algorithms
to the solutions of continuous stochastic optimization problems.
