Smoothed functional-based gradient algorithms for off-policy
  reinforcement learning: A non-asymptotic viewpoint
  We propose two policy gradient algorithms for solving the problem of control
in an off-policy reinforcement learning (RL) context. Both algorithms
incorporate a smoothed functional (SF) based gradient estimation scheme. The
first algorithm is a straightforward combination of importance sampling-based
off-policy evaluation with SF-based gradient estimation. The second algorithm,
inspired by the stochastic variance-reduced gradient (SVRG) algorithm,
incorporates variance reduction in the update iteration. For both algorithms,
we derive non-asymptotic bounds that establish convergence to an approximate
stationary point. From these results, we infer that the first algorithm
converges at a rate that is comparable to the well-known REINFORCE algorithm in
an off-policy RL context, while the second algorithm exhibits an improved rate
of convergence.
