Theoretical Guarantees of Fictitious Discount Algorithms for Episodic
  Reinforcement Learning and Global Convergence of Policy Gradient Methods
  When designing algorithms for finite-time-horizon episodic reinforcement
learning problems, a common approach is to introduce a fictitious discount
factor and use stationary policies for approximations. Empirically, it has been
shown that the fictitious discount factor helps reduce variance, and stationary
policies serve to save the per-iteration computational cost. Theoretically,
however, there is no existing work on convergence analysis for algorithms with
this fictitious discount recipe. This paper takes the first step towards
analyzing these algorithms. It focuses on two vanilla policy gradient (VPG)
variants: the first being a widely used variant with discounted advantage
estimations (DAE), the second with an additional fictitious discount factor in
the score functions of the policy gradient estimators. Non-asymptotic
convergence guarantees are established for both algorithms, and the additional
discount factor is shown to reduce the bias introduced in DAE and thus improve
the algorithm convergence asymptotically. A key ingredient of our analysis is
to connect three settings of Markov decision processes (MDPs): the
finite-time-horizon, the average reward and the discounted settings. To our
best knowledge, this is the first theoretical guarantee on fictitious discount
algorithms for the episodic reinforcement learning of finite-time-horizon MDPs,
which also leads to the (first) global convergence of policy gradient methods
for finite-time-horizon episodic reinforcement learning.
