A nearly Blackwell-optimal policy gradient method
  For continuing environments, reinforcement learning methods commonly maximize
a discounted reward criterion with discount factor close to 1 in order to
approximate the steady-state reward (the gain). However, such a criterion only
considers the long-run performance, ignoring the transient behaviour. In this
work, we develop a policy gradient method that optimizes the gain, then the
bias (which indicates the transient performance and is important to capably
select from policies with equal gain). We derive expressions that enable
sampling for the gradient of the bias, and its preconditioning Fisher matrix.
We further propose an algorithm that solves the corresponding bi-level
optimization using a logarithmic barrier. Experimental results provide insights
into the fundamental mechanisms of our proposal.
