Approximate discounting-free policy evaluation from transient and
  recurrent states
  In order to distinguish policies that prescribe good from bad actions in
transient states, we need to evaluate the so-called bias of a policy from
transient states. However, we observe that most (if not all) works in
approximate discounting-free policy evaluation thus far are developed for
estimating the bias solely from recurrent states. We therefore propose a system
of approximators for the bias (specifically, its relative value) from transient
and recurrent states. Its key ingredient is a seminorm LSTD (least-squares
temporal difference), for which we derive its minimizer expression that enables
approximation by sampling required in model-free reinforcement learning. This
seminorm LSTD also facilitates the formulation of a general unifying procedure
for LSTD-based policy value approximators. Experimental results validate the
effectiveness of our proposed method.
