A Greedy Approach to Adapting the Trace Parameter for Temporal
  Difference Learning
  One of the main obstacles to broad application of reinforcement learning
methods is the parameter sensitivity of our core learning algorithms. In many
large-scale applications, online computation and function approximation
represent key strategies in scaling up reinforcement learning algorithms. In
this setting, we have effective and reasonably well understood algorithms for
adapting the learning-rate parameter, online during learning. Such
meta-learning approaches can improve robustness of learning and enable
specialization to current task, improving learning speed. For
temporal-difference learning algorithms which we study here, there is yet
another parameter, $\lambda$, that similarly impacts learning speed and
stability in practice. Unfortunately, unlike the learning-rate parameter,
$\lambda$ parametrizes the objective function that temporal-difference methods
optimize. Different choices of $\lambda$ produce different fixed-point
solutions, and thus adapting $\lambda$ online and characterizing the
optimization is substantially more complex than adapting the learning-rate
parameter. There are no meta-learning method for $\lambda$ that can achieve (1)
incremental updating, (2) compatibility with function approximation, and (3)
maintain stability of learning under both on and off-policy sampling. In this
paper we contribute a novel objective function for optimizing $\lambda$ as a
function of state rather than time. We derive a new incremental, linear
complexity $\lambda$-adaption algorithm that does not require offline batch
updating or access to a model of the world, and present a suite of experiments
illustrating the practicality of our new algorithm in three different settings.
Taken together, our contributions represent a concrete step towards black-box
application of temporal-difference learning methods in real world problems.
