Temporal Difference Updating without a Learning Rate
  We derive an equation for temporal difference learning from statistical
principles. Specifically, we start with the variational principle and then
bootstrap to produce an updating rule for discounted state value estimates. The
resulting equation is similar to the standard equation for temporal difference
learning with eligibility traces, so called TD(lambda), however it lacks the
parameter alpha that specifies the learning rate. In the place of this free
parameter there is now an equation for the learning rate that is specific to
each state transition. We experimentally test this new learning rule against
TD(lambda) and find that it offers superior performance in various settings.
Finally, we make some preliminary investigations into how to extend our new
temporal difference algorithm to reinforcement learning. To do this we combine
our update equation with both Watkins' Q(lambda) and Sarsa(lambda) and find
that it again offers superior performance without a learning rate parameter.
