Multi-timescale Nexting in a Reinforcement Learning Robot
  The term "nexting" has been used by psychologists to refer to the propensity
of people and many other animals to continually predict what will happen next
in an immediate, local, and personal sense. The ability to "next" constitutes a
basic kind of awareness and knowledge of one's environment. In this paper we
present results with a robot that learns to next in real time, predicting
thousands of features of the world's state, including all sensory inputs, at
timescales from 0.1 to 8 seconds. This was achieved by treating each state
feature as a reward-like target and applying temporal-difference methods to
learn a corresponding value function with a discount rate corresponding to the
timescale. We show that two thousand predictions, each dependent on six
thousand state features, can be learned and updated online at better than 10Hz
on a laptop computer, using the standard TD(lambda) algorithm with linear
function approximation. We show that this approach is efficient enough to be
practical, with most of the learning complete within 30 minutes. We also show
that a single tile-coded feature representation suffices to accurately predict
many different signals at a significant range of timescales. Finally, we show
that the accuracy of our learned predictions compares favorably with the
optimal off-line solution.
