Fixed-Horizon Temporal Difference Methods for Stable Reinforcement
  Learning
  We explore fixed-horizon temporal difference (TD) methods, reinforcement
learning algorithms for a new kind of value function that predicts the sum of
rewards over a $\textit{fixed}$ number of future time steps. To learn the value
function for horizon $h$, these algorithms bootstrap from the value function
for horizon $h-1$, or some shorter horizon. Because no value function
bootstraps from itself, fixed-horizon methods are immune to the stability
problems that plague other off-policy TD methods using function approximation
(also known as "the deadly triad"). Although fixed-horizon methods require the
storage of additional value functions, this gives the agent additional
predictive power, while the added complexity can be substantially reduced via
parallel updates, shared weights, and $n$-step bootstrapping. We show how to
use fixed-horizon value functions to solve reinforcement learning problems
competitively with methods such as Q-learning that learn conventional value
functions. We also prove convergence of fixed-horizon temporal difference
methods with linear and general function approximation. Taken together, our
results establish fixed-horizon TD methods as a viable new way of avoiding the
stability problems of the deadly triad.
