General Value Function Networks
  State construction is important for learning in partially observable
environments. A general purpose strategy for state construction is to learn the
state update using a Recurrent Neural Network (RNN), which updates the internal
state using the current internal state and the most recent observation. This
internal state provides a summary of the observed sequence, to facilitate
accurate predictions and decision-making. At the same time, specifying and
training RNNs is notoriously tricky, particularly as the common strategy to
approximate gradients back in time, called truncated Back-prop Through Time
(BPTT), can be sensitive to the truncation window. Further,
domain-expertise--which can usually help constrain the function class and so
improve trainability--can be difficult to incorporate into complex recurrent
units used within RNNs. In this work, we explore how to use multi-step
predictions to constrain the RNN and incorporate prior knowledge. In
particular, we revisit the idea of using predictions to construct state and
ask: does constraining (parts of) the state to consist of predictions about the
future improve RNN trainability? We formulate a novel RNN architecture, called
a General Value Function Network (GVFN), where each internal state component
corresponds to a prediction about the future represented as a value function.
We first provide an objective for optimizing GVFNs, and derive several
algorithms to optimize this objective. We then show that GVFNs are more robust
to the truncation level, in many cases only requiring one-step gradient
updates.
