Learning Agent State Online with Recurrent Generate-and-Test
  Learning continually and online from a continuous stream of data is
challenging, especially for a reinforcement learning agent with sequential
data. When the environment only provides observations giving partial
information about the state of the environment, the agent must learn the agent
state based on the data stream of experience. We refer to the state learned
directly from the data stream of experience as the agent state. Recurrent
neural networks can learn the agent state, but the training methods are
computationally expensive and sensitive to the hyper-parameters, making them
unideal for online learning. This work introduces methods based on the
generate-and-test approach to learn the agent state. A generate-and-test
algorithm searches for state features by generating features and testing their
usefulness. In this process, features useful for the agent's performance on the
task are preserved, and the least useful features get replaced with newly
generated features. We study the effectiveness of our methods on two online
multi-step prediction problems. The first problem, trace conditioning, focuses
on the agent's ability to remember a cue for a prediction multiple steps into
the future. In the second problem, trace patterning, the agent needs to learn
patterns in the observation signals and remember them for future predictions.
We show that our proposed methods can effectively learn the agent state online
and produce accurate predictions.
