A First-Occupancy Representation for Reinforcement Learning
  Both animals and artificial agents benefit from state representations that
support rapid transfer of learning across tasks and which enable them to
efficiently traverse their environments to reach rewarding states. The
successor representation (SR), which measures the expected cumulative,
discounted state occupancy under a fixed policy, enables efficient transfer to
different reward structures in an otherwise constant Markovian environment and
has been hypothesized to underlie aspects of biological behavior and neural
activity. However, in the real world, rewards may move or only be available for
consumption once, may shift location, or agents may simply aim to reach goal
states as rapidly as possible without the constraint of artificially imposed
task horizons. In such cases, the most behaviorally-relevant representation
would carry information about when the agent was likely to first reach states
of interest, rather than how often it should expect to visit them over a
potentially infinite time span. To reflect such demands, we introduce the
first-occupancy representation (FR), which measures the expected temporal
discount to the first time a state is accessed. We demonstrate that the FR
facilitates exploration, the selection of efficient paths to desired states,
allows the agent, under certain conditions, to plan provably optimal
trajectories defined by a sequence of subgoals, and induces similar behavior to
animals avoiding threatening stimuli.
