Adapting the Function Approximation Architecture in Online Reinforcement
  Learning
  The performance of a reinforcement learning (RL) system depends on the
computational architecture used to approximate a value function. Deep learning
methods provide both optimization techniques and architectures for
approximating nonlinear functions from noisy, high-dimensional observations.
However, prevailing optimization techniques are not designed for
strictly-incremental online updates. Nor are standard architectures designed
for observations with an a priori unknown structure: for example, light sensors
randomly dispersed in space. This paper proposes an online RL prediction
algorithm with an adaptive architecture that efficiently finds useful nonlinear
features. The algorithm is evaluated in a spatial domain with high-dimensional,
stochastic observations. The algorithm outperforms non-adaptive baseline
architectures and approaches the performance of an architecture given
side-channel information. These results are a step towards scalable RL
algorithms for more general problems, where the observation structure is not
available.
