Unsupervised Basis Function Adaptation for Reinforcement Learning
  When using reinforcement learning (RL) algorithms it is common, given a large
state space, to introduce some form of approximation architecture for the value
function (VF). The exact form of this architecture can have a significant
effect on an agent's performance, however, and determining a suitable
approximation architecture can often be a highly complex task. Consequently
there is currently interest among researchers in the potential for allowing RL
algorithms to adaptively generate (i.e. to learn) approximation architectures.
One relatively unexplored method of adapting approximation architectures
involves using feedback regarding the frequency with which an agent has visited
certain states to guide which areas of the state space to approximate with
greater detail. In this article we will: (a) informally discuss the potential
advantages offered by such methods; (b) introduce a new algorithm based on such
methods which adapts a state aggregation approximation architecture on-line and
is designed for use in conjunction with SARSA; (c) provide theoretical results,
in a policy evaluation setting, regarding this particular algorithm's
complexity, convergence properties and potential to reduce VF error; and
finally (d) test experimentally the extent to which this algorithm can improve
performance given a number of different test problems. Taken together our
results suggest that our algorithm (and potentially such methods more
generally) can provide a versatile and computationally lightweight means of
significantly boosting RL performance given suitable conditions which are
commonly encountered in practice.
