Gaussian Processes for Sample Efficient Reinforcement Learning with
  RMAX-like Exploration
  We present an implementation of model-based online reinforcement learning
(RL) for continuous domains with deterministic transitions that is specifically
designed to achieve low sample complexity. To achieve low sample complexity,
since the environment is unknown, an agent must intelligently balance
exploration and exploitation, and must be able to rapidly generalize from
observations. While in the past a number of related sample efficient RL
algorithms have been proposed, to allow theoretical analysis, mainly
model-learners with weak generalization capabilities were considered. Here, we
separate function approximation in the model learner (which does require
samples) from the interpolation in the planner (which does not require
samples). For model-learning we apply Gaussian processes regression (GP) which
is able to automatically adjust itself to the complexity of the problem (via
Bayesian hyperparameter selection) and, in practice, often able to learn a
highly accurate model from very little data. In addition, a GP provides a
natural way to determine the uncertainty of its predictions, which allows us to
implement the "optimism in the face of uncertainty" principle used to
efficiently control exploration. Our method is evaluated on four common
benchmark domains.
