Modeling transition dynamics in MDPs with RKHS embeddings of conditional
  distributions
  We propose a new, nonparametric approach to estimating the value function in
reinforcement learning. This approach makes use of a recently developed
representation of conditional distributions as functions in a reproducing
kernel Hilbert space. Such representations bypass the need for estimating
transition probabilities, and apply to any domain on which kernels can be
defined. Our approach avoids the need to approximate intractable integrals
since expectations are represented as RKHS inner products whose computation has
linear complexity in the sample size. Thus, we can efficiently perform value
function estimation in a wide variety of settings, including finite state
spaces, continuous states spaces, and partially observable tasks where only
sensor measurements are available. A second advantage of the approach is that
we learn the conditional distribution representation from a training sample,
and do not require an exhaustive exploration of the state space. We prove
convergence of our approach either to the optimal policy, or to the closest
projection of the optimal policy in our model class, under reasonable
assumptions. In experiments, we demonstrate the performance of our algorithm on
a learning task in a continuous state space (the under-actuated pendulum), and
on a navigation problem where only images from a sensor are observed. We
compare with least-squares policy iteration where a Gaussian process is used
for value function estimation. Our algorithm achieves better performance in
both tasks.
