Training and Evaluation of Deep Policies using Reinforcement Learning
  and Generative Models
  We present a data-efficient framework for solving sequential decision-making
problems which exploits the combination of reinforcement learning (RL) and
latent variable generative models. The framework, called GenRL, trains deep
policies by introducing an action latent variable such that the feed-forward
policy search can be divided into two parts: (i) training a sub-policy that
outputs a distribution over the action latent variable given a state of the
system, and (ii) unsupervised training of a generative model that outputs a
sequence of motor actions conditioned on the latent action variable. GenRL
enables safe exploration and alleviates the data-inefficiency problem as it
exploits prior knowledge about valid sequences of motor actions. Moreover, we
provide a set of measures for evaluation of generative models such that we are
able to predict the performance of the RL policy training prior to the actual
training on a physical robot. We experimentally determine the characteristics
of generative models that have most influence on the performance of the final
policy training on two robotics tasks: shooting a hockey puck and throwing a
basketball. Furthermore, we empirically demonstrate that GenRL is the only
method which can safely and efficiently solve the robotics tasks compared to
two state-of-the-art RL methods.
