Plan-Space State Embeddings for Improved Reinforcement Learning
  Robot control problems are often structured with a policy function that maps
state values into control values, but in many dynamic problems the observed
state can have a difficult to characterize relationship with useful policy
actions. In this paper we present a new method for learning state embeddings
from plans or other forms of demonstrations such that the embedding space has a
specified geometric relationship with the demonstrations. We present a novel
variational framework for learning these embeddings that attempts to optimize
trajectory linearity in the learned embedding space. We show how these
embedding spaces can then be used as an augmentation to the robot state in
reinforcement learning problems. We use kinodynamic planning to generate
training trajectories for some example environments, and then train embedding
spaces for these environments. We show empirically that observing a system in
the learned embedding space improves the performance of policy gradient
reinforcement learning algorithms, particularly by reducing the variance
between training runs. Our technique is limited to environments where
demonstration data is available, but places no limits on how that data is
collected. Our embedding technique provides a way to transfer domain knowledge
from existing technologies such as planning and control algorithms, into more
flexible policy learning algorithms, by creating an abstract representation of
the robot state with meaningful geometry.
