Variational Model-based Policy Optimization
  Model-based reinforcement learning (RL) algorithms allow us to combine
model-generated data with those collected from interaction with the real system
in order to alleviate the data efficiency problem in RL. However, designing
such algorithms is often challenging because the bias in simulated data may
overshadow the ease of data generation. A potential solution to this challenge
is to jointly learn and improve model and policy using a universal objective
function. In this paper, we leverage the connection between RL and
probabilistic inference, and formulate such an objective function as a
variational lower-bound of a log-likelihood. This allows us to use expectation
maximization (EM) and iteratively fix a baseline policy and learn a variational
distribution, consisting of a model and a policy (E-step), followed by
improving the baseline policy given the learned variational distribution
(M-step). We propose model-based and model-free policy iteration (actor-critic)
style algorithms for the E-step and show how the variational distribution
learned by them can be used to optimize the M-step in a fully model-based
fashion. Our experiments on a number of continuous control tasks show that
despite being more complex, our model-based (E-step) algorithm, called {\em
variational model-based policy optimization} (VMBPO), is more sample-efficient
and robust to hyper-parameter tuning than its model-free (E-step) counterpart.
Using the same control tasks, we also compare VMBPO with several
state-of-the-art model-based and model-free RL algorithms and show its sample
efficiency and performance.
