Model-Ensemble Trust-Region Policy Optimization
  Model-free reinforcement learning (RL) methods are succeeding in a growing
number of tasks, aided by recent advances in deep learning. However, they tend
to suffer from high sample complexity, which hinders their use in real-world
domains. Alternatively, model-based reinforcement learning promises to reduce
sample complexity, but tends to require careful tuning and to date have
succeeded mainly in restrictive domains where simple models are sufficient for
learning. In this paper, we analyze the behavior of vanilla model-based
reinforcement learning methods when deep neural networks are used to learn both
the model and the policy, and show that the learned policy tends to exploit
regions where insufficient data is available for the model to be learned,
causing instability in training. To overcome this issue, we propose to use an
ensemble of models to maintain the model uncertainty and regularize the
learning process. We further show that the use of likelihood ratio derivatives
yields much more stable learning than backpropagation through time. Altogether,
our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO)
significantly reduces the sample complexity compared to model-free deep RL
methods on challenging continuous control benchmark tasks.
