Deployment-Efficient Reinforcement Learning via Model-Based Offline
  Optimization
  Most reinforcement learning (RL) algorithms assume online access to the
environment, in which one may readily interleave updates to the policy with
experience collection using that policy. However, in many real-world
applications such as health, education, dialogue agents, and robotics, the cost
or potential risk of deploying a new data-collection policy is high, to the
point that it can become prohibitive to update the data-collection policy more
than a few times during learning. With this view, we propose a novel concept of
deployment efficiency, measuring the number of distinct data-collection
policies that are used during policy learning. We observe that na\"{i}vely
applying existing model-free offline RL algorithms recursively does not lead to
a practical deployment-efficient and sample-efficient algorithm. We propose a
novel model-based algorithm, Behavior-Regularized Model-ENsemble (BREMEN) that
can effectively optimize a policy offline using 10-20 times fewer data than
prior works. Furthermore, the recursive application of BREMEN is able to
achieve impressive deployment efficiency while maintaining the same or better
sample efficiency, learning successful policies from scratch on simulated
robotic environments with only 5-10 deployments, compared to typical values of
hundreds to millions in standard RL baselines. Codes and pre-trained models are
available at https://github.com/matsuolab/BREMEN .
