Overcoming Model Bias for Robust Offline Deep Reinforcement Learning
  State-of-the-art reinforcement learning algorithms mostly rely on being
allowed to directly interact with their environment to collect millions of
observations. This makes it hard to transfer their success to industrial
control problems, where simulations are often very costly or do not exist, and
exploring in the real environment can potentially lead to catastrophic events.
Recently developed, model-free, offline RL algorithms, can learn from a single
dataset (containing limited exploration) by mitigating extrapolation error in
value functions. However, the robustness of the training process is still
comparatively low, a problem known from methods using value functions. To
improve robustness and stability of the learning process, we use dynamics
models to assess policy performance instead of value functions, resulting in
MOOSE (MOdel-based Offline policy Search with Ensembles), an algorithm which
ensures low model bias by keeping the policy within the support of the data. We
compare MOOSE with state-of-the-art model-free, offline RL algorithms { BRAC,}
BEAR and BCQ on the Industrial Benchmark and MuJoCo continuous control tasks in
terms of robust performance, and find that MOOSE outperforms its model-free
counterparts in almost all considered cases, often even by far.
