Safe Imitation Learning via Fast Bayesian Reward Inference from
  Preferences
  Bayesian reward learning from demonstrations enables rigorous safety and
uncertainty analysis when performing imitation learning. However, Bayesian
reward learning methods are typically computationally intractable for complex
control problems. We propose Bayesian Reward Extrapolation (Bayesian REX), a
highly efficient Bayesian reward learning algorithm that scales to
high-dimensional imitation learning problems by pre-training a low-dimensional
feature encoding via self-supervised tasks and then leveraging preferences over
demonstrations to perform fast Bayesian inference. Bayesian REX can learn to
play Atari games from demonstrations, without access to the game score and can
generate 100,000 samples from the posterior over reward functions in only 5
minutes on a personal laptop. Bayesian REX also results in imitation learning
performance that is competitive with or better than state-of-the-art methods
that only learn point estimates of the reward function. Finally, Bayesian REX
enables efficient high-confidence policy evaluation without having access to
samples of the reward function. These high-confidence performance bounds can be
used to rank the performance and risk of a variety of evaluation policies and
provide a way to detect reward hacking behaviors.
