Safe Deep RL in 3D Environments using Human Feedback
  Agents should avoid unsafe behaviour during both training and deployment.
This typically requires a simulator and a procedural specification of unsafe
behaviour. Unfortunately, a simulator is not always available, and procedurally
specifying constraints can be difficult or impossible for many real-world
tasks. A recently introduced technique, ReQueST, aims to solve this problem by
learning a neural simulator of the environment from safe human trajectories,
then using the learned simulator to efficiently learn a reward model from human
feedback. However, it is yet unknown whether this approach is feasible in
complex 3D environments with feedback obtained from real humans - whether
sufficient pixel-based neural simulator quality can be achieved, and whether
the human data requirements are viable in terms of both quantity and quality.
In this paper we answer this question in the affirmative, using ReQueST to
train an agent to perform a 3D first-person object collection task using data
entirely from human contractors. We show that the resulting agent exhibits an
order of magnitude reduction in unsafe behaviour compared to standard
reinforcement learning.
