Safe Reinforcement Learning with Natural Language Constraints
  While safe reinforcement learning (RL) holds great promise for many practical
applications like robotics or autonomous cars, current approaches require
specifying constraints in mathematical form. Such specifications demand domain
expertise, limiting the adoption of safe RL. In this paper, we propose learning
to interpret natural language constraints for safe RL. To this end, we first
introduce HazardWorld, a new multi-task benchmark that requires an agent to
optimize reward while not violating constraints specified in free-form text. We
then develop an agent with a modular architecture that can interpret and adhere
to such textual constraints while learning new tasks. Our model consists of (1)
a constraint interpreter that encodes textual constraints into spatial and
temporal representations of forbidden states, and (2) a policy network that
uses these representations to produce a policy achieving minimal constraint
violations during training. Across different domains in HazardWorld, we show
that our method achieves higher rewards (up to11x) and fewer constraint
violations (by 1.8x) compared to existing approaches. However, in terms of
absolute performance, HazardWorld still poses significant challenges for agents
to learn efficiently, motivating the need for future work.
