MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance
  Safe exploration is critical for using reinforcement learning (RL) in
risk-sensitive environments. Recent work learns risk measures which measure the
probability of violating constraints, which can then be used to enable safety.
However, learning such risk measures requires significant interaction with the
environment, resulting in excessive constraint violations during learning.
Furthermore, these measures are not easily transferable to new environments. We
cast safe exploration as an offline meta-RL problem, where the objective is to
leverage examples of safe and unsafe behavior across a range of environments to
quickly adapt learned risk measures to a new environment with previously unseen
dynamics. We then propose MEta-learning for Safe Adaptation (MESA), an approach
for meta-learning a risk measure for safe RL. Simulation experiments across 5
continuous control domains suggest that MESA can leverage offline data from a
range of different environments to reduce constraint violations in unseen
environments by up to a factor of 2 while maintaining task performance. See
https://tinyurl.com/safe-meta-rl for code and supplementary material.
