Safer Reinforcement Learning through Transferable Instinct Networks
  Random exploration is one of the main mechanisms through which reinforcement
learning (RL) finds well-performing policies. However, it can lead to
undesirable or catastrophic outcomes when learning online in safety-critical
environments. In fact, safe learning is one of the major obstacles towards
real-world agents that can learn during deployment. One way of ensuring that
agents respect hard limitations is to explicitly configure boundaries in which
they can operate. While this might work in some cases, we do not always have
clear a-priori information which states and actions can lead dangerously close
to hazardous states. Here, we present an approach where an additional policy
can override the main policy and offer a safer alternative action. In our
instinct-regulated RL (IR^2L) approach, an "instinctual" network is trained to
recognize undesirable situations, while guarding the learning policy against
entering them. The instinct network is pre-trained on a single task where it is
safe to make mistakes, and transferred to environments in which learning a new
task safely is critical. We demonstrate IR^2L in the OpenAI Safety gym domain,
in which it receives a significantly lower number of safety violations during
training than a baseline RL approach while reaching similar task performance.
