A Simple Unified Framework for Anomaly Detection in Deep Reinforcement
  Learning
  Abnormal states in deep reinforcement learning~(RL) are states that are
beyond the scope of an RL policy. Such states may make the RL system unsafe and
impede its deployment in real scenarios. In this paper, we propose a simple yet
effective anomaly detection framework for deep RL algorithms that
simultaneously considers random, adversarial and out-of-distribution~(OOD)
state outliers. In particular, we attain the class-conditional distributions
for each action class under the Gaussian assumption, and rely on these
distributions to discriminate between inliers and outliers based on Mahalanobis
Distance~(MD) and Robust Mahalanobis Distance. We conduct extensive experiments
on Atari games that verify the effectiveness of our detection strategies. To
the best of our knowledge, we present the first in-detail study of statistical
and adversarial anomaly detection in deep RL algorithms. This simple unified
anomaly detection paves the way towards deploying safe RL systems in real-world
applications.
