Certifiable Robustness to Adversarial State Uncertainty in Deep
  Reinforcement Learning
  Deep Neural Network-based systems are now the state-of-the-art in many
robotics tasks, but their application in safety-critical domains remains
dangerous without formal guarantees on network robustness. Small perturbations
to sensor inputs (from noise or adversarial examples) are often enough to
change network-based decisions, which was recently shown to cause an autonomous
vehicle to swerve into another lane. In light of these dangers, numerous
algorithms have been developed as defensive mechanisms from these adversarial
inputs, some of which provide formal robustness guarantees or certificates.
This work leverages research on certified adversarial robustness to develop an
online certifiably robust for deep reinforcement learning algorithms. The
proposed defense computes guaranteed lower bounds on state-action values during
execution to identify and choose a robust action under a worst-case deviation
in input space due to possible adversaries or noise. Moreover, the resulting
policy comes with a certificate of solution quality, even though the true state
and optimal action are unknown to the certifier due to the perturbations. The
approach is demonstrated on a Deep Q-Network policy and is shown to increase
robustness to noise and adversaries in pedestrian collision avoidance scenarios
and a classic control task. This work extends one of our prior works with new
performance guarantees, extensions to other RL algorithms, expanded results
aggregated across more scenarios, an extension into scenarios with adversarial
behavior, comparisons with a more computationally expensive method, and
visualizations that provide intuition about the robustness algorithm.
