Generating Probabilistic Safety Guarantees for Neural Network
  Controllers
  Neural networks serve as effective controllers in a variety of complex
settings due to their ability to represent expressive policies. The complex
nature of neural networks, however, makes their output difficult to verify and
predict, which limits their use in safety-critical applications. While
simulations provide insight into the performance of neural network controllers,
they are not enough to guarantee that the controller will perform safely in all
scenarios. To address this problem, recent work has focused on formal methods
to verify properties of neural network outputs. For neural network controllers,
we can use a dynamics model to determine the output properties that must hold
for the controller to operate safely. In this work, we develop a method to use
the results from neural network verification tools to provide probabilistic
safety guarantees on a neural network controller. We develop an adaptive
verification approach to efficiently generate an overapproximation of the
neural network policy. Next, we modify the traditional formulation of Markov
decision process (MDP) model checking to provide guarantees on the
overapproximated policy given a stochastic dynamics model. Finally, we
incorporate techniques in state abstraction to reduce overapproximation error
during the model checking process. We show that our method is able to generate
meaningful probabilistic safety guarantees for aircraft collision avoidance
neural networks that are loosely inspired by Airborne Collision Avoidance
System X (ACAS X), a family of collision avoidance systems that formulates the
problem as a partially observable Markov decision process (POMDP).
