Can Offline Testing of Deep Neural Networks Replace Their Online
  Testing?
  We distinguish two general modes of testing for Deep Neural Networks (DNNs):
Offline testing where DNNs are tested as individual units based on test
datasets obtained without involving the DNNs under test, and online testing
where DNNs are embedded into a specific application environment and tested in a
closed-loop mode in interaction with the application environment. Typically,
DNNs are subjected to both types of testing during their development life cycle
where offline testing is applied immediately after DNN training and online
testing follows after offline testing and once a DNN is deployed within a
specific application environment. In this paper, we study the relationship
between offline and online testing. Our goal is to determine how offline
testing and online testing differ or complement one another and if offline
testing results can be used to help reduce the cost of online testing? Though
these questions are generally relevant to all autonomous systems, we study them
in the context of automated driving systems where, as study subjects, we use
DNNs automating end-to-end controls of steering functions of self-driving
vehicles. Our results show that offline testing is less effective than online
testing as many safety violations identified by online testing could not be
identified by offline testing, while large prediction errors generated by
offline testing always led to severe safety violations detectable by online
testing. Further, we cannot exploit offline testing results to reduce the cost
of online testing in practice since we are not able to identify specific
situations where offline testing could be as accurate as online testing in
identifying safety requirement violations.
