Comparing Offline and Online Testing of Deep Neural Networks: An
  Autonomous Car Case Study
  There is a growing body of research on developing testing techniques for Deep
Neural Networks (DNN). We distinguish two general modes of testing for DNNs:
Offline testing where DNNs are tested as individual units based on test
datasets obtained independently from the DNNs under test, and online testing
where DNNs are embedded into a specific application and tested in a close-loop
mode in interaction with the application environment. In addition, we identify
two sources for generating test datasets for DNNs: Datasets obtained from
real-life and datasets generated by simulators. While offline testing can be
used with datasets obtained from either sources, online testing is largely
confined to using simulators since online testing within real-life applications
can be time-consuming, expensive and dangerous. In this paper, we study the
following two important questions aiming to compare test datasets and testing
modes for DNNs: First, can we use simulator-generated data as a reliable
substitute to real-world data for the purpose of DNN testing? Second, how do
online and offline testing results differ and complement each other? Though
these questions are generally relevant to all autonomous systems, we study them
in the context of automated driving systems where, as study subjects, we use
DNNs automating end-to-end control of cars' steering actuators. Our results
show that simulator-generated datasets are able to yield DNN prediction errors
that are similar to those obtained by testing DNNs with real-life datasets.
Further, offline testing is more optimistic than online testing as many safety
violations identified by online testing could not be identified by offline
testing, while large prediction errors generated by offline testing always led
to severe safety violations detectable by online testing.
