Testing Autonomous Systems with Believed Equivalence Refinement
  Continuous engineering of autonomous driving functions commonly requires
deploying vehicles in road testing to obtain inputs that cause problematic
decisions. Although the discovery leads to producing an improved system, it
also challenges the foundation of testing using equivalence classes and the
associated relative test coverage criterion. In this paper, we propose believed
equivalence, where the establishment of an equivalence class is initially based
on expert belief and is subject to a set of available test cases having a
consistent valuation. Upon a newly encountered test case that breaks the
consistency, one may need to refine the established categorization in order to
split the originally believed equivalence into two. Finally, we focus on
modules implemented using deep neural networks where every category partitions
an input over the real domain. We present both analytical and lazy methods to
suggest the refinement. The concept is demonstrated in analyzing multiple
autonomous driving modules, indicating the potential of our proposed approach.
