Exploring Credibility Scoring Metrics of Perception Systems for
  Autonomous Driving
  Autonomous and semi-autonomous vehicles' perception algorithms can encounter
situations with erroneous object detection, such as misclassification of
objects on the road, which can lead to safety violations and potentially fatal
consequences. While there has been substantial work in the robustness of object
detection algorithms and online metric learning, there is little research on
benchmarking scoring metrics to determine any possible indicators of potential
misclassification. An emphasis is put on exploring the potential of taking
these scoring metrics online in order to allow the AV to make perception-based
decisions given real-time constraints. In this work, we explore which, if any,
metrics act as online indicators of when perception algorithms and object
detectors are failing. Our work provides insight on better design principles
and characteristics of online metrics to accurately evaluate the credibility of
object detectors. Our approach employs non-adversarial and realistic
perturbations to images, on which we evaluate various quantitative metrics. We
found that offline metrics can be designed to account for real-world
corruptions such as poor weather conditions and that the analysis of such
metrics can provide a segue into designing online metrics. This is a clear next
step as it can allow for error-free autonomous vehicle perception and safer
time-critical and safety-critical decision-making.
