Towards Driving-Oriented Metric for Lane Detection Models
  After the 2017 TuSimple Lane Detection Challenge, its dataset and evaluation
based on accuracy and F1 score have become the de facto standard to measure the
performance of lane detection methods. While they have played a major role in
improving the performance of lane detection methods, the validity of this
evaluation method in downstream tasks has not been adequately researched. In
this study, we design 2 new driving-oriented metrics for lane detection:
End-to-End Lateral Deviation metric (E2E-LD) is directly formulated based on
the requirements of autonomous driving, a core downstream task of lane
detection; Per-frame Simulated Lateral Deviation metric (PSLD) is a lightweight
surrogate metric of E2E-LD. To evaluate the validity of the metrics, we conduct
a large-scale empirical study with 4 major types of lane detection approaches
on the TuSimple dataset and our newly constructed dataset Comma2k19-LD. Our
results show that the conventional metrics have strongly negative correlations
($\leq$-0.55) with E2E-LD, meaning that some recent improvements purely
targeting the conventional metrics may not have led to meaningful improvements
in autonomous driving, but rather may actually have made it worse by
overfitting to the conventional metrics. As autonomous driving is a
security/safety-critical system, the underestimation of robustness hinders the
sound development of practical lane detection models. We hope that our study
will help the community achieve more downstream task-aware evaluations for lane
detection.
