Statistical Evaluation of Anomaly Detectors for Sequences
  Although precision and recall are standard performance measures for anomaly
detection, their statistical properties in sequential detection settings are
poorly understood. In this work, we formalize a notion of precision and recall
with temporal tolerance for point-based anomaly detection in sequential data.
These measures are based on time-tolerant confusion matrices that may be used
to compute time-tolerant variants of many other standard measures. However,
care has to be taken to preserve interpretability. We perform a statistical
simulation study to demonstrate that precision and recall may overestimate the
performance of a detector, when computed with temporal tolerance. To alleviate
this problem, we show how to obtain null distributions for the two measures to
assess the statistical significance of reported results.
