Towards a Rigorous Evaluation of Time-series Anomaly Detection
  In recent years, proposed studies on time-series anomaly detection (TAD)
report high F1 scores on benchmark TAD datasets, giving the impression of clear
improvements in TAD. However, most studies apply a peculiar evaluation protocol
called point adjustment (PA) before scoring. In this paper, we theoretically
and experimentally reveal that the PA protocol has a great possibility of
overestimating the detection performance; that is, even a random anomaly score
can easily turn into a state-of-the-art TAD method. Therefore, the comparison
of TAD methods after applying the PA protocol can lead to misguided rankings.
Furthermore, we question the potential of existing TAD methods by showing that
an untrained model obtains comparable detection performance to the existing
methods even when PA is forbidden. Based on our findings, we propose a new
baseline and an evaluation protocol. We expect that our study will help a
rigorous evaluation of TAD and lead to further improvement in future
researches.
