Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly
  Benchmark
  Much of the world's data is streaming, time-series data, where anomalies give
significant information in critical situations; examples abound in domains such
as finance, IT, security, medical, and energy. Yet detecting anomalies in
streaming data is a difficult task, requiring detectors to process data in
real-time, not batches, and learn while simultaneously making predictions.
There are no benchmarks to adequately test and score the efficacy of real-time
anomaly detectors. Here we propose the Numenta Anomaly Benchmark (NAB), which
attempts to provide a controlled and repeatable environment of open-source
tools to test and measure anomaly detection algorithms on streaming data. The
perfect detector would detect all anomalies as soon as possible, trigger no
false alarms, work with real-world time-series data across a variety of
domains, and automatically adapt to changing statistics. Rewarding these
characteristics is formalized in NAB, using a scoring algorithm designed for
streaming data. NAB evaluates detectors on a benchmark dataset with labeled,
real-world time-series data. We present these components, and give results and
analyses for several open source, commercially-used algorithms. The goal for
NAB is to provide a standard, open source framework with which the research
community can compare and evaluate different algorithms for detecting anomalies
in streaming data.
