Benchmarking Unsupervised Outlier Detection with Realistic Synthetic
  Data
  Benchmarking unsupervised outlier detection is difficult. Outliers are rare,
and existing benchmark data contains outliers with various and unknown
characteristics. Fully synthetic data usually consists of outliers and regular
instance with clear characteristics and thus allows for a more meaningful
evaluation of detection methods in principle. Nonetheless, there have only been
few attempts to include synthetic data in benchmarks for outlier detection.
This might be due to the imprecise notion of outliers or to the difficulty to
arrive at a good coverage of different domains with synthetic data. In this
work we propose a generic process for the generation of data sets for such
benchmarking. The core idea is to reconstruct regular instances from existing
real-world benchmark data while generating outliers so that they exhibit
insightful characteristics. This allows both for a good coverage of domains and
for helpful interpretations of results. We also describe three instantiations
of the generic process that generate outliers with specific characteristics,
like local outliers. A benchmark with state-of-the-art detection methods
confirms that our generic process is indeed practical.
