Challenges in Benchmarking Stream Learning Algorithms with Real-world
  Data
  Streaming data are increasingly present in real-world applications such as
sensor measurements, satellite data feed, stock market, and financial data. The
main characteristics of these applications are the online arrival of data
observations at high speed and the susceptibility to changes in the data
distributions due to the dynamic nature of real environments. The data stream
mining community still faces some primary challenges and difficulties related
to the comparison and evaluation of new proposals, mainly due to the lack of
publicly available non-stationary real-world datasets. The comparison of stream
algorithms proposed in the literature is not an easy task, as authors do not
always follow the same recommendations, experimental evaluation procedures,
datasets, and assumptions. In this paper, we mitigate problems related to the
choice of datasets in the experimental evaluation of stream classifiers and
drift detectors. To that end, we propose a new public data repository for
benchmarking stream algorithms with real-world data. This repository contains
the most popular datasets from literature and new datasets related to a highly
relevant public health problem that involves the recognition of disease vector
insects using optical sensors. The main advantage of these new datasets is the
prior knowledge of their characteristics and patterns of changes to evaluate
new adaptive algorithm proposals adequately. We also present an in-depth
discussion about the characteristics, reasons, and issues that lead to
different types of changes in data distribution, as well as a critical review
of common problems concerning the current benchmark datasets available in the
literature.
