A Comprehensive Benchmark Framework for Active Learning Methods in
  Entity Matching
  Entity Matching (EM) is a core data cleaning task, aiming to identify
different mentions of the same real-world entity. Active learning is one way to
address the challenge of scarce labeled data in practice, by dynamically
collecting the necessary examples to be labeled by an Oracle and refining the
learned model (classifier) upon them. In this paper, we build a unified active
learning benchmark framework for EM that allows users to easily combine
different learning algorithms with applicable example selection algorithms. The
goal of the framework is to enable concrete guidelines for practitioners as to
what active learning combinations will work well for EM. Towards this, we
perform comprehensive experiments on publicly available EM datasets from
product and publication domains to evaluate active learning methods, using a
variety of metrics including EM quality, #labels and example selection
latencies. Our most surprising result finds that active learning with fewer
labels can learn a classifier of comparable quality as supervised learning. In
fact, for several of the datasets, we show that there is an active learning
combination that beats the state-of-the-art supervised learning result. Our
framework also includes novel optimizations that improve the quality of the
learned model by roughly 9% in terms of F1-score and reduce example selection
latencies by up to 10x without affecting the quality of the model.
