A Unified Framework for Task-Driven Data Quality Management
  High-quality data is critical to train performant Machine Learning (ML)
models, highlighting the importance of Data Quality Management (DQM). Existing
DQM schemes often cannot satisfactorily improve ML performance because, by
design, they are oblivious to downstream ML tasks. Besides, they cannot handle
various data quality issues (especially those caused by adversarial attacks)
and have limited applications to only certain types of ML models. Recently,
data valuation approaches (e.g., based on the Shapley value) have been
leveraged to perform DQM; yet, empirical studies have observed that their
performance varies considerably based on the underlying data and training
process. In this paper, we propose a task-driven, multi-purpose, model-agnostic
DQM framework, DataSifter, which is optimized towards a given downstream ML
task, capable of effectively removing data points with various defects, and
applicable to diverse models. Specifically, we formulate DQM as an optimization
problem and devise a scalable algorithm to solve it. Furthermore, we propose a
theoretical framework for comparing the worst-case performance of different DQM
strategies. Remarkably, our results show that the popular strategy based on the
Shapley value may end up choosing the worst data subset in certain practical
scenarios. Our evaluation shows that DataSifter achieves and most often
significantly improves the state-of-the-art performance over a wide range of
DQM tasks, including backdoor, poison, noisy/mislabel data detection, data
summarization, and data debiasing.
