Training Data Subset Selection for Regression with Controlled
  Generalization Error
  Data subset selection from a large number of training instances has been a
successful approach toward efficient and cost-effective machine learning.
However, models trained on a smaller subset may show poor generalization
ability. In this paper, our goal is to design an algorithm for selecting a
subset of the training data, so that the model can be trained quickly, without
significantly sacrificing on accuracy. More specifically, we focus on data
subset selection for L2 regularized regression problems and provide a novel
problem formulation which seeks to minimize the training loss with respect to
both the trainable parameters and the subset of training data, subject to error
bounds on the validation set. We tackle this problem using several technical
innovations. First, we represent this problem with simplified constraints using
the dual of the original training problem and show that the objective of this
new representation is a monotone and alpha-submodular function, for a wide
variety of modeling choices. Such properties lead us to develop SELCON, an
efficient majorization-minimization algorithm for data subset selection, that
admits an approximation guarantee even when the training provides an imperfect
estimate of the trained model. Finally, our experiments on several datasets
show that SELCON trades off accuracy and efficiency more effectively than the
current state-of-the-art.
