Recommending Training Set Sizes for Classification
  Based on a comprehensive study of 20 established data sets, we recommend
training set sizes for any classification data set. We obtain our
recommendations by systematically withholding training data and developing
models through five different classification methods for each resulting
training set. Based on these results, we construct accuracy confidence
intervals for each training set size and fit the lower bounds to inverse power
low learning curves. We also estimate a sufficient training set size (STSS) for
each data set based on established convergence criteria. We compare STSS to the
data sets' characteristics; based on identified trends, we recommend training
set sizes between 3000 and 30000 data points, according to a data set's number
of classes and number of features. Because obtaining and preparing training
data has non-negligible costs that are proportional to data set size, these
results afford the potential opportunity for substantial savings for predictive
modeling efforts.
