Early-stopped neural networks are consistent
  This work studies the behavior of shallow ReLU networks trained with the
logistic loss via gradient descent on binary classification data where the
underlying data distribution is general, and the (optimal) Bayes risk is not
necessarily zero. In this setting, it is shown that gradient descent with early
stopping achieves population risk arbitrarily close to optimal in terms of not
just logistic and misclassification losses, but also in terms of calibration,
meaning the sigmoid mapping of its outputs approximates the true underlying
conditional distribution arbitrarily finely. Moreover, the necessary iteration,
sample, and architectural complexities of this analysis all scale naturally
with a certain complexity measure of the true conditional model. Lastly, while
it is not shown that early stopping is necessary, it is shown that any
univariate classifier satisfying a local interpolation property is
inconsistent.
