Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks
  Trained with the Logistic Loss
  Neural networks trained to minimize the logistic (a.k.a. cross-entropy) loss
with gradient-based methods are observed to perform well in many supervised
classification tasks. Towards understanding this phenomenon, we analyze the
training and generalization behavior of infinitely wide two-layer neural
networks with homogeneous activations. We show that the limits of the gradient
flow on exponentially tailed losses can be fully characterized as a max-margin
classifier in a certain non-Hilbertian space of functions. In presence of
hidden low-dimensional structures, the resulting margin is independent of the
ambiant dimension, which leads to strong generalization bounds. In contrast,
training only the output layer implicitly solves a kernel support vector
machine, which a priori does not enjoy such an adaptivity. Our analysis of
training is non-quantitative in terms of running time but we prove
computational guarantees in simplified settings by showing equivalences with
online mirror descent. Finally, numerical experiments suggest that our analysis
describes well the practical behavior of two-layer neural networks with ReLU
activation and confirm the statistical benefits of this implicit bias.
