Quantitative Propagation of Chaos for SGD in Wide Neural Networks
  In this paper, we investigate the limiting behavior of a continuous-time
counterpart of the Stochastic Gradient Descent (SGD) algorithm applied to
two-layer overparameterized neural networks, as the number or neurons (ie, the
size of the hidden layer) $N \to +\infty$. Following a probabilistic approach,
we show 'propagation of chaos' for the particle system defined by this
continuous-time dynamics under different scenarios, indicating that the
statistical interaction between the particles asymptotically vanishes. In
particular, we establish quantitative convergence with respect to $N$ of any
particle to a solution of a mean-field McKean-Vlasov equation in the metric
space endowed with the Wasserstein distance. In comparison to previous works on
the subject, we consider settings in which the sequence of stepsizes in SGD can
potentially depend on the number of neurons and the iterations. We then
identify two regimes under which different mean-field limits are obtained, one
of them corresponding to an implicitly regularized version of the minimization
problem at hand. We perform various experiments on real datasets to validate
our theoretical results, assessing the existence of these two regimes on
classification problems and illustrating our convergence results.
