On the training dynamics of deep networks with $L_2$ regularization
  We study the role of $L_2$ regularization in deep learning, and uncover
simple relations between the performance of the model, the $L_2$ coefficient,
the learning rate, and the number of training steps. These empirical relations
hold when the network is overparameterized. They can be used to predict the
optimal regularization parameter of a given model. In addition, based on these
observations we propose a dynamical schedule for the regularization parameter
that improves performance and speeds up training. We test these proposals in
modern image classification settings. Finally, we show that these empirical
relations can be understood theoretically in the context of infinitely wide
networks. We derive the gradient flow dynamics of such networks, and compare
the role of $L_2$ regularization in this context with that of linear models.
