Implicit bias of deep linear networks in the large learning rate phase
  Most theoretical studies explaining the regularization effect in deep
learning have only focused on gradient descent with a sufficient small learning
rate or even gradient flow (infinitesimal learning rate). Such researches,
however, have neglected a reasonably large learning rate applied in most
practical applications. In this work, we characterize the implicit bias effect
of deep linear networks for binary classification using the logistic loss in
the large learning rate regime, inspired by the seminal work by Lewkowycz et
al. [26] in a regression setting with squared loss. They found a learning rate
regime with a large stepsize named the catapult phase, where the loss grows at
the early stage of training and eventually converges to a minimum that is
flatter than those found in the small learning rate regime. We claim that
depending on the separation conditions of data, the gradient descent iterates
will converge to a flatter minimum in the catapult phase. We rigorously prove
this claim under the assumption of degenerate data by overcoming the difficulty
of the non-constant Hessian of logistic loss and further characterize the
behavior of loss and Hessian for non-separable data. Finally, we demonstrate
that flatter minima in the space spanned by non-separable data along with the
learning rate in the catapult phase can lead to better generalization
empirically.
