Unique Properties of Flat Minima in Deep Networks
  It is well known that (stochastic) gradient descent has an implicit bias
towards flat minima. In deep neural network training, this mechanism serves to
screen out minima. However, the precise effect that this has on the trained
network is not yet fully understood. In this paper, we characterize the flat
minima in linear neural networks trained with a quadratic loss. First, we show
that linear ResNets with zero initialization necessarily converge to the
flattest of all minima. We then prove that these minima correspond to nearly
balanced networks whereby the gain from the input to any intermediate
representation does not change drastically from one layer to the next. Finally,
we show that consecutive layers in flat minima solutions are coupled. That is,
one of the left singular vectors of each weight matrix, equals one of the right
singular vectors of the next matrix. This forms a distinct path from input to
output, that, as we show, is dedicated to the signal that experiences the
largest gain end-to-end. Experiments indicate that these properties are
characteristic of both linear and nonlinear models trained in practice.
