Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit
  Bias towards Low Rank
  In deep learning, it is common to use more network parameters than training
points. In such scenarioof over-parameterization, there are usually multiple
networks that achieve zero training error so that thetraining algorithm induces
an implicit bias on the computed solution. In practice, (stochastic)
gradientdescent tends to prefer solutions which generalize well, which provides
a possible explanation of thesuccess of deep learning. In this paper we analyze
the dynamics of gradient descent in the simplifiedsetting of linear networks
and of an estimation problem. Although we are not in an
overparameterizedscenario, our analysis nevertheless provides insights into the
phenomenon of implicit bias. In fact, wederive a rigorous analysis of the
dynamics of vanilla gradient descent, and characterize the dynamicalconvergence
of the spectrum. We are able to accurately locate time intervals where the
effective rankof the iterates is close to the effective rank of a low-rank
projection of the ground-truth matrix. Inpractice, those intervals can be used
as criteria for early stopping if a certain regularity is desired. Wealso
provide empirical evidence for implicit bias in more general scenarios, such as
matrix sensing andrandom initialization. This suggests that deep learning
prefers trajectories whose complexity (measuredin terms of effective rank) is
monotonically increasing, which we believe is a fundamental concept for
thetheoretical understanding of deep learning.
