Spherical Motion Dynamics: Learning Dynamics of Neural Network with
  Normalization, Weight Decay, and SGD
  In this work, we comprehensively reveal the learning dynamics of neural
network with normalization, weight decay (WD), and SGD (with momentum), named
as Spherical Motion Dynamics (SMD). Most related works study SMD by focusing on
"effective learning rate" in "equilibrium" condition, where weight norm remains
unchanged. However, their discussions on why equilibrium condition can be
reached in SMD is either absent or less convincing. Our work investigates SMD
by directly exploring the cause of equilibrium condition. Specifically, 1) we
introduce the assumptions that can lead to equilibrium condition in SMD, and
prove that weight norm can converge at linear rate with given assumptions; 2)
we propose "angular update" as a substitute for effective learning rate to
measure the evolving of neural network in SMD, and prove angular update can
also converge to its theoretical value at linear rate; 3) we verify our
assumptions and theoretical results on various computer vision tasks including
ImageNet and MSCOCO with standard settings. Experiment results show our
theoretical findings agree well with empirical observations.
