Understanding and Scheduling Weight Decay
  Weight decay is a popular and even necessary regularization technique for
training deep neural networks that generalize well. Previous work usually
interpreted weight decay as a Gaussian prior from the Bayesian perspective.
However, weight decay sometimes shows mysterious behaviors beyond the
conventional understanding. For example, the optimal weight decay value tends
to be zero given long enough training time. Moreover, existing work typically
failed to recognize the importance of scheduling weight decay during training.
Our work aims at theoretically understanding novel behaviors of weight decay
and designing schedulers for weight decay in deep learning. This paper mainly
has three contributions. First, we propose a novel theoretical interpretation
of weight decay from the perspective of learning dynamics. Second, we propose a
novel weight-decay linear scaling rule for large-batch training that
proportionally increases weight decay rather than the learning rate as the
batch size increases. Third, we provide an effective learning-rate-aware
scheduler for weight decay, called the Stable Weight Decay (SWD) method, which,
to the best of our knowledge, is the first practical design for weight decay
scheduling. In our various experiments, the SWD method often makes improvements
over $L_{2}$ Regularization and Decoupled Weight Decay.
