Logit Attenuating Weight Normalization
  Over-parameterized deep networks trained using gradient-based optimizers are
a popular choice for solving classification and ranking problems. Without
appropriately tuned $\ell_2$ regularization or weight decay, such networks have
the tendency to make output scores (logits) and network weights large, causing
training loss to become too small and the network to lose its adaptivity
(ability to move around) in the parameter space. Although regularization is
typically understood from an overfitting perspective, we highlight its role in
making the network more adaptive and enabling it to escape more easily from
weights that generalize poorly. To provide such a capability, we propose a
method called Logit Attenuating Weight Normalization (LAWN), that can be
stacked onto any gradient-based optimizer. LAWN controls the logits by
constraining the weight norms of layers in the final homogeneous sub-network.
Empirically, we show that the resulting LAWN variant of the optimizer makes a
deep network more adaptive to finding minimas with superior generalization
performance on large-scale image classification and recommender systems. While
LAWN is particularly impressive in improving Adam, it greatly improves all
optimizers when used with large batch sizes
