Explicit Regularization via Regularizer Mirror Descent
  Despite perfectly interpolating the training data, deep neural networks
(DNNs) can often generalize fairly well, in part due to the "implicit
regularization" induced by the learning algorithm. Nonetheless, various forms
of regularization, such as "explicit regularization" (via weight decay), are
often used to avoid overfitting, especially when the data is corrupted. There
are several challenges with explicit regularization, most notably unclear
convergence properties. Inspired by convergence properties of stochastic mirror
descent (SMD) algorithms, we propose a new method for training DNNs with
regularization, called regularizer mirror descent (RMD). In highly
overparameterized DNNs, SMD simultaneously interpolates the training data and
minimizes a certain potential function of the weights. RMD starts with a
standard cost which is the sum of the training loss and a convex regularizer of
the weights. Reinterpreting this cost as the potential of an "augmented"
overparameterized network and applying SMD yields RMD. As a result, RMD
inherits the properties of SMD and provably converges to a point "close" to the
minimizer of this cost. RMD is computationally comparable to stochastic
gradient descent (SGD) and weight decay, and is parallelizable in the same
manner. Our experimental results on training sets with various levels of
corruption suggest that the generalization performance of RMD is remarkably
robust and significantly better than both SGD and weight decay, which
implicitly and explicitly regularize the $\ell_2$ norm of the weights. RMD can
also be used to regularize the weights to a desired weight vector, which is
particularly relevant for continual learning.
