AdaS: Adaptive Scheduling of Stochastic Gradients
  The choice of step-size used in Stochastic Gradient Descent (SGD)
optimization is empirically selected in most training procedures. Moreover, the
use of scheduled learning techniques such as Step-Decaying, Cyclical-Learning,
and Warmup to tune the step-size requires extensive practical
experience--offering limited insight into how the parameters update--and is not
consistent across applications. This work attempts to answer a question of
interest to both researchers and practitioners, namely \textit{"how much
knowledge is gained in iterative training of deep neural networks?"} Answering
this question introduces two useful metrics derived from the singular values of
the low-rank factorization of convolution layers in deep neural networks. We
introduce the notions of \textit{"knowledge gain"} and \textit{"mapping
condition"} and propose a new algorithm called Adaptive Scheduling (AdaS) that
utilizes these derived metrics to adapt the SGD learning rate proportionally to
the rate of change in knowledge gain over successive iterations.
Experimentation reveals that, using the derived metrics, AdaS exhibits: (a)
faster convergence and superior generalization over existing adaptive learning
methods; and (b) lack of dependence on a validation set to determine when to
stop training. Code is available at
\url{https://github.com/mahdihosseini/AdaS}.
