A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training
  of DNNs
  Training neural networks on image datasets generally require extensive
experimentation to find the optimal learning rate regime. Especially, for the
cases of adversarial training or for training a newly synthesized model, one
would not know the best learning rate regime beforehand. We propose an
automated algorithm for determining the learning rate trajectory, that works
across datasets and models for both natural and adversarial training, without
requiring any dataset/model specific tuning. It is a stand-alone,
parameterless, adaptive approach with no computational overhead. We
theoretically discuss the algorithm's convergence behavior. We empirically
validate our algorithm extensively. Our results show that our proposed approach
\emph{consistently} achieves top-level accuracy compared to SOTA baselines in
the literature in natural as well as adversarial training.
