Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping
  Training neural networks with large batch is of fundamental significance to
deep learning. Large batch training remarkably reduces the amount of training
time but has difficulties in maintaining accuracy. Recent works have put
forward optimization methods such as LARS and LAMB to tackle this issue through
adaptive layer-wise optimization using trust ratios. Though prevailing, such
methods are observed to still suffer from unstable and extreme trust ratios
which degrades performance. In this paper, we propose a new variant of LAMB,
called LAMBC, which employs trust ratio clipping to stabilize its magnitude and
prevent extreme values. We conducted experiments on image classification tasks
such as ImageNet and CIFAR-10 and our empirical results demonstrate promising
improvements across different batch sizes.
