A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks
In distributed training of deep neural networks or Federated Learning (FL),
people usually run Stochastic Gradient Descent (SGD) or its variants on each
machine and communicate with other machines periodically. However, SGD might
converge slowly in training some deep neural networks (e.g., RNN, LSTM) because
of the exploding gradient issue. Gradient clipping is usually employed to
address this issue in the single machine setting, but exploring this technique
in the FL setting is still in its infancy: it remains mysterious whether the
gradient clipping scheme can take advantage of multiple machines to enjoy
parallel speedup. The main technical difficulty lies in dealing with nonconvex
loss function, non-Lipschitz continuous gradient, and skipping communication
rounds simultaneously. In this paper, we explore a relaxed-smoothness
assumption of the loss landscape which LSTM was shown to satisfy in previous
works and design a communication-efficient gradient clipping algorithm. This
algorithm can be run on multiple machines, where each machine employs a
gradient clipping scheme and communicate with other machines after multiple
steps of gradient-based updates. Our algorithm is proved to have
$O\left(\frac{1}{N\epsilon^4}\right)$ iteration complexity for finding an
$\epsilon$-stationary point, where $N$ is the number of machines. This
indicates that our algorithm enjoys linear speedup. We prove this result by
introducing novel analysis techniques of estimating truncated random variables,
which we believe are of independent interest. Our experiments on several
benchmark datasets and various scenarios demonstrate that our algorithm indeed
exhibits fast convergence speed in practice and thus validates our theory.