P-ADMMiRNN: Training RNN with Stable Convergence via An Efficient and
  Paralleled ADMM Approach
  It is hard to train Recurrent Neural Network (RNN) with stable convergence
and avoid gradient vanishing and exploding problems, as the weights in the
recurrent unit are repeated from iteration to iteration. Moreover, RNN is
sensitive to the initialization of weights and bias, which brings difficulties
in training. The Alternating Direction Method of Multipliers (ADMM) has become
a promising algorithm to train neural networks beyond traditional stochastic
gradient algorithms with the gradient-free features and immunity to
unsatisfactory conditions. However, ADMM could not be applied to train RNN
directly since the state in the recurrent unit is repetitively updated over
timesteps. Therefore, this work builds a new framework named ADMMiRNN upon the
unfolded form of RNN to address the above challenges simultaneously. We also
provide novel update rules and theoretical convergence analysis. We explicitly
specify essential update rules in the iterations of ADMMiRNN with constructed
approximation techniques and solutions to each sub-problem instead of vanilla
ADMM. Numerical experiments are conducted on MNIST, IMDb, and text
classification tasks. ADMMiRNN achieves convergent results and outperforms the
compared baselines. Furthermore, ADMMiRNN trains RNN more stably without
gradient vanishing or exploding than stochastic gradient algorithms. We also
provide a distributed paralleled algorithm regarding ADMMiRNN, named
P-ADMMiRNN, including Synchronous Parallel ADMMiRNN (SP-ADMMiRNN) and
Asynchronous Parallel ADMMiRNN (AP-ADMMiRNN), which is the first to train RNN
with ADMM in an asynchronous parallel manner. The source code is publicly
available.
