Low-bit Quantization of Recurrent Neural Network Language Models Using
  Alternating Direction Methods of Multipliers
  The high memory consumption and computational costs of Recurrent neural
network language models (RNNLMs) limit their wider application on resource
constrained devices. In recent years, neural network quantization techniques
that are capable of producing extremely low-bit compression, for example,
binarized RNNLMs, are gaining increasing research interests. Directly training
of quantized neural networks is difficult. By formulating quantized RNNLMs
training as an optimization problem, this paper presents a novel method to
train quantized RNNLMs from scratch using alternating direction methods of
multipliers (ADMM). This method can also flexibly adjust the trade-off between
the compression rate and model performance using tied low-bit quantization
tables. Experiments on two tasks: Penn Treebank (PTB), and Switchboard (SWBD)
suggest the proposed ADMM quantization achieved a model size compression factor
of up to 31 times over the full precision baseline RNNLMs. Faster convergence
of 5 times in model training over the baseline binarized RNNLM quantization was
also obtained. Index Terms: Language models, Recurrent neural networks,
Quantization, Alternating direction methods of multipliers.
