Quantized Memory-Augmented Neural Networks
  Memory-augmented neural networks (MANNs) refer to a class of neural network
models equipped with external memory (such as neural Turing machines and memory
networks). These neural networks outperform conventional recurrent neural
networks (RNNs) in terms of learning long-term dependency, allowing them to
solve intriguing AI tasks that would otherwise be hard to address. This paper
concerns the problem of quantizing MANNs. Quantization is known to be effective
when we deploy deep models on embedded systems with limited resources.
Furthermore, quantization can substantially reduce the energy consumption of
the inference procedure. These benefits justify recent developments of
quantized multi layer perceptrons, convolutional networks, and RNNs. However,
no prior work has reported the successful quantization of MANNs. The in-depth
analysis presented here reveals various challenges that do not appear in the
quantization of the other networks. Without addressing them properly, quantized
MANNs would normally suffer from excessive quantization error which leads to
degraded performance. In this paper, we identify memory addressing
(specifically, content-based addressing) as the main reason for the performance
degradation and propose a robust quantization method for MANNs to address the
challenge. In our experiments, we achieved a computation-energy gain of 22x
with 8-bit fixed-point and binary quantization compared to the floating-point
implementation. Measured on the bAbI dataset, the resulting model, named the
quantized MANN (Q-MANN), improved the error rate by 46% and 30% with 8-bit
fixed-point and binary quantization, respectively, compared to the MANN
quantized using conventional techniques.
