Training Recurrent Neural Networks against Noisy Computations during
  Inference
  We explore the robustness of recurrent neural networks when the computations
within the network are noisy. One of the motivations for looking into this
problem is to reduce the high power cost of conventional computing of neural
network operations through the use of analog neuromorphic circuits. Traditional
GPU/CPU-centered deep learning architectures exhibit bottlenecks in
power-restricted applications, such as speech recognition in embedded systems.
The use of specialized neuromorphic circuits, where analog signals passed
through memory-cell arrays are sensed to accomplish matrix-vector
multiplications, promises large power savings and speed gains but brings with
it the problems of limited precision of computations and unavoidable analog
noise.
  In this paper we propose a method, called {\em Deep Noise Injection
training}, to train RNNs to obtain a set of weights/biases that is much more
robust against noisy computation during inference. We explore several RNN
architectures, such as vanilla RNN and long-short-term memories (LSTM), and
show that after convergence of Deep Noise Injection training the set of trained
weights/biases has more consistent performance over a wide range of noise
powers entering the network during inference. Surprisingly, we find that Deep
Noise Injection training improves overall performance of some networks even for
numerically accurate inference.
