Noisy Computations during Inference: Harmful or Helpful?
  We study two aspects of noisy computations during inference. The first aspect
is how to mitigate their side effects for naturally trained deep learning
systems. One of the motivations for looking into this problem is to reduce the
high power cost of conventional computing of neural networks through the use of
analog neuromorphic circuits. Traditional GPU/CPU-centered deep learning
architectures exhibit bottlenecks in power-restricted applications (e.g.,
embedded systems). The use of specialized neuromorphic circuits, where analog
signals passed through memory-cell arrays are sensed to accomplish
matrix-vector multiplications, promises large power savings and speed gains but
brings with it the problems of limited precision of computations and
unavoidable analog noise. We manage to improve inference accuracy from 21.1% to
99.5% for MNIST images, from 29.9% to 89.1% for CIFAR10, and from 15.5% to
89.6% for MNIST stroke sequences with the presence of strong noise (with
signal-to-noise power ratio being 0 dB) by noise-injected training and a voting
method. This observation promises neural networks that are insensitive to
inference noise, which reduces the quality requirements on neuromorphic
circuits and is crucial for their practical usage. The second aspect is how to
utilize the noisy inference as a defensive architecture against black-box
adversarial attacks. During inference, by injecting proper noise to signals in
the neural networks, the robustness of adversarially-trained neural networks
against black-box attacks has been further enhanced by 0.5% and 1.13% for two
adversarially trained models for MNIST and CIFAR10, respectively.
