Dynamic Precision Analog Computing for Neural Networks
  Analog electronic and optical computing exhibit tremendous advantages over
digital computing for accelerating deep learning when operations are executed
at low precision. In this work, we derive a relationship between analog
precision, which is limited by noise, and digital bit precision. We propose
extending analog computing architectures to support varying levels of precision
by repeating operations and averaging the result, decreasing the impact of
noise. Such architectures enable programmable tradeoffs between precision and
other desirable performance metrics such as energy efficiency or throughput. To
utilize dynamic precision, we propose a method for learning the precision of
each layer of a pre-trained model without retraining network weights. We
evaluate this method on analog architectures subject to a variety of noise
sources such as shot noise, thermal noise, and weight noise and find that
employing dynamic precision reduces energy consumption by up to 89% for
computer vision models such as Resnet50 and by 24% for natural language
processing models such as BERT. In one example, we apply dynamic precision to a
shot-noise limited homodyne optical neural network and simulate inference at an
optical energy consumption of 2.7 aJ/MAC for Resnet50 and 1.6 aJ/MAC for BERT
with <2% accuracy degradation.
