Low-Precision Training in Logarithmic Number System using Multiplicative
  Weight Update
  Representing deep neural networks (DNNs) in low-precision is a promising
approach to enable efficient acceleration and memory reduction. Previous
methods that train DNNs in low-precision typically keep a copy of weights in
high-precision during the weight updates. Directly training with low-precision
weights leads to accuracy degradation due to complex interactions between the
low-precision number systems and the learning algorithms. To address this
issue, we develop a co-designed low-precision training framework, termed
LNS-Madam, in which we jointly design a logarithmic number system (LNS) and a
multiplicative weight update algorithm (Madam). We prove that LNS-Madam results
in low quantization error during weight updates, leading to a stable
convergence even if the precision is limited. We further propose a hardware
design of LNS-Madam that resolves practical challenges in implementing an
efficient datapath for LNS computations. Our implementation effectively reduces
energy overhead incurred by LNS-to-integer conversion and partial sum
accumulation. Experimental results show that LNS-Madam achieves comparable
accuracy to full-precision counterparts with only 8 bits on popular computer
vision and natural language tasks. Compared to a full-precision floating-point
implementation, LNS-Madam reduces the energy consumption by over 90.
