Training and Inference with Integers in Deep Neural Networks
  Researches on deep neural networks with discrete parameters and their
deployment in embedded systems have been active and promising topics. Although
previous works have successfully reduced precision in inference, transferring
both training and inference processes to low-bitwidth integers has not been
demonstrated simultaneously. In this work, we develop a new method termed as
"WAGE" to discretize both training and inference, where weights (W),
activations (A), gradients (G) and errors (E) among layers are shifted and
linearly constrained to low-bitwidth integers. To perform pure discrete
dataflow for fixed-point devices, we further replace batch normalization by a
constant scaling layer and simplify other components that are arduous for
integer implementation. Improved accuracies can be obtained on multiple
datasets, which indicates that WAGE somehow acts as a type of regularization.
Empirically, we demonstrate the potential to deploy training in hardware
systems such as integer-based deep learning accelerators and neuromorphic chips
with comparable accuracy and higher energy efficiency, which is crucial to
future AI applications in variable scenarios with transfer and continual
learning demands.
