Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
  The high computational and parameter complexity of neural networks makes
their training very slow and difficult to deploy on energy and
storage-constrained computing systems. Many network complexity reduction
techniques have been proposed including fixed-point implementation. However, a
systematic approach for designing full fixed-point training and inference of
deep neural networks remains elusive. We describe a precision assignment
methodology for neural network training in which all network parameters, i.e.,
activations and weights in the feedforward path, gradients and weight
accumulators in the feedback path, are assigned close to minimal precision. The
precision assignment is derived analytically and enables tracking the
convergence behavior of the full precision training, known to converge a
priori. Thus, our work leads to a systematic methodology of determining
suitable precision for fixed-point training. The near optimality (minimality)
of the resulting precision assignment is validated empirically for four
networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The complexity
reduction arising from our approach is compared with other fixed-point neural
network designs.
