Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network
  using Truncated Gaussian Approximation
  In the past years, Deep convolution neural network has achieved great success
in many artificial intelligence applications. However, its enormous model size
and massive computation cost have become the main obstacle for deployment of
such powerful algorithm in the low power and resource-limited mobile systems.
As the countermeasure to this problem, deep neural networks with ternarized
weights (i.e. -1, 0, +1) have been widely explored to greatly reduce the model
size and computational cost, with limited accuracy degradation. In this work,
we propose a novel ternarized neural network training method which
simultaneously optimizes both weights and quantizer during training,
differentiating from prior works. Instead of fixed and uniform weight
ternarization, we are the first to incorporate the thresholds of weight
ternarization into a closed-form representation using the truncated Gaussian
approximation, enabling simultaneous optimization of weights and quantizer
through back-propagation training. With both of the first and last layer
ternarized, the experiments on the ImageNet classification task show that our
ternarized ResNet-18/34/50 only has 3.9/2.52/2.16% accuracy degradation in
comparison to the full-precision counterparts.
