Ternary Neural Networks for Resource-Efficient AI Applications
  The computation and storage requirements for Deep Neural Networks (DNNs) are
usually high. This issue limits their deployability on ubiquitous computing
devices such as smart phones, wearables and autonomous drones. In this paper,
we propose ternary neural networks (TNNs) in order to make deep learning more
resource-efficient. We train these TNNs using a teacher-student approach based
on a novel, layer-wise greedy methodology. Thanks to our two-stage training
procedure, the teacher network is still able to use state-of-the-art methods
such as dropout and batch normalization to increase accuracy and reduce
training time. Using only ternary weights and activations, the student ternary
network learns to mimic the behavior of its teacher network without using any
multiplication. Unlike its -1,1 binary counterparts, a ternary neural network
inherently prunes the smaller weights by setting them to zero during training.
This makes them sparser and thus more energy-efficient. We design a
purpose-built hardware architecture for TNNs and implement it on FPGA and ASIC.
We evaluate TNNs on several benchmark datasets and demonstrate up to 3.1x
better energy efficiency with respect to the state of the art while also
improving accuracy.
