Kernel Based Progressive Distillation for Adder Neural Networks
  Adder Neural Networks (ANNs) which only contain additions bring us a new way
of developing deep neural networks with low energy consumption. Unfortunately,
there is an accuracy drop when replacing all convolution filters by adder
filters. The main reason here is the optimization difficulty of ANNs using
$\ell_1$-norm, in which the estimation of gradient in back propagation is
inaccurate. In this paper, we present a novel method for further improving the
performance of ANNs without increasing the trainable parameters via a
progressive kernel based knowledge distillation (PKKD) method. A convolutional
neural network (CNN) with the same architecture is simultaneously initialized
and trained as a teacher network, features and weights of ANN and CNN will be
transformed to a new space to eliminate the accuracy drop. The similarity is
conducted in a higher-dimensional space to disentangle the difference of their
distributions using a kernel based method. Finally, the desired ANN is learned
based on the information from both the ground-truth and teacher, progressively.
The effectiveness of the proposed method for learning ANN with higher
performance is then well-verified on several benchmarks. For instance, the
ANN-50 trained using the proposed PKKD method obtains a 76.8\% top-1 accuracy
on ImageNet dataset, which is 0.6\% higher than that of the ResNet-50.
