ReCU: Reviving the Dead Weights in Binary Neural Networks
  Binary neural networks (BNNs) have received increasing attention due to their
superior reductions of computation and memory. Most existing works focus on
either lessening the quantization error by minimizing the gap between the
full-precision weights and their binarization or designing a gradient
approximation to mitigate the gradient mismatch, while leaving the "dead
weights" untouched. This leads to slow convergence when training BNNs. In this
paper, for the first time, we explore the influence of "dead weights" which
refer to a group of weights that are barely updated during the training of
BNNs, and then introduce rectified clamp unit (ReCU) to revive the "dead
weights" for updating. We prove that reviving the "dead weights" by ReCU can
result in a smaller quantization error. Besides, we also take into account the
information entropy of the weights, and then mathematically analyze why the
weight standardization can benefit BNNs. We demonstrate the inherent
contradiction between minimizing the quantization error and maximizing the
information entropy, and then propose an adaptive exponential scheduler to
identify the range of the "dead weights". By considering the "dead weights",
our method offers not only faster BNN training, but also state-of-the-art
performance on CIFAR-10 and ImageNet, compared with recent methods. Code can be
available at https://github.com/z-hXu/ReCU.
