ShiftNAS: Towards Automatic Generation of Advanced Mulitplication-Less
  Neural Networks
  Multiplication-less neural networks significantly reduce the time and energy
cost on the hardware platform, as the compute-intensive multiplications are
replaced with lightweight bit-shift operations. However, existing bit-shift
networks are all directly transferred from state-of-the-art convolutional
neural networks (CNNs), which lead to non-negligible accuracy drop or even
failure of model convergence. To combat this, we propose ShiftNAS, the first
framework tailoring Neural Architecture Search (NAS) to substantially reduce
the accuracy gap between bit-shift neural networks and their real-valued
counterparts. Specifically, we pioneer dragging NAS into a shift-oriented
search space and endow it with the robust topology-related search strategy and
custom regularization and stabilization. As a result, our ShiftNAS breaks
through the incompatibility of traditional NAS methods for bit-shift neural
networks and achieves more desirable performance in terms of accuracy and
convergence. Extensive experiments demonstrate that ShiftNAS sets a new
state-of-the-art for bit-shift neural networks, where the accuracy increases
(1.69-8.07)% on CIFAR10, (5.71-18.09)% on CIFAR100 and (4.36-67.07)% on
ImageNet, especially when many conventional CNNs fail to converge on ImageNet
with bit-shift weights.
