Mixed Precision DNNs: All you need is a good parametrization
  Efficient deep neural network (DNN) inference on mobile or embedded devices
typically involves quantization of the network parameters and activations. In
particular, mixed precision networks achieve better performance than networks
with homogeneous bitwidth for the same size constraint. Since choosing the
optimal bitwidths is not straight forward, training methods, which can learn
them, are desirable. Differentiable quantization with straight-through
gradients allows to learn the quantizer's parameters using gradient methods. We
show that a suited parametrization of the quantizer is the key to achieve a
stable training and a good final performance. Specifically, we propose to
parametrize the quantizer with the step size and dynamic range. The bitwidth
can then be inferred from them. Other parametrizations, which explicitly use
the bitwidth, consistently perform worse. We confirm our findings with
experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with
learned quantization parameters, achieving state-of-the-art performance.
