Recurrence of Optimum for Training Weight and Activation Quantized
  Networks
  Deep neural networks (DNNs) are quantized for efficient inference on
resource-constrained platforms. However, training deep learning models with
low-precision weights and activations involves a demanding optimization task,
which calls for minimizing a stage-wise loss function subject to a discrete
set-constraint. While numerous training methods have been proposed, existing
studies for full quantization of DNNs are mostly empirical. From a theoretical
point of view, we study practical techniques for overcoming the combinatorial
nature of network quantization. Specifically, we investigate a simple yet
powerful projected gradient-like algorithm for quantizing two-linear-layer
networks, which proceeds by repeatedly moving one step at float weights in the
negation of a heuristic \emph{fake} gradient of the loss function (so-called
coarse gradient) evaluated at quantized weights. For the first time, we prove
that under mild conditions, the sequence of quantized weights recurrently
visits the global optimum of the discrete minimization problem for training
fully quantized network. We also show numerical evidence of the recurrence
phenomenon of weight evolution in training quantized deep networks.
