Demystifying and Generalizing BinaryConnect
  BinaryConnect (BC) and its many variations have become the de facto standard
for neural network quantization. However, our understanding of the inner
workings of BC is still quite limited. We attempt to close this gap in four
different aspects: (a) we show that existing quantization algorithms, including
post-training quantization, are surprisingly similar to each other; (b) we
argue for proximal maps as a natural family of quantizers that is both easy to
design and analyze; (c) we refine the observation that BC is a special case of
dual averaging, which itself is a special case of the generalized conditional
gradient algorithm; (d) consequently, we propose ProxConnect (PC) as a
generalization of BC and we prove its convergence properties by exploiting the
established connections. We conduct experiments on CIFAR-10 and ImageNet, and
verify that PC achieves competitive performance.
