The ZipML Framework for Training Models with End-to-End Low Precision:
  The Cans, the Cannots, and a Little Bit of Deep Learning
  Recently there has been significant interest in training machine-learning
models at low precision: by reducing precision, one can reduce computation and
communication by one order of magnitude. We examine training at reduced
precision, both from a theoretical and practical perspective, and ask: is it
possible to train models at end-to-end low precision with provable guarantees?
Can this lead to consistent order-of-magnitude speedups? We present a framework
called ZipML to answer these questions. For linear models, the answer is yes.
We develop a simple framework based on one simple but novel strategy called
double sampling. Our framework is able to execute training at low precision
with no bias, guaranteeing convergence, whereas naive quantization would
introduce significant bias. We validate our framework across a range of
applications, and show that it enables an FPGA prototype that is up to 6.5x
faster than an implementation using full 32-bit precision. We further develop a
variance-optimal stochastic quantization strategy and show that it can make a
significant difference in a variety of settings. When applied to linear models
together with double sampling, we save up to another 1.7x in data movement
compared with uniform quantization. When training deep networks with quantized
models, we achieve higher accuracy than the state-of-the-art XNOR-Net. Finally,
we extend our framework through approximation to non-linear models, such as
SVM. We show that, although using low-precision data induces bias, we can
appropriately bound and control the bias. We find in practice 8-bit precision
is often sufficient to converge to the correct solution. Interestingly,
however, in practice we notice that our framework does not always outperform
the naive rounding approach. We discuss this negative result in detail.
