Enabling Binary Neural Network Training on the Edge
  The ever-growing computational demands of increasingly complex machine
learning models frequently necessitate the use of powerful cloud-based
infrastructure for their training. Binary neural networks are known to be
promising candidates for on-device inference due to their extreme compute and
memory savings over higher-precision alternatives. However, their existing
training methods require the concurrent storage of high-precision activations
for all layers, generally making learning on memory-constrained devices
infeasible. In this paper, we demonstrate that the backward propagation
operations needed for binary neural network training are strongly robust to
quantization, thereby making on-the-edge learning with modern models a
practical proposition. We introduce a low-cost binary neural network training
strategy exhibiting sizable memory footprint and energy reductions while
inducing little to no accuracy loss vs Courbariaux & Bengio's standard
approach. These resource decreases are primarily enabled through the retention
of activations exclusively in binary format. Against the latter algorithm, our
drop-in replacement sees coincident memory requirement and energy consumption
drops of 2--6$\times$, while reaching similar test accuracy in comparable time,
across a range of small-scale models trained to classify popular datasets. We
also demonstrate from-scratch ImageNet training of binarized ResNet-18,
achieving a 3.12$\times$ memory reduction. Such savings will allow for
unnecessary cloud offloading to be avoided, reducing latency, increasing energy
efficiency and safeguarding privacy.
