Incomplete Dot Products for Dynamic Computation Scaling in Neural
  Network Inference
  We propose the use of incomplete dot products (IDP) to dynamically adjust the
number of input channels used in each layer of a convolutional neural network
during feedforward inference. IDP adds monotonically non-increasing
coefficients, referred to as a "profile", to the channels during training. The
profile orders the contribution of each channel in non-increasing order. At
inference time, the number of channels used can be dynamically adjusted to
trade off accuracy for lowered power consumption and reduced latency by
selecting only a beginning subset of channels. This approach allows for a
single network to dynamically scale over a computation range, as opposed to
training and deploying multiple networks to support different levels of
computation scaling. Additionally, we extend the notion to multiple profiles,
each optimized for some specific range of computation scaling. We present
experiments on the computation and accuracy trade-offs of IDP for popular image
classification models and datasets. We demonstrate that, for MNIST and
CIFAR-10, IDP reduces computation significantly, e.g., by 75%, without
significantly compromising accuracy. We argue that IDP provides a convenient
and effective means for devices to lower computation costs dynamically to
reflect the current computation budget of the system. For example, VGG-16 with
50% IDP (using only the first 50% of channels) achieves 70% in accuracy on the
CIFAR-10 dataset compared to the standard network which achieves only 35%
accuracy when using the reduced channel set.
