Memory-Aware Fusing and Tiling of Neural Networks for Accelerated Edge
  Inference
  A rising research challenge is running costly machine learning (ML) networks
locally on resource-constrained edge devices. ML networks with large
convolutional layers can easily exceed available memory, increasing latency due
to excessive swapping. Previous memory reduction techniques such as pruning and
quantization reduce model accuracy and often require retraining. Alternatively,
distributed methods partition the convolutions into equivalent smaller
sub-computations, but the implementations introduce communication costs and
require a network of devices. However, a distributed partitioning approach can
also be used to run in a reduced memory footprint on a single device by
subdividing the network into smaller operations.
  This report extends prior work on distributed partitioning using tiling and
fusing of convolutional layers into a memory-aware execution on a single
device. Our approach extends prior fusing strategies to allow for two groups of
convolutional layers that are fused and tiled independently. This approach
reduces overhead via data reuse, and reduces the memory footprint further. We
also propose a memory usage predictor coupled with a search algorithm to
provide fusing and tiling configurations for an arbitrary set of convolutional
layers. When applied to the YOLOv2 object detection network, results show that
our approach can run in less than half the memory, and with a speedup of up to
2.78 under severe memory constraints. Additionally, our algorithm will return a
configuration with a latency that is within 6% of the best latency measured in
a manual search.
