TASO: Time and Space Optimization for Memory-Constrained DNN Inference
  Convolutional neural networks (CNNs) are used in many embedded applications,
from industrial robotics and automation systems to biometric identification on
mobile devices. State-of-the-art classification is typically achieved by large
networks, which are prohibitively expensive to run on mobile and embedded
devices with tightly constrained memory and energy budgets. We propose an
approach for ahead-of-time domain specific optimization of CNN models, based on
an integer linear programming (ILP) for selecting primitive operations to
implement convolutional layers. We optimize the trade-off between execution
time and memory consumption by: 1) attempting to minimize execution time across
the whole network by selecting data layouts and primitive operations to
implement each layer; and 2) allocating an appropriate workspace that reflects
the upper bound of memory footprint per layer. These two optimization
strategies can be used to run any CNN on any platform with a C compiler. Our
evaluation with a range of popular ImageNet neural architectures (GoogleNet,
AlexNet, VGG, ResNet and SqueezeNet) on the ARM Cortex-A15 yields speedups of
8x compared to a greedy algorithm based primitive selection, reduces memory
requirement by 2.2x while sacrificing only 15% of inference time compared to a
solver that considers inference time only. In addition, our optimization
approach exposes a range of optimal points for different configurations across
the Pareto frontier of memory and latency trade-off, which can be used under
arbitrary system constraints.
