To Compress, or Not to Compress: Characterizing Deep Learning Model
  Compression for Embedded Inference
  The recent advances in deep neural networks (DNNs) make them attractive for
embedded systems. However, it can take a long time for DNNs to make an
inference on resource-constrained computing devices. Model compression
techniques can address the computation issue of deep inference on embedded
devices. This technique is highly attractive, as it does not rely on
specialized hardware, or computation-offloading that is often infeasible due to
privacy concerns or high latency. However, it remains unclear how model
compression techniques perform across a wide range of DNNs. To design efficient
embedded deep learning solutions, we need to understand their behaviors. This
work develops a quantitative approach to characterize model compression
techniques on a representative embedded deep learning architecture, the NVIDIA
Jetson Tx2. We perform extensive experiments by considering 11 influential
neural network architectures from the image classification and the natural
language processing domains. We experimentally show that how two mainstream
compression techniques, data quantization and pruning, perform on these network
architectures and the implications of compression techniques to the model
storage size, inference time, energy consumption and performance metrics. We
demonstrate that there are opportunities to achieve fast deep inference on
embedded systems, but one must carefully choose the compression settings. Our
results provide insights on when and how to apply model compression techniques
and guidelines for designing efficient embedded deep learning systems.
