Neural Network Inference on Mobile SoCs
  The ever-increasing demand from mobile Machine Learning (ML) applications
calls for evermore powerful on-chip computing resources. Mobile devices are
empowered with heterogeneous multi-processor Systems-on-Chips (SoCs) to process
ML workloads such as Convolutional Neural Network (CNN) inference. Mobile SoCs
house several different types of ML capable components on-die, such as CPU,
GPU, and accelerators. These different components are capable of independently
performing inference but with very different power-performance characteristics.
In this article, we provide a quantitative evaluation of the inference
capabilities of the different components on mobile SoCs. We also present
insights behind their respective power-performance behavior. Finally, we
explore the performance limit of the mobile SoCs by synergistically engaging
all the components concurrently. We observe that a mobile SoC provides up to 2x
improvement with parallel inference when all its components are engaged, as
opposed to engaging only one component.
