Scaling Up Deep Neural Network Optimization for Edge Inference
  Deep neural networks (DNNs) have been increasingly deployed on and integrated
with edge devices, such as mobile phones, drones, robots and wearables. To run
DNN inference directly on edge devices (a.k.a. edge inference) with a
satisfactory performance, optimizing the DNN design (e.g., network architecture
and quantization policy) is crucial. While state-of-the-art DNN designs have
leveraged performance predictors to speed up the optimization process, they are
device-specific (i.e., each predictor for only one target device) and hence
cannot scale well in the presence of extremely diverse edge devices. Moreover,
even with performance predictors, the optimizer (e.g., search-based
optimization) can still be time-consuming when optimizing DNNs for many
different devices. In this work, we propose two approaches to scaling up DNN
optimization. In the first approach, we reuse the performance predictors built
on a proxy device, and leverage the performance monotonicity to scale up the
DNN optimization without re-building performance predictors for each different
device. In the second approach, we build scalable performance predictors that
can estimate the resulting performance (e.g., inference
accuracy/latency/energy) given a DNN-device pair, and use a neural
network-based automated optimizer that takes both device features and
optimization parameters as input and then directly outputs the optimal DNN
design without going through a lengthy optimization process for each individual
device.
