Improving Device-Edge Cooperative Inference of Deep Learning via 2-Step
  Pruning
  Deep neural networks (DNNs) are state-of-the-art solutions for many machine
learning applications, and have been widely used on mobile devices. Running
DNNs on resource-constrained mobile devices often requires the help from edge
servers via computation offloading. However, offloading through a
bandwidth-limited wireless link is non-trivial due to the tight interplay
between the computation resources on mobile devices and wireless resources.
Existing studies have focused on cooperative inference where DNN models are
partitioned at different neural network layers, and the two parts are executed
at the mobile device and the edge server, respectively. Since the output data
size of a DNN layer can be larger than that of the raw data, offloading
intermediate data between layers can suffer from high transmission latency
under limited wireless bandwidth. In this paper, we propose an efficient and
flexible 2-step pruning framework for DNN partition between mobile devices and
edge servers. In our framework, the DNN model only needs to be pruned once in
the training phase where unimportant convolutional filters are removed
iteratively. By limiting the pruning region, our framework can greatly reduce
either the wireless transmission workload of the device or the total
computation workload. A series of pruned models are generated in the training
phase, from which the framework can automatically select to satisfy varying
latency and accuracy requirements. Furthermore, coding for the intermediate
data is added to provide extra transmission workload reduction. Our experiments
show that the proposed framework can achieve up to 25.6$\times$ reduction on
transmission workload, 6.01$\times$ acceleration on total computation and
4.81$\times$ reduction on end-to-end latency as compared to partitioning the
original DNN model without pruning.
