Performance Aware Convolutional Neural Network Channel Pruning for
  Embedded GPUs
  Convolutional Neural Networks (CNN) are becoming a common presence in many
applications and services, due to their superior recognition accuracy. They are
increasingly being used on mobile devices, many times just by porting large
models designed for server space, although several model compression techniques
have been considered. One model compression technique intended to reduce
computations is channel pruning. Mobile and embedded systems now have GPUs
which are ideal for the parallel computations of neural networks and for their
lower energy cost per operation. Specialized libraries perform these neural
network computations through highly optimized routines. As we find in our
experiments, these libraries are optimized for the most common network shapes,
making uninstructed channel pruning inefficient. We evaluate higher level
libraries, which analyze the input characteristics of a convolutional layer,
based on which they produce optimized OpenCL (Arm Compute Library and TVM) and
CUDA (cuDNN) code. However, in reality, these characteristics and subsequent
choices intended for optimization can have the opposite effect. We show that a
reduction in the number of convolutional channels, pruning 12% of the initial
size, is in some cases detrimental to performance, leading to 2x slowdown. On
the other hand, we also find examples where performance-aware pruning achieves
the intended results, with performance speedups of 3x with cuDNN and above 10x
with Arm Compute Library and TVM. Our findings expose the need for
hardware-instructed neural network pruning.
