Pruning-Aware Merging for Efficient Multitask Inference
  Many mobile applications demand selective execution of multiple correlated
deep learning inference tasks on resource-constrained platforms. Given a set of
deep neural networks, each pre-trained for a single task, it is desired that
executing arbitrary combinations of tasks yields minimal computation cost.
Pruning each network separately yields suboptimal computation cost due to task
relatedness. A promising remedy is to merge the networks into a multitask
network to eliminate redundancy across tasks before network pruning. However,
pruning a multitask network combined by existing network merging schemes cannot
minimise the computation cost of every task combination because they do not
consider such a future pruning. To this end, we theoretically identify the
conditions such that pruning a multitask network minimises the computation of
all task combinations. On this basis, we propose Pruning-Aware Merging (PAM), a
heuristic network merging scheme to construct a multitask network that
approximates these conditions. The merged network is then ready to be further
pruned by existing network pruning methods. Evaluations with different pruning
schemes, datasets, and network architectures show that PAM achieves up to 4.87x
less computation against the baseline without network merging, and up to 2.01x
less computation against the baseline with a state-of-the-art network merging
scheme.
