DNNFusion: Accelerating Deep Neural Networks Execution with Advanced
  Operator Fusion
  Deep Neural Networks (DNNs) have emerged as the core enabler of many major
applications on mobile devices. To achieve high accuracy, DNN models have
become increasingly deep with hundreds or even thousands of operator layers,
leading to high memory and computational requirements for inference. Operator
fusion (or kernel/layer fusion) is key optimization in many state-of-the-art
DNN execution frameworks, such as TensorFlow, TVM, and MNN. However, these
frameworks usually adopt fusion approaches based on certain patterns that are
too restrictive to cover the diversity of operators and layer connections.
Polyhedral-based loop fusion techniques, on the other hand, work on a low-level
view of the computation without operator-level information, and can also miss
potential fusion opportunities. To address this challenge, this paper proposes
a novel and extensive loop fusion framework called DNNFusion. The basic idea of
this work is to work at an operator view of DNNs, but expand fusion
opportunities by developing a classification of both individual operators and
their combinations. In addition, DNNFusion includes 1) a novel
mathematical-property-based graph rewriting framework to reduce evaluation
costs and facilitate subsequent operator fusion, 2) an integrated fusion plan
generation that leverages the high-level analysis and accurate light-weight
profiling, and 3) additional optimizations during fusion code generation.
DNNFusion is extensively evaluated on 15 DNN models with varied types of tasks,
model sizes, and layer counts. The evaluation results demonstrate that
DNNFusion finds up to 8.8x higher fusion opportunities, outperforms four
state-of-the-art DNN execution frameworks with 9.3x speedup. The memory
requirement reduction and speedups can enable the execution of many of the
target models on mobile devices and even make them part of a real-time
application.
