FusionStitching: Boosting Memory Intensive Computations for Deep
  Learning Workloads
  We show in this work that memory intensive computations can result in severe
performance problems due to off-chip memory access and CPU-GPU context switch
overheads in a wide range of deep learning models. For this problem, current
just-in-time (JIT) kernel fusion and code generation techniques have
limitations, such as rough fusion plan exploration strategies and limited code
generation ability. We propose FusionStitching, a deep learning compiler
capable of fusing memory intensive operators, with varied data dependencies and
non-homogeneous parallelism, into large GPU kernels to reduce global memory
access and context switch overhead automatically. FusionStitching widens the
range of operation combinations that fusion can target beyond previous JIT
works by introducing data reuse of intermediate values. It explores large
fusion spaces to decide optimal fusion plans with considerations of memory
access costs, kernel calls and resource usage constraints. FusionStitching
tunes the optimal stitching scheme with a domain-specific cost model
efficiently. Experimental results show that FusionStitching can reach up to
2.21x speedup compared to state-of-the-art, with 1.45x on average. Besides
these experimental results, we integrated our approach into a compiler product
and deployed it onto a production cluster for AI workloads with thousands of
GPUs. The system has been in operation for more than 4 months and saves 7,000
GPU hours on average for approximately 30,000 tasks per month.
