Benanza: Automatic $\mu$Benchmark Generation to Compute "Lower-bound"
  Latency and Inform Optimizations of Deep Learning Models on GPUs
  As Deep Learning (DL) models have been increasingly used in latency-sensitive
applications, there has been a growing interest in improving their response
time. An important venue for such improvement is to profile the execution of
these models and characterize their performance to identify possible
optimization opportunities. However, the current profiling tools lack the
highly desired abilities to characterize ideal performance, identify sources of
inefficiency, and quantify the benefits of potential optimizations. Such
deficiencies have led to slow characterization/optimization cycles that cannot
keep up with the fast pace at which new DL models are introduced.
  We propose Benanza, a sustainable and extensible benchmarking and analysis
design that speeds up the characterization/optimization cycle of DL models on
GPUs. Benanza consists of four major components: a model processor that parses
models into an internal representation, a configurable benchmark generator that
automatically generates micro-benchmarks given a set of models, a database of
benchmark results, and an analyzer that computes the "lower-bound" latency of
DL models using the benchmark data and informs optimizations of model
execution. The "lower-bound" latency metric estimates the ideal model execution
on a GPU system and serves as the basis for identifying optimization
opportunities in frameworks or system libraries. We used Benanza to evaluate 30
ONNX models in MXNet, ONNX Runtime, and PyTorch on 7 GPUs ranging from Kepler
to the latest Turing, and identified optimizations in parallel layer execution,
cuDNN convolution algorithm selection, framework inefficiency, layer fusion,
and using Tensor Cores.
