Intel nGraph: An Intermediate Representation, Compiler, and Executor for
  Deep Learning
  The Deep Learning (DL) community sees many novel topologies published each
year. Achieving high performance on each new topology remains challenging, as
each requires some level of manual effort. This issue is compounded by the
proliferation of frameworks and hardware platforms. The current approach, which
we call "direct optimization", requires deep changes within each framework to
improve the training performance for each hardware backend (CPUs, GPUs, FPGAs,
ASICs) and requires $\mathcal{O}(fp)$ effort; where $f$ is the number of
frameworks and $p$ is the number of platforms. While optimized kernels for
deep-learning primitives are provided via libraries like Intel Math Kernel
Library for Deep Neural Networks (MKL-DNN), there are several compiler-inspired
ways in which performance can be further optimized. Building on our experience
creating neon (a fast deep learning library on GPUs), we developed Intel
nGraph, a soon to be open-sourced C++ library to simplify the realization of
optimized deep learning performance across frameworks and hardware platforms.
Initially-supported frameworks include TensorFlow, MXNet, and Intel neon
framework. Initial backends are Intel Architecture CPUs (CPU), the Intel(R)
Nervana Neural Network Processor(R) (NNP), and NVIDIA GPUs. Currently supported
compiler optimizations include efficient memory management and data layout
abstraction. In this paper, we describe our overall architecture and its core
components. In the future, we envision extending nGraph API support to a wider
range of frameworks, hardware (including FPGAs and ASICs), and compiler
optimizations (training versus inference optimizations, multi-node and
multi-device scaling via efficient sub-graph partitioning, and HW-specific
compounding of operations).
