Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning
  Programs
  Imperative programming allows users to implement their deep neural networks
(DNNs) easily and has become an essential part of recent deep learning (DL)
frameworks. Recently, several systems have been proposed to combine the
usability of imperative programming with the optimized performance of symbolic
graph execution. Such systems convert imperative Python DL programs to
optimized symbolic graphs and execute them. However, they cannot fully support
the usability of imperative programming. For example, if an imperative DL
program contains a Python feature with no corresponding symbolic representation
(e.g., third-party library calls or unsupported dynamic control flows) they
fail to execute the program. To overcome this limitation, we propose Terra, an
imperative-symbolic co-execution system that can handle any imperative DL
programs while achieving the optimized performance of symbolic graph execution.
To achieve this, Terra builds a symbolic graph by decoupling DL operations from
Python features. Then, Terra conducts the imperative execution to support all
Python features, while delegating the decoupled operations to the symbolic
execution. We evaluated the performance improvement and coverage of Terra with
ten imperative DL programs for several DNN architectures. The results show that
Terra can speed up the execution of all ten imperative DL programs, whereas
AutoGraph, one of the state-of-the-art systems, fails to execute five of them.
