A System for Real-Time Interactive Analysis of Deep Learning Training
  Performing diagnosis or exploratory analysis during the training of deep
learning models is challenging but often necessary for making a sequence of
decisions guided by the incremental observations. Currently available systems
for this purpose are limited to monitoring only the logged data that must be
specified before the training process starts. Each time a new information is
desired, a cycle of stop-change-restart is required in the training process.
These limitations make interactive exploration and diagnosis tasks difficult,
imposing long tedious iterations during the model development. We present a new
system that enables users to perform interactive queries on live processes
generating real-time information that can be rendered in multiple formats on
multiple surfaces in the form of several desired visualizations simultaneously.
To achieve this, we model various exploratory inspection and diagnostic tasks
for deep learning training processes as specifications for streams using a
map-reduce paradigm with which many data scientists are already familiar. Our
design achieves generality and extensibility by defining composable primitives
which is a fundamentally different approach than is used by currently available
systems. The open source implementation of our system is available as
TensorWatch project at https://github.com/microsoft/tensorwatch.
