Scanflow: A multi-graph framework for Machine Learning workflow
  management, supervision, and debugging
  Machine Learning (ML) is more than just training models, the whole workflow
must be considered. Once deployed, a ML model needs to be watched and
constantly supervised and debugged to guarantee its validity and robustness in
unexpected situations. Debugging in ML aims to identify (and address) the model
weaknesses in not trivial contexts. Several techniques have been proposed to
identify different types of model weaknesses, such as bias in classification,
model decay, adversarial attacks, etc., yet there is not a generic framework
that allows them to work in a collaborative, modular, portable, iterative way
and, more importantly, flexible enough to allow both human- and machine-driven
techniques. In this paper, we propose a novel containerized directed graph
framework to support and accelerate end-to-end ML workflow management,
supervision, and debugging. The framework allows defining and deploying ML
workflows in containers, tracking their metadata, checking their behavior in
production, and improving the models by using both learned and human-provided
knowledge. We demonstrate these capabilities by integrating in the framework
two hybrid systems to detect data drift distribution which identify the samples
that are far from the latent space of the original distribution, ask for human
intervention, and whether retrain the model or wrap it with a filter to remove
the noise of corrupted data at inference time. We test these systems on
MNIST-C, CIFAR-10-C, and FashionMNIST-C datasets, obtaining promising accuracy
results with the help of human involvement.
