A Visual Technique to Analyze Flow of Information in a Machine Learning
  System
  Machine learning (ML) algorithms and machine learning based software systems
implicitly or explicitly involve complex flow of information between various
entities such as training data, feature space, validation set and results.
Understanding the statistical distribution of such information and how they
flow from one entity to another influence the operation and correctness of such
systems, especially in large-scale applications that perform classification or
prediction in real time. In this paper, we propose a visual approach to
understand and analyze flow of information during model training and serving
phases. We build the visualizations using a technique called Sankey Diagram -
conventionally used to understand data flow among sets - to address various use
cases of in a machine learning system. We demonstrate how the proposed
technique, tweaked and twisted to suit a classification problem, can play a
critical role in better understanding of the training data, the features, and
the classifier performance. We also discuss how this technique enables
diagnostic analysis of model predictions and comparative analysis of
predictions from multiple classifiers. The proposed concept is illustrated with
the example of categorization of millions of products in the e-commerce domain
- a multi-class hierarchical classification problem.
