Who supervises the supervisor? Model monitoring in production using deep
  feature embeddings with applications to workpiece inspection
  The automation of condition monitoring and workpiece inspection plays an
essential role in maintaining high quality as well as high throughput of the
manufacturing process. To this end, the recent rise of developments in machine
learning has lead to vast improvements in the area of autonomous process
supervision. However, the more complex and powerful these models become, the
less transparent and explainable they generally are as well. One of the main
challenges is the monitoring of live deployments of these machine learning
systems and raising alerts when encountering events that might impact model
performance. In particular, supervised classifiers are typically build under
the assumption of stationarity in the underlying data distribution. For
example, a visual inspection system trained on a set of material surface
defects generally does not adapt or even recognize gradual changes in the data
distribution - an issue known as "data drift" - such as the emergence of new
types of surface defects. This, in turn, may lead to detrimental
mispredictions, e.g. samples from new defect classes being classified as
non-defective. To this end, it is desirable to provide real-time tracking of a
classifier's performance to inform about the putative onset of additional error
classes and the necessity for manual intervention with respect to classifier
re-training. Here, we propose an unsupervised framework that acts on top of a
supervised classification system, thereby harnessing its internal deep feature
representations as a proxy to track changes in the data distribution during
deployment and, hence, to anticipate classifier performance degradation.
