On The Reliability Of Machine Learning Applications In Manufacturing
  Environments
  The increasing deployment of advanced digital technologies such as Internet
of Things (IoT) devices and Cyber-Physical Systems (CPS) in industrial
environments is enabling the productive use of machine learning (ML) algorithms
in the manufacturing domain. As ML applications transcend from research to
productive use in real-world industrial environments, the question of
reliability arises. Since the majority of ML models are trained and evaluated
on static datasets, continuous online monitoring of their performance is
required to build reliable systems. Furthermore, concept and sensor drift can
lead to degrading accuracy of the algorithm over time, thus compromising
safety, acceptance and economics if undetected and not properly addressed. In
this work, we exemplarily highlight the severity of the issue on a publicly
available industrial dataset which was recorded over the course of 36 months
and explain possible sources of drift. We assess the robustness of ML
algorithms commonly used in manufacturing and show, that the accuracy strongly
declines with increasing drift for all tested algorithms. We further
investigate how uncertainty estimation may be leveraged for online performance
estimation as well as drift detection as a first step towards continually
learning applications. The results indicate, that ensemble algorithms like
random forests show the least decay of confidence calibration under drift.
