Amazon SageMaker Model Monitor: A System for Real-Time Insights into
  Deployed Machine Learning Models
  With the increasing adoption of machine learning (ML) models and systems in
high-stakes settings across different industries, guaranteeing a model's
performance after deployment has become crucial. Monitoring models in
production is a critical aspect of ensuring their continued performance and
reliability. We present Amazon SageMaker Model Monitor, a fully managed service
that continuously monitors the quality of machine learning models hosted on
Amazon SageMaker. Our system automatically detects data, concept, bias, and
feature attribution drift in models in real-time and provides alerts so that
model owners can take corrective actions and thereby maintain high quality
models. We describe the key requirements obtained from customers, system design
and architecture, and methodology for detecting different types of drift.
Further, we provide quantitative evaluations followed by use cases, insights,
and lessons learned from more than 1.5 years of production deployment.
