Did the Model Change? Efficiently Assessing Machine Learning API Shifts
  Machine learning (ML) prediction APIs are increasingly widely used. An ML API
can change over time due to model updates or retraining. This presents a key
challenge in the usage of the API because it is often not clear to the user if
and how the ML model has changed. Model shifts can affect downstream
application performance and also create oversight issues (e.g. if consistency
is desired). In this paper, we initiate a systematic investigation of ML API
shifts. We first quantify the performance shifts from 2020 to 2021 of popular
ML APIs from Google, Microsoft, Amazon, and others on a variety of datasets. We
identified significant model shifts in 12 out of 36 cases we investigated.
Interestingly, we found several datasets where the API's predictions became
significantly worse over time. This motivated us to formulate the API shift
assessment problem at a more fine-grained level as estimating how the API
model's confusion matrix changes over time when the data distribution is
constant. Monitoring confusion matrix shifts using standard random sampling can
require a large number of samples, which is expensive as each API call costs a
fee. We propose a principled adaptive sampling algorithm, MASA, to efficiently
estimate confusion matrix shifts. MASA can accurately estimate the confusion
matrix shifts in commercial ML APIs using up to 90% fewer samples compared to
random sampling. This work establishes ML API shifts as an important problem to
study and provides a cost-effective approach to monitor such shifts.
