{'fraction_non_alphanumeric': 0.03760304868564318, 'fraction_numerical': 0.008282781147923472, 'mean_word_length': 5.011337073398784, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Observability for Production Machine Learning Pipelines

Software organizations are increasingly incorporating machine learning (ML) into their product offerings, driving a need for new data management tools. Many of these tools facilitate the initial development of ML applications, but sustaining these applications post-deployment is difficult due to lack of real-time feedback (i.e., labels) for predictions and silent failures that could occur at any component of the ML pipeline (e.g., data distribution shift or anomalous features). We propose a new type of data management system that offers end-to-end observability, or visibility into complex system behavior, for deployed ML pipelines through assisted (1) detection, (2) diagnosis, and (3) reaction to ML-related bugs. We describe new research challenges and suggest preliminary solution ideas in all three aspects. Finally, we introduce an example architecture for a"bolt-on"ML observability system, or one that wraps around existing tools in the stack.


INTRODUCTION
Organizations are devoting increasingly more resources towards developing and deploying applications powered by machine learning (ML). ML applications rely on pipelines that span multiple heterogeneous stages or components, such as feature generation and model training, requiring specialized data management tools. Most work in data management for ML concentrates on specific components, e.g., preprocessing [1,2], or model training [3,4,5,6]. Additionally, some industry solutions have garnered widespread adoption by handling data management issues that stem from experimenting with models [7,8].
However, there are many unaddressed challenges in sustaining ML pipelines once built: maintaining, debugging, and improving them after the initial deployment. Various best practices for "production ML" and failure case studies highlight the dire need for ML sustainability [9,10]. We posit that for sustainability, ML practitioners should be able to (1) detect, (2) diagnose, and (3) react to bugs post-deployment. Compared to traditional software systems, which typically only break when there are infrastructure issues, ML pipelines can also fail unpredictably due to data issues-and therefore are uniquely challenging to sustain in all three aspects.
Bug Detection: Hard Due to Feedback Delays. It is well-known that data distributions change or shift over time, causing model performance to drop [11,12]. Detecting performance drops postdeployment is challenging due to lack of "ground-truth" data: in many production ML systems, feedback on predictions, or labels, can arrive at a later time. Furthermore, in many pipelines, only a few labels arrive (e.g., labelers manually annotate some predictions, or only a handful of predicted outputs are displayed to the user). As a result, practitioners are unable to monitor simple ML metrics such as accuracy in real-time. As an alternative, end-to-end ML frameworks such as TFX [13] and Sagemaker [14] monitor internal pipeline state or health via distance metrics [15] over distributions of ML features and outputs over time. These proxies often produce too many false positives and thus do not accurately determine when models are underperforming, as we will discuss further in Section 2. Bug Diagnosis: Hard Due to Pipeline Complexity. Even if a failure is confidently detected, the complex, highly intertwined nature of components in the ML pipeline makes it hard to understand where bugs could lie. For production ML pipelines, "changing anything changes everything (CACE), " causing predictions to vary unpredictably [10]. Consequently, production ML uniquely suffers from silent pipeline bugs, such as corrupted or stale subsets of features. Practitioners painstakingly enumerate and maintain (i.e., tune) data quality constraints for component inputs and outputs [16,17], motivating automatic specification and maintenance of precise constraints at the component level. Bug Fixes: Hard Due to No Obviously Correct Answers. Even if users can successfully pinpoint all pipeline bugs, there can be many ways to bring model performance back up to a desirable level, and effectiveness depends on the nature of the data or task. For example, different components, when fixed, can cause different magnitudes of improvement in ML performance. Users often have no sense of what to fix first, relative to the costs in resources and time. ML Observability. The challenges outlined above motivate the need for observability [18], or "better visibility into understanding the complex behavior of software using telemetry collected ... at run time" [19], tailored for ML pipelines. Observability encompasses more than just monitoring predefined metrics that capture holistic system health (i.e., known-unknowns)-it also allows practitioners to ask questions about how systems behaved on historical outputs (i.e., unknown-unknowns), or perform "needle-in-a-haystack" queries. The north star for software observability systems is to give users the power to ask new questions of historical system behavior without gathering new data [20]. Contributions. In this paper, we discuss unaddressed research challenges in ML observability as a call-to-arms for the database community to contribute to this nascent research direction. We propose the concept of a "bolt-on" observability system for ML pipelines-one that does not require users to rewrite all their code to use a specific framework. ML application developers assemble their pipelines in an ad-hoc manner employing a myriad of tools along the way, and our bolt-on observability system must interoperate with such heterogeneous pipelines. For example, practitioners may use a Hive metastore to catalog raw data [21], Deequ for data validation [17], and Weights & Biases for experiment tracking [8].
For our bolt-on observability system to address bug detection, diagnosis, and fixing needs, we propose a three-pronged approach: (1) Monitoring approximations of top-line, i.e., business-critical, ML metrics to alert users of ML performance drops even when there may not be real-time labels. In Section 3.3, we propose automated techniques that rely on lightweight proxies to bin predictions and estimate metrics based on importance weighting, drawing on the approximate query processing and streaming literature.
(2) Given ML performance drops, identifying issues in inputs and outputs for each component in the pipeline to aid diagnosis. In Section 3.4, we propose logging fine-grained information across provenance snapshots, automatically specifying and tuning data quality constraints, and adversarially learning differences between training and live data to track distribution shift.
(3) Tracing ML bugs back to silent data and engineering-related issues (i.e., pipeline bugs). In Section 3.5, we describe tracking feedback delays and column-wide error scores across dataflow graphs to assist practitioners in repairing broken components.
In Section 3, we present a roadmap ( Figure 2) of challenges and preliminary solutions-detecting drops in holistic ML metrics (e.g., accuracy), diagnosing them by tracking point-in-time, componentlevel issues, and reacting to the drops by analyzing cross-time, cross-component issues. In Section 4, we discuss an example of a bolt-on ML observability system architecture and introduce our vision for mltrace, a lightweight bolt-on ML observability tool, which has already received preliminary interest from practitioners with over 400 GitHub stars (github.com/loglabs/mltrace).

BACKGROUND
We discuss prior work in data management for ML pipelines and current end-to-end ML pipeline frameworks. ETL, Assertions, and Experiment Tracking. Input data for ML models is typically constructed through a series of ETL workloads. Faulty predictions can stem from such workloads, such as incorrectly performing missing value imputation [22]. Tools like Dagger [2] and mlinspect [1] help practitioners detect data-related bugs in ML preprocessing components. Our focus is instead on bugs that arise post-deployment. Other tools [7,3,8] focus on experiment tracking, one of the biggest pain points in generating models for production ML pipelines. However, none of these tools determine if and why production pipelines are failing. Data Quality Assertions. Other work [17,23,13,24] proposes libraries of assertions to be embedded in ML code, however, they provide no guidance for which assertions to embed. Since these assertions are often written as part of a main application, they may not be easily reusable across pipelines. Additionally, results of these tests must be externally logged with a separate service for users to query post-hoc. While data quality assertions are certainly valuable for catching egregious issues (e.g., negative values for columns that should be positive), ML pipeline performance can drop over time without failing user-embedded assertions. Detecting data shift. Many papers in the ML literature discuss how various forms of data shift (e.g., concept shift, covariate shift, prior probability shift) cause model performance to degrade [11,25,26]. To address such shift problems, the ML community has proposed monitoring distance metrics across distributions of features and predictions, such as the Kolmogorov-Smirnov (K-S) test statistic for numerical features or dimensionality-reduced features and the Chi-Squared test statistic for categorical features [27]. However, with thousands of features and seasonal changes in data, such methods may not correctly flag shift, might trigger too many alarms and cause alert "fatigue" or result in confusion (e.g., the K-S test statistic is significant for one feature but not another) [16]. Thus, there is a need for higher-precision methods that detect, diagnose, and react to data shift. Some research in post-deployment ML debugging focuses on finding slices (i.e., predicates) where models perform poorly [28,29], but these methods require labels, which are not always available. Unresolved Observability Challenges in Existing Tools. Endto-end frameworks such as Sagemaker [14] and TFX [13] provide logging at the component level but only support primitive monitoring based on user-specified metrics and similarly do not help address data shift. These frameworks also force their users to rewrite their pipeline using their DSLs. For example, to use TFX, users must write their data processing pipelines using Apache Beam, manipulate data with TFData, build models in Tensorflow, and serve models via Tensorflow Serving. To avoid having users perform a cumbersome rewrite, some proprietary tools only monitor features and predictions through an API, which cannot flag all problems or suggest where problems lie in the pipeline because they lack end-to-end visibility. Other declarative frameworks [30,31] allow users to declaratively specify end-to-end ML pipelines without supporting the identification of deployment bugs. Moreover, ML practitioners often prefer to use their homegrown hodgepodge of tools rather than rewrite their code in a separate framework. Thus, we advocate for an observability solution that can interoperate with such tools.

RESEARCH CHALLENGES
Before we describe our research challenges, we first introduce key definitions and an example ML pipeline ( Figure 1) to ground our discussion.

ML Pipeline Preliminaries
3.1.1 Definitions. Here, we define several terms used throughout this paper. An ML pipeline involves multiple data processing components, leading to one or more ML models that provide predictions for a specific task. A metric is a measure of success for an ML pipeline, such as prediction accuracy. A tuple is an individual feature vector used to generate predictions. A live prediction is a prediction made after deployment, as opposed to predictions made during training. The consumers of predictions provide feedback, or some data that indicates the quality of a prediction (e.g., item selection for recommendations, correctness for binary classification). Labels, or "ground-truth" for predictions, are derived from feedback.  Finally, we refer to groups of tuples, defined on conjunctions of predicates on features, as buckets. A bucketing strategy refers to how tuples are assigned to buckets.

Example ML Pipeline.
Using data from the New York City Taxi and Limousine Coalition [32], our ML task is to predict whether a rider will give their driver a tip > 20% of the fare. Predictions are probabilities (i.e., floats between 0 and 1). Each tuple in the dataset (Yellow Trips) represents a single ride, with 17 attributes. Our ML pipeline includes five components, as described by the rectangular boxes in Figure 1. We have two sub-pipelines-training and inference-that share the cleaning and feature generation components. The pipeline includes one model, an sklearn random forest classifier. The ML pipeline is evaluated on accuracy, or the fraction of correct predictions, when the prediction is rounded to the nearest integer.

Formalizing Distribution Shift.
In deployment settings, real-time accuracy is often difficult to measure due to feedback delays, so practitioners monitor changes, or shifts, in distributions of features and predictions. ML practitioners have introduced any number of types of shifts, such as concept, data, covariate, label, subpopulation, prior probability, and low-data shifts, among othersand these definitions often conflict [11,25,26,33,34]. If is the label space and is the feature or covariate space (e.g., location of ride, number of passengers), we note that all of the aforementioned shift definitions boil down to at least one of the two shift scenarios: Concept shift: ( | ) changes; ( ) changes but ( ) doesn't Covariate shift: ( ) and ( ) change but ( | ) doesn't A concrete example of concept shift is a recession: riders tip less, changing ( ) but not ( ). A concrete example of covariate shift is New Year's Eve: the number of taxi rides will be relatively higher near Times Square in New York, changing ( ) and ( ) as a result, even though the nature of a taxi ride that results in a high tip does not change, i.e., ( | ). There's ML literature on learning under these natural shifts [35,36]; however, our research challenges focus on the unexpected combinations of shifts that arise in production.
The rationale for tracking ( ) and ( ) over time in production pipelines is that significant changes in these values can indicate new data quality or engineering bugs that need to be fixed. However, methods to flag changes in distributions, as mentioned in Section 2, cause too many false positive alerts. For example, practitioners compute the K-S test statistic between training and live tuples for each feature to approximate how ( ) has changed, often yielding thousands of measures, which can be confusing to navigate. For instance, what would a user do with an alert saying a handful of their thousand features' K-S test statistics are now statistically significant? Does this alert really impact ML accuracy? Additionally, on large datasets, -values can go to zero even without actual significance [37], further exacerbating alert fatigue.

Research Roadmap
As shown in Figure 2, we employ a three-pronged framework of detecting (Section 3.3), diagnosing (Section 3.4), and reacting (Section 3.5) to bugs in ML pipelines after deployment: Detection. This prong answers the question how is the deployed ML pipeline doing in real-time, with a focus on performance measures such as accuracy. There are two challenges in estimating performance. First, the lack of labels, which happens soon after deployment (Section 3.3.1), and second, labels are available but arbitrarily delayed (Section 3.3.2). In the latter case, estimating realtime performance requires a join across the out-of-sync label and prediction streams, which is difficult at scale.
Diagnosis. Given a drop in performance, diagnosis answers the question which components of the pipeline are potential sources of errors, with a focus on a single component and a single point in time. To make sense of errors in individual components, we need to log intermediate inputs/outputs and provenance (Section 3.4.1). With this logging in place, we should automatically specify and tune data validation constraints to address spectrum of pipeline bugs, from hard and soft constraint violations (Section 3.4.2), to data shift (Section 3.4.3).
Reaction. With errors in individual components identified, reaction answers the question what fixes to the pipeline can help address errors, across components and time. To help fix the pipeline, we need to both make sure that any sources of label lag are addressed (Section 3.5.1)-to ensure that we have better estimates of performance measures, and that the cross-component and cross-time issues are addressed (Section 3.5.2).

Detecting ML Performance Issues
Post-deployment, the starting point for identifying issues is monitoring drops in ML metrics such as accuracy. This becomes challenging when labels or predictions are delayed or absent. Moreover, delays may not be uniform across buckets (e.g., a power outage in East Village might prevent taxicab meter information from being uploaded). As shown in Figure 1, predictions and feedback arrive at different timestamps and are joined on some identifier. At every timestamp, ML pipelines can move between three feedback scenarios: no feedback, partial feedback, and full feedback. There are two key challenges: first, the lack of labels (impacting the partial and no feedback settings), and second, arbitrary label delays (impacting the partial and full feedback settings). We discuss both of these challenges in turn.
Before we discuss the challenges, when estimating real-time accuracy, there are at least three variants of interest: (a) cumulative accuracy for all predictions made until now; (b) accuracy for predictions made in the last time window ; (c) accuracy for the last predictions. The last two variants provide accuracies over a sliding window. The cumulative setting is not just relevant when we are evaluating accuracy from = 0; it is also useful when we reset the clock regularly, e.g., accuracy on a per-day basis. Increasing granularity and sparsity Figure 3: Bucketing strategies, normalized to show bucket density. As buckets become more finer-grained, they also become sparse. labels may arrive in batches at a later date, possibly after human review, motivating us to still find ways to estimate real-time performance without them. To estimate cumulative accuracy, we may use importance weighting (IW) techniques [11]. We can identify buckets based on input feature combinations, determine the training accuracy for each bucket, and weight these accuracies based on the number of points in each bucket in the live (post-deployment, unlabeled) data. Consider neighborhood as a bucketing strategy: if the training set had FiDi and Midtown accuracies of 80% and 50% respectively and we have 100 FiDi and 500 Midtown live predictions, we can estimate an accuracy of 0.8 × 100 + 0.5 × 500 = 55%.
There are multiple competing objectives in determining which bucketing strategy would lead to best estimates of accuracy, among the ( !) possible bucketing schemes, where is the number of features, i.e., any subset of input features. Figure 3 illustrates three bucketing schemes. The first couple have representation in each bucket, which gives us some confidence in per-bucket accuracy. However, the last one has some empty buckets-so if a live tuple were to be assigned to such a bucket, we would not have an accuracy estimate for it. Overall, finer-grained bucketing schemes may capture patterns not found in coarse-grained ones but could also be more sparse, which can impact the correctness of our accuracy estimates. Moreover, finer-grained bucketing schemes would occupy more space than coarse-grained ones. Beyond (i) Space and (ii) Sparsity, there are other objectives we need to consider. (iii) Variance: the buckets should have high variance in training accuracies, leading to more useful estimates. (iv) Predictiveness: the training accuracy for each bucket should be predictive of actual accuracies for live data in that bucket. Balancing these objectives is non-trivial. We may take inspiration from stratified sampling [38] in Approximate Query Processing (AQP) [39,40], and also ensemble schemes [41].
Extending this technique to the sliding window accuracy setting if we are using a fixed offline bucketing may be straightforward. Per bucket, we can apply ideas from prior work in streaming algorithms to update counts for the last tuples [42]; similar techniques may also apply for the sliding window defined by time.
Finally, we may gain additional benefits from changing the bucketing strategy in response to live data. To do so, we must efficiently identify buckets in high-dimensional, changing data streams with a reference dataset in mind (i.e., the training set). A starting point could be to extend streaming clustering algorithms that are explicitly robust to changing data distributions [43]: in addition to the live data, we could feed the training set to such a clustering algorithm.

Label
Delays. Labels are often delayed in arbitrary ways. So, estimating accuracy, which requires a join between prediction and label/feedback streams, is challenging to do at scale, since keeping both streams in memory is impossible. One option is to uniformly subsample both streams, since the usual problems with sampling over joins [44,45,46] don't apply when each prediction tuple joins with precisely one feedback tuple.(It is well-known that uniformly subsampling streams before a join can yield quadratically fewer tuples in the result [44]. Unlike the standard join setting, here, each prediction tuple joins precisely with a single feedback tuple, meaning that the challenges of quadratically fewer samples with AQP over joins do not apply [44,45,46].) Since we do not know the size of the streams, one can apply reservoir sampling [47] on both streams using a shared hash function on the common identifier. However, this approach is wasteful, since once the pair of prediction and feedback tuples are received, they no longer both need to be in memory. Moreover, the quality of the estimate degrades over time since we are maintaining a fixed size sample over growing streams. Ideally we want to maintain both a reservoir (for prediction tuples without feedback) plus partial aggregates (for prediction tuples with feedback). Joined tuples can make way for new slots in the reservoir. However, doing so while respecting the reservoir sampling guarantee of each having the same probability of being sampled, is non-trivial. For example, the sudden arrival of a number of feedback tuples can cause multiple slots in the reservoir to become vacant, leading to an increasing probability for the next prediction tuple to be included in the reservoir.
Extending this reservoir sampling approach to (b) and (c) is also challenging. We can leverage prior work on reservoir sampling over windows [48,47,49,50], where we can evict old tuples from the reservoir when they expire [48], or update the probabilities to favor newer tuples more, using an exponential decay weighting [50]. As before, we will want to modify these techniques to be less wasteful of memory, while also ensuring that they are unbiased.
Finally, we'd want to combine our techniques for dealing with label delays with that for dealing with missing labels (Section 3.3.1). This is because many ML pipelines will be in the partial-feedback setting: often, live data is only partially labeled or arrives on a specific schedule. Or, some upstream data collection issues might delay feedback (e.g., there's a cell tower outage in a region of Tribeca, causing payment meter data to be delayed). Here, aggregating the full-feedback and no-feedback estimates, weighted by the count of tuples in each case, may produce a reasonable real-time accuracy estimate.
Beyond labels, another way to approximate ML pipeline performance is to directly monitor changes in an important business metric (e.g., user satisfaction, click-through-rate, revenue). Sometimes the ML metric (e.g., model accuracy) does not align with a business metric (e.g., user satisfaction, revenue), requiring users to rethink the ML objective or discard the model altogther. An engineering challenge is to provide integrations with other system components that are not directly part of the ML pipeline-e.g., sales tools that record metrics like daily active users. To help users understand the effectiveness of their ML models, we can show correlations between ML metrics and business metrics over time.

Diagnosing ML Performance Issues
After detecting an ML performance drop, we next need to diagnose it by identifying which components have bugs at that time. We focus on bugs that arise after a pipeline deployment (i.e., related to a mismatch in data between training and serving). There is a spectrum of data-centric ML production bugs [51,52]: hard → soft → drift, from most to least time-sensitive. Hard errors, such as some data sources failing to ingest and resulting in missing feature values, need immediate attention. Soft errors, such as features having anomalous means, require more tedious manual investigation because of false positives: many columns can deviate significantly while only a few are responsible for pipeline performance drops. Hard and soft errors are both forms of pipeline errors, which are often addressed by engineering changes to pipeline components. Finally, bugs can result from natural data drift, causing model performance to slowly decrease; nevertheless, they require attention. Unlike pipeline errors, data drift occurs naturally as data evolves and models no longer faithfully capture the underlying relationships. We discuss the challenges addressing pipeline errors and drift errors next, after discussing a prerequisite: logging and provenance.
3.4.1 Logging and Provenance. Logging at the component level helps us uncover whether the output of a given pipeline component has an error. Then, to trace errors across components, we additionally need provenance. Without provenance, practitioners typically diagnose at the model feature and prediction level and use "tribal knowledge" to trace misbehaving features to potential upstream causes. There is extensive work on logging and provenance, e.g., [53,54,55,56,57], and for ML and data science pipelines [58,59,60,53,61,62,7]. Some approaches require using a specific end-to-end ML framework [59] or logging API [58,53,62,7]. Others instrument the AST or bytecode to capture lineage [1,63,61] or employ error-prone static analysis [60], forcing users to remain in a particular language or ML framework. We instead propose a simple bolt-on approach: users annotate pipeline components (e.g., with decorators for Python) with pointers to inputs and outputs (e.g., dataframe variables) that automatically get logged to an observability store. When users want to retrieve a trace, we perform a depth-first-search through logs on the fly, to construct the provenance. This approach offers a nice middle ground: not too onerous, while also letting users control how provenance is captured.
Unfortunately, logging raw inputs and outputs for each component in the ML pipeline can quickly get expensive. As an anecdote, the first author worked at a startup where the MLFlow [7] logs would require a "purge" every few months. To minimize log size, we can use the same approach as in the previous section and use a reservoir sample for prediction tuples; a uniform sample may suffice for training tuples. In addition, we can log histograms instead of full data streams; however, bins should change as data evolves over time. Research challenges lie in combining ideas from incrementally-maintained approximate histograms with ideas from adaptive histograms to produce evolving summaries of windows of data [64]. Another insight is that users will only selectively query logged intermediates (e.g., inspect the head of a dataframe). For each component, we can learn from query patterns over time to inform what goes into logs, thereby reducing latency and storage footprints.

Tracking Pipeline Errors via Auto-tuned Data Integrity
Constraints. ML pipeline errors are typically caught by data validation constraints [16,1,22,65,66]. For example, Schelter et al. [22] defines 25 different types of single-column ML-specific constraints, and two constraints on column pairs, each requiring tediously setting thresholds per column (or pair). The long-term maintenance of these constraints is also a headache: users spend months or years silencing hard constraints into soft constraints to avoid falsely rejecting predictions, converting soft constraints into hard constraints when they experience pipeline bugs, and tuning thresholds for each constraint. Automating creation and maintenance for these constraints while preserving high precision (i.e., all violations correspond to bugs) and recall (i.e., all bugs are caught by violations) is an important challenge. Existing solutions suggest basic automatic constraints such as type checks and set membership for categorical columns; although they have decent precision, recall is low [13].
Given provenance snapshots, we can execute constraint checks on intermediate inputs and outputs and log the results; when ML performance drops are flagged, users can then trace a prediction and inspect such results to determine which component(s) to address. The key question is, what constraint checks do we execute? We propose auto-generating suites of input and output validation constraints and auto-tuning them over time to maximize precision and recall of violations. Constraints should be explainable and thus more actionable for users, rather than seemingly random floatvalued column bounds. If we have access to the user's model, we can narrow the search space by restricting constraints to columns involved in important features [67]; however, it is unrealistic to assume such access, especially for a bolt-on observability system. A solution idea is to learn an autoregressive model [68] that predicts the likelihood of a column's value for a tuple given values of the other columns; then we can aggregate likelihood scores for each column over sliding windows of post-deployment tuples. Deep autoregressive models have demonstrated success in capturing rich multivariate distributions in challenging tasks, such as cardinality estimation [68]. One approach is to fit an autoregressive function for each column to model its distribution at time given the joint distribution of other columns at time , as well as tuples corresponding to times < . However, this does not scale to more than a handful of columns, so we can use masking techniques to train a single model for all columns [69,70,68]. Another challenge is that most off-the-shelf autoregressive models are trained explicitly to predict the next value or token, not to learn the distributions, or density functions, of columns. Thus, we need to discretize inputs and outputs so the model learns a distribution of buckets. A simple bucketization strategy can be derived from the CDF (e.g., quantiles), but this will fail as distributions change over time. Alternatively, recent progress in language modeling suggests a different bucketization strategy-numbers can be discretized into digits [71]. To turn the density estimation model(s) into a suite of constraints, users can define a common threshold for columns, e.g., tuples should have an aggregated column likelihood score above 80%. We can also fine-tune the threshold based on the size of the data (i.e., number of columns and tuples) to find a good trade-off between precision and recall of alerts. Further challenges include efficiently maintaining these autoregressive models and their data (e.g., fine-tuning, running across provenance snapshots).

Addressing Natural Data
Drift. Data integrity checks do not flag slower, longer-term distribution shift, motivating the need to track how data distributions change over time. For instance, a recession could cause riders to tip less across the population, changing ( ) but not ( ). To approximately compute shifts in ( ) and ( ), existing work proposes tracking metrics like KL divergence and KS tests [27] between sliding windows in live inference data and train datasets (i.e., for train-serve skew as described in Breck et al. [16]). There are two problems with this approach: (1) it    requires the inference and training data to be kept in memory, and (2) it doesn't work well when there are many tuples--values go to zero even if shifts aren't significant enough to warrant a retrain, as discussed in Section 3.1.3 and shown in Figure 4.
To solve (1), the memory issue, we can leverage a reservoir of live tuples (as in Section 3.3), but it is impractical to keep the entire training set in memory. We can keep a materialized sample of the training set in-memory, but randomly sampling the training set might neglect important tuples, such as those from minority classes. As a solution, we can obtain a weighted random sample of the train set, where each tuple is weighted by its loss.
To solve (2), the -value issue, we can draw inspiration from adversarial validation, a Kaggle community-originated method to determine whether train and test sets are drawn from the same distribution [72]. Adversarial validation trains a binary classifier to predict whether a tuple came from either the train or test dataset. If converges to ∼ 50% AUC [73], then one can assume the datasets are similar [74]. We can extend this to track shift: we train to predict whether comes from the training sample or the reservoir sample of live data (as in Section 3.3), and log the AUC. However, adapting this method to the streaming setting is computationally challenging because we would need to train a new classifier every time we log an AUC, and computing AUC requires multiple passes through the data.
One insight is that users don't exactly care about the AUC, they only care about how the AUC changes over time, as an increasing AUC indicates that live data is diverging from training set data. As a proxy, we can log ( )'s loss over time, which can be computed in a single pass. To avoid frequently retraining ( ) from scratch, every time we get a new tuple in the reservoir sample of live data, we can sample from the reservoir with = 0.5 and the training set with = 0.5; then, we can fine-tune ( ) on with stochastic gradient descent. Here, the intuition is that decreases in loss are coupled with increases in AUC, as shown in Figure 5. As loss decreases, it becomes easier to separate the training and live data, indicating distribution shift. The onset of distribution shift as flagged by the adversarial classifier aligns with the beginning of the ML model accuracy drop (late March 2020). The features highly weighted in ( ) are also the ones most likely to be responsible for the shift, further aiding diagnosis.

Reacting to Bugs in ML Pipelines
Once bugs are isolated, they need to be fixed. Slower distribution shift can be fixed by a retrain, but silent pipeline errors require immediate engineering attention. These pipeline errors require careful analysis across components and time: the challenge is to determine which pipeline errors that, upon fixing, will have the largest positive impact on ML accuracy. Unlike traditional data repair problems [65,75,76,77,78], where the focus is on cleaning a snapshot of the data, here we want to point users to pipeline errors that, if addressed, can best improve future prediction quality. We discuss two such pipeline errors: reacting to label feedback delays and repairing broken pipeline components.

Reacting to Feedback Delays.
Knowing how the distribution of feedback delays changes over time can uncover pipeline errors and enable practitioners to quickly respond to them. Assuming the distribution of label delay is unknown and nonstationary (i.e., it may not be feasible to train a separate model to predict which predictions won't have feedback), a challenge lies in identifying groups of tuples that have similar feedback delay times to understand patterns. Most streaming clustering algorithms may not produce interpretable groups, i.e., those described with only a few predicates [79]. For debugging purposes, users may also care about how these clusters of delayed tuples change over time, or anomalies in delays; especially in the sliding window settings.
Consider the cumulative setting first. Overall, we want to pick predicate combinations that "cover" all of the tuples that have severe label delays. This is analogous to frequent itemsets [80]; recent work has extended it to work in an approximate setting, while optimizing for metrics like coverage [81]. Unlike that setting, here, we cannot materialize a sample upfront and operate on it; instead, we must operate on a stream directly, and determine what predicate combinations may have high coverage "on the fly". For this, we can draw on incremental maintenance for frequent itemsets [82], however this work focuses on updating itemsets given the addition of new tuples. In our setting some prediction tuples that are missing feedback may have their feedback arrive a bit later than expected. Therefore, we will need to both add and remove tuples and thereby update the counts of the current frequent itemsets during incremental maintenance.
These challenges are exacerbated in the sliding window setting. Here, we may be able to draw on work on streaming frequent itemsets [83,84]. For example, Chang et al. [84] use time-weighting to decay frequencies of itemsets over time unless they were seen recently. Doing this in the presence of feedback tuples appearing later in a delayed fashion is not straightforward.

Assisted
Repair of Broken Components. Besides feedback, there are two types of pipeline errors that cause performance drops: data staleness (no change) and corruption (unexpected change). A staleness example is if the pipeline to regenerate rider-related features (e.g., historical average tip) broke, forcing reads of old feature values. A corruption example is if an engineer changed geographical features to read from a better maps API, but the API returned distances in kilometers instead of miles.
A key insight for both types of pipeline errors is that they are caused by columns, because columns tend to be outputs of pipeline logic (e.g., creating features) [16]. Quantitative data cleaning techniques from the statistics and database literature typically define units of data to be cleaned as subsets of tuples, not columns [65,85,86]. Column-level changes are hard to catch-in our corruption example, a few anomalous trip distances isn't unusual, but all of them suddenly increasing is. We could leverage functional dependency (FD) discovery techniques to identify which columns most violate FDs [87,76]. However, these are hard to apply in a noisy multivariate setting and consistently tune for production pipelines, especially without prior specifications from users.
To assist repairing broken components, our research question is: what (component, column set) pairs best explain an ML performance drop? We can leverage the auto-tuned constraints from Section 3.4.2 to identify fuzzy changes in behavior at the column level. First, we determine individual column error scores for each component based on its historical behavior; then, we group columns by statistical correlations. Finally, we rank the (component, column set) error scores based on pipeline behavior by aggregating them across the dataflow graph. We discuss all three steps in turn.
To measure column error, we can adapt statistical anomaly detection techniques to track how our auto-tuned data integrity constraints behave over time. Concretely, suppose we have a column variable , a representation of Pr[ ] that determines our data integrity constraints, and seasonality window size (e.g, weekly). We derive an corruption error score ( ) for column at time from the following z-score outlier formula: Intuitively, a larger corresponds to more "anomalous" behavior (as per the definition of z-score). This approach is practical because column values do not need to be normally distributed for this method to work -the z-score is being computed across a window of aggregations of the density function Pr[ ] over time, not across the column values themselves. To determine staleness error scores, we can also apply Equation (1): the challenge is to determine the aggregations of Pr[ ] to set to. Staleness is defined by ( ) = ( −1) = ( −2) = . . . = ( − ) for the time steps that a component has not refreshed. One issue in detecting staleness is that some components are designed to produce static features; for example, many user-related, dimensional features should not change. It would be unhelpful to tell a practitioner to repair such a component. Thus, we formulate staleness as an anomaly detection problem. Intuitively, the variance of a column statistic-e.g., mean-should not change if the column is stale. So, a simple solution is to set to the standard deviation of recent means: Pr[ ] ( − ) In this example, we chose the mean because it is unlikely, in large datasets, that column values change while leaving the mean unchanged. However, we can easily replace mean with a different statistic, e.g., Pr[ < ] for some threshold , which could be a historical 25 or 75 value. We could also extend this to a multivariate setting to leverage multiple thresholds and more accurately compare distributions over time. A practical challenge is that columns are highly correlated, but we can group co-erroneous columns based on correlation. One option is to use probabilistic graphical models [76], but a simpler, non-ML idea is to sample a covariance matrix to represent "distances" between features and apply spectral clustering.
For the last step-ranking errors across all pipeline componentswe track error across the dataflow graph. Each node represents a (column group, component) pair. Two nodes share an edge if their corresponding components share an edge in the ML pipeline graph. Each node is labeled or weighted with its error score. Algorithm 1 describes how error scores can be propagated across the pipelineintuitively, intermediate nodes with small errors should deprioritize errors earlier in the pipeline, as our goal is to find nodes that cause high errors later in the pipeline. The algorithm outputs a list of sorted (column group, component) pairs that users can address, in order. We illustrate the proposed diagnosis and reaction techniques with a short case study of real ML performance drops. Some details are modified and omitted due to NDA constraints. This case study involves an organization that is currently implementing the detection, diagnosis, and reaction framework to achieve ML observability.

Input
The organization's flagship product is a mobile app with a large recommender system, which consists of approximately 20 separate ML prediction tasks (i.e., ML pipelines) that read from the same data sources. Each model takes in O(10,000) features, generated from different pipelines such as user_id_features, content_features, and engagement_features. Feedback delay is not the most important problem for this organization -the main pain points for ML engineers include, when one of the models has a performance drop, identifying relevant broken columns and quickly determining the engineering issue that caused the performance drop.
First, we describe an example of a staleness bug: the user ID features pipeline had not run successfully for weeks. This bug was detected by a drop in model accuracy. The ML pipelines simply read old user_id_features when making predictions, which performed poorly for new users that had recently signed up. The assisted repair tool analyzed soft integrity constraints on all the features and noticed high corruption scores for user_id_features, . This meant that the user_id_features output columns were becoming more null-valued, indicating more missing data. When slicing error scores by a popular user region (e.g., USA), the tool showed high staleness error scores. Thus, the organization was able to learn that the user_id_features pipeline had stopped running and make appropriate fixes.
Second, we describe an example of a corruption bug: a recent release of the app did not have audio working. As a result, users could not engage with recommended content normally, and the organization detected a drop in model accuracy. None of the hard integrity constraints (e.g., schema check, nonnegativity) were violated, and most distribution shift trackers (e.g., KS test) fired an alert for each column, which did not help with diagnosis or reaction. The assisted repair tool similarly found that most features had higher-than-usual corruption scores, but sound-related features (e.g., audio_on, num_sound_button_toggles) had the highest error scores. Thus, an on-call engineer was able to hypothesize that the audio might be broken and found a breaking PR in the changelog. Finally, sometimes model performance boosts don't improve topline business metrics. For example, the model to predict the likelihood of a comment improves, but since users treat viewing, liking, and commenting differently, the average time spent viewing a recommendation can decrease. A separate engineering challenge for ML observability systems is to provide integrations with other system components that are not directly part of the ML pipelinee.g., sales tools to log metrics like daily active users -such that users understand how much ML models impact business performance.

SYSTEM
A bolt-on ML observability system must be able to compute and store (1) history of and (2) interactions between components, requiring logging state at component runtime. Data and model integrity checks (e.g., expected number of nulls, model assertions [24]) can be programmed as constraints. Metric computation (e.g., approximate accuracy) can run as triggers.
Interface Layer. Users should be able to view real-time pipeline performance (i.e., accuracy) and query fine-grained data summaries,  Figure 6: Proposed ML observability system architecture.
traces for outputs, and other information in component logs. Furthermore, in debugging low ML performance, users need interfaces to assess label delay, inspect data shifts and constraints, and react across components, motivating techniques from visualization recommendation to highlight the most salient components [88,89,90,91]. Execution Layer. The execution layer, which wraps around a component, must be able to run trigger computation such as importanceweighting, executing and fine-tuning data quality constraints, drift detectors, time-based anomaly detection, and other methods described in Section 3. Additionally, the execution layer must identify component dependencies to track provenance for predictions. Storage Layer. As shown in Figure 6, we must store at least three types of data: pointers to inputs and outputs, ML metrics monitored across consecutive runs of the same component (Section 3.3), and logs capturing fine-grained state (provenance, data validation results, and drift detection as described in Section 3.4) every time a component is run. Additionally, the system must keep samples of training sets and live inference tuples in-memory for the execution layer to use while computing fine-grained information (e.g., K-S test results, adversarial classifier weights).

mltrace Abstractions
Our bolt-on ML observability system, mltrace, will eventually have the following functionality: (1) a library of functions that can support predefined computation before or after component runs for metric calculation or any relevant alerts, triggers, or constraints; (2) automatic logging of inputs, outputs, and metadata at the component run level; and (3) an interface for users to ask arbitrary post-hoc queries about their pipelines. Our current prototype has preliminary approaches for (2) and (3) and we are working on populating our library (1). We provide declarative, client-facing abstractions for users to specify components and the metrics and tests they would like to compute at every run of the component. The current prototype of mltrace is publicly available on Github [92] and PyPI [93]. Component. The Component abstraction represents a stage in a pipeline, similar to Kubeflow [94] notation, and houses its static metadata, such as the name (primary key), description, owner, and any string-valued tags. The Component abstraction also includes beforeRun and afterRun methods for the user to define computation, or triggers, to be run before and after the component is run.
These methods will primarily be used for testing and monitoring. mltrace will have a library of common components that practitioners can use off-the-shelf, such as a TrainingComponent that might check for train-test leakage in its beforeRun method and verify there is no overfitting in the afterRun method. Additionally, users can create their own types of components if they want to have finer-grained control.
ComponentRun. The ComponentRun (CR for short) abstraction represents dynamic metadata associated with a run or execution of a component. It includes the relevant Component name (foreign key), start timestamp of the run, end timestamp of the run, inputs, outputs, source code snapshot or git hash, extra notes, staleness indicator, and dependent CRs. Unlike other DAG-based tools, users do not need to explicitly define dependent components. mltrace sets the dependencies at runtime based on the input values; for example, if a feature generation CR produced an output features.csv and an inference CR used features.csv as an input, mltrace would add the feature generation CR as a dependency for the inference CR.
IOPointer. Inputs and outputs for a ComponentRun are represented by IOPointers. In the current prototype, the IOPointer holds only a string identifier (e.g., features.csv or model.joblib) and its serialized raw data. We plan to make historical inputs and outputs available to users in beforeRun and afterRun triggers. For mltrace to be as light as possible, we only require users to interact with the Component abstraction. CRs and IOPointers are created at component runtime via decorators on functions that represent component execution (e.g., the function that preprocesses data).

CONCLUSION
We proposed new research challenges in ML observability through a taxonomy of detecting, diagnosing, and reacting to ML bugs, helping make sense of performance with incomplete information, identify potential issues in ML pipeline components, and trace the source of errors. We discussed a high-level architecture of a bolt-on ML observability system, and introduced our prototype, mltrace, a lightweight, platform-agnostic end-to-end observability tool for ML applications. We call on the database community to contribute to the vision of ML observability, supporting users who are comfortable with their existing toolstack, while alleviating the many the data management and querying concerns that come with production ML.