Data-SUITE: Data-centric identification of in-distribution incongruous
  examples
  Systematic quantification of data quality is critical for consistent model
performance. Prior works have focused on out-of-distribution data. Instead, we
tackle an understudied yet equally important problem of characterizing
incongruous regions of in-distribution (ID) data, which may arise from feature
space heterogeneity. To this end, we propose a paradigm shift with Data-SUITE:
a data-centric AI framework to identify these regions, independent of a
task-specific model. Data-SUITE leverages copula modeling, representation
learning, and conformal prediction to build feature-wise confidence interval
estimators based on a set of training instances. These estimators can be used
to evaluate the congruence of test instances with respect to the training set,
to answer two practically useful questions: (1) which test instances will be
reliably predicted by a model trained with the training instances? and (2) can
we identify incongruous regions of the feature space so that data owners
understand the data's limitations or guide future data collection? We
empirically validate Data-SUITE's performance and coverage guarantees and
demonstrate on cross-site medical data, biased data, and data with concept
drift, that Data-SUITE best identifies ID regions where a downstream model may
be reliable (independent of said model). We also illustrate how these
identified regions can provide insights into datasets and highlight their
limitations.
