Estimating Model Performance on External Samples from Their Limited
  Statistical Characteristics
  Methods that address data shifts usually assume full access to multiple
datasets. In the healthcare domain, however, privacy-preserving regulations as
well as commercial interests limit data availability and, as a result,
researchers can typically study only a small number of datasets. In contrast,
limited statistical characteristics of specific patient samples are much easier
to share and may be available from previously published literature or focused
collaborative efforts.
  Here, we propose a method that estimates model performance in external
samples from their limited statistical characteristics. We search for weights
that induce internal statistics that are similar to the external ones; and that
are closest to uniform. We then use model performance on the weighted internal
sample as an estimation for the external counterpart.
  We evaluate the proposed algorithm on simulated data as well as electronic
medical record data for two risk models, predicting complications in ulcerative
colitis patients and stroke in women diagnosed with atrial fibrillation. In the
vast majority of cases, the estimated external performance is much closer to
the actual one than the internal performance. Our proposed method may be an
important building block in training robust models and detecting potential
model failures in external environments.
