Holdout-Based Fidelity and Privacy Assessment of Mixed-Type Synthetic
  Data
  AI-based data synthesis has seen rapid progress over the last several years,
and is increasingly recognized for its promise to enable privacy-respecting
high-fidelity data sharing. However, adequately evaluating the quality of
generated synthetic datasets is still an open challenge. We introduce and
demonstrate a holdout-based empirical assessment framework for quantifying the
fidelity as well as the privacy risk of synthetic data solutions for mixed-type
tabular data. Measuring fidelity is based on statistical distances of
lower-dimensional marginal distributions, which provide a model-free and
easy-to-communicate empirical metric for the representativeness of a synthetic
dataset. Privacy risk is assessed by calculating the individual-level distances
to closest record with respect to the training data. By showing that the
synthetic samples are just as close to the training as to the holdout data, we
yield strong evidence that the synthesizer indeed learned to generalize
patterns and is independent of individual training records. We demonstrate the
presented framework for seven distinct synthetic data solutions across four
mixed-type datasets and compare these to more traditional statistical
disclosure techniques. The results highlight the need to systematically assess
the fidelity just as well as the privacy of these emerging class of synthetic
data generators.
