Distribution Density, Tails, and Outliers in Machine Learning: Metrics
  and Applications
  We develop techniques to quantify the degree to which a given (training or
testing) example is an outlier in the underlying distribution. We evaluate five
methods to score examples in a dataset by how well-represented the examples
are, for different plausible definitions of "well-represented", and apply these
to four common datasets: MNIST, Fashion-MNIST, CIFAR-10, and ImageNet. Despite
being independent approaches, we find all five are highly correlated,
suggesting that the notion of being well-represented can be quantified. Among
other uses, we find these methods can be combined to identify (a) prototypical
examples (that match human expectations); (b) memorized training examples; and,
(c) uncommon submodes of the dataset. Further, we show how we can utilize our
metrics to determine an improved ordering for curriculum learning, and impact
adversarial robustness. We release all metric values on training and test sets
we studied.
