Cold Case: The Lost MNIST Digits
  Although the popular MNIST dataset [LeCun et al., 1994] is derived from the
NIST database [Grother and Hanaoka, 1995], the precise processing steps for
this derivation have been lost to time. We propose a reconstruction that is
accurate enough to serve as a replacement for the MNIST dataset, with
insignificant changes in accuracy. We trace each MNIST digit to its NIST source
and its rich metadata such as writer identifier, partition identifier, etc. We
also reconstruct the complete MNIST test set with 60,000 samples instead of the
usual 10,000. Since the balance 50,000 were never distributed, they enable us
to investigate the impact of twenty-five years of MNIST experiments on the
reported testing performances. Our results unambiguously confirm the trends
observed by Recht et al. [2018, 2019]: although the misclassification rates are
slightly off, classifier ordering and model selection remain broadly reliable.
We attribute this phenomenon to the pairing benefits of comparing classifiers
on the same digits.
