HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative
  Models
  Generative models often use human evaluations to measure the perceived
quality of their outputs. Automated metrics are noisy indirect proxies, because
they rely on heuristics or pretrained embeddings. However, up until now, direct
human evaluation strategies have been ad-hoc, neither standardized nor
validated. Our work establishes a gold standard human benchmark for generative
realism. We construct Human eYe Perceptual Evaluation (HYPE) a human benchmark
that is (1) grounded in psychophysics research in perception, (2) reliable
across different sets of randomly sampled outputs from a model, (3) able to
produce separable model performances, and (4) efficient in cost and time. We
introduce two variants: one that measures visual perception under adaptive time
constraints to determine the threshold at which a model's outputs appear real
(e.g. 250ms), and the other a less expensive variant that measures human error
rate on fake and real images sans time constraints. We test HYPE across six
state-of-the-art generative adversarial networks and two sampling techniques on
conditional and unconditional image generation using four datasets: CelebA,
FFHQ, CIFAR-10, and ImageNet. We find that HYPE can track model improvements
across training epochs, and we confirm via bootstrap sampling that HYPE
rankings are consistent and replicable.
