On the Evaluation of Conditional GANs
  Conditional Generative Adversarial Networks (cGANs) are finding increasingly
widespread use in many application domains. Despite outstanding progress,
quantitative evaluation of such models often involves multiple distinct metrics
to assess different desirable properties, such as image quality, conditional
consistency, and intra-conditioning diversity. In this setting, model
benchmarking becomes a challenge, as each metric may indicate a different
"best" model. In this paper, we propose the Frechet Joint Distance (FJD), which
is defined as the Frechet distance between joint distributions of images and
conditioning, allowing it to implicitly capture the aforementioned properties
in a single metric. We conduct proof-of-concept experiments on a controllable
synthetic dataset, which consistently highlight the benefits of FJD when
compared to currently established metrics. Moreover, we use the newly
introduced metric to compare existing cGAN-based models for a variety of
conditioning modalities (e.g. class labels, object masks, bounding boxes,
images, and text captions). We show that FJD can be used as a promising single
metric for cGAN benchmarking and model selection. Code can be found at
https://github.com/facebookresearch/fjd.
