Adversarial Fisher Vectors for Unsupervised Representation Learning
  We examine Generative Adversarial Networks (GANs) through the lens of deep
Energy Based Models (EBMs), with the goal of exploiting the density model that
follows from this formulation. In contrast to a traditional view where the
discriminator learns a constant function when reaching convergence, here we
show that it can provide useful information for downstream tasks, e.g., feature
extraction for classification. To be concrete, in the EBM formulation, the
discriminator learns an unnormalized density function (i.e., the negative
energy term) that characterizes the data manifold. We propose to evaluate both
the generator and the discriminator by deriving corresponding Fisher Score and
Fisher Information from the EBM. We show that by assuming that the generated
examples form an estimate of the learned density, both the Fisher Information
and the normalized Fisher Vectors are easy to compute. We also show that we are
able to derive a distance metric between examples and between sets of examples.
We conduct experiments showing that the GAN-induced Fisher Vectors demonstrate
competitive performance as unsupervised feature extractors for classification
and perceptual similarity tasks. Code is available at
\url{https://github.com/apple/ml-afv}.
