Controllable and Compositional Generation with Latent-Space Energy-Based
  Models
  Controllable generation is one of the key requirements for successful
adoption of deep generative models in real-world applications, but it still
remains as a great challenge. In particular, the compositional ability to
generate novel concept combinations is out of reach for most current models. In
this work, we use energy-based models (EBMs) to handle compositional generation
over a set of attributes. To make them scalable to high-resolution image
generation, we introduce an EBM in the latent space of a pre-trained generative
model such as StyleGAN. We propose a novel EBM formulation representing the
joint distribution of data and attributes together, and we show how sampling
from it is formulated as solving an ordinary differential equation (ODE). Given
a pre-trained generator, all we need for controllable generation is to train an
attribute classifier. Sampling with ODEs is done efficiently in the latent
space and is robust to hyperparameters. Thus, our method is simple, fast to
train, and efficient to sample. Experimental results show that our method
outperforms the state-of-the-art in both conditional sampling and sequential
editing. In compositional generation, our method excels at zero-shot generation
of unseen attribute combinations. Also, by composing energy functions with
logical operators, this work is the first to achieve such compositionality in
generating photo-realistic images of resolution 1024x1024. Code is available at
https://github.com/NVlabs/LACE.
