Bilateral Denoising Diffusion Models
  Denoising diffusion probabilistic models (DDPMs) have emerged as competitive
generative models yet brought challenges to efficient sampling. In this paper,
we propose novel bilateral denoising diffusion models (BDDMs), which take
significantly fewer steps to generate high-quality samples. From a bilateral
modeling objective, BDDMs parameterize the forward and reverse processes with a
score network and a scheduling network, respectively. We show that a new lower
bound tighter than the standard evidence lower bound can be derived as a
surrogate objective for training the two networks. In particular, BDDMs are
efficient, simple-to-train, and capable of further improving any pre-trained
DDPM by optimizing the inference noise schedules. Our experiments demonstrated
that BDDMs can generate high-fidelity samples with as few as 3 sampling steps
and produce comparable or even higher quality samples than DDPMs using 1000
steps with only 16 sampling steps (a 62x speedup).
