Heavy-tailed denoising score matching
  Score-based model research in the last few years has produced state of the
art generative models by employing Gaussian denoising score-matching (DSM).
However, the Gaussian noise assumption has several high-dimensional
limitations, motivating a more concrete route toward even higher dimension PDF
estimation in future. We outline this limitation, before extending the theory
to a broader family of noising distributions -- namely, the generalised normal
distribution. To theoretically ground this, we relax a key assumption in
(denoising) score matching theory, demonstrating that distributions which are
differentiable almost everywhere permit the same objective simplification as
Gaussians. For noise vector norm distributions, we demonstrate favourable
concentration of measure in the high-dimensional spaces prevalent in deep
learning. In the process, we uncover a skewed noise vector norm distribution
and develop an iterative noise scaling algorithm to consistently initialise the
multiple levels of noise in annealed Langevin dynamics (LD). On the practical
side, our use of heavy-tailed DSM leads to improved score estimation,
controllable sampling convergence, and more balanced unconditional generative
performance for imbalanced datasets.
