Estimating High Order Gradients of the Data Distribution by Denoising
  The first order derivative of a data density can be estimated efficiently by
denoising score matching, and has become an important component in many
applications, such as image generation and audio synthesis. Higher order
derivatives provide additional local information about the data distribution
and enable new applications. Although they can be estimated via automatic
differentiation of a learned density model, this can amplify estimation errors
and is expensive in high dimensional settings. To overcome these limitations,
we propose a method to directly estimate high order derivatives (scores) of a
data density from samples. We first show that denoising score matching can be
interpreted as a particular case of Tweedie's formula. By leveraging Tweedie's
formula on higher order moments, we generalize denoising score matching to
estimate higher order derivatives. We demonstrate empirically that models
trained with the proposed method can approximate second order derivatives more
efficiently and accurately than via automatic differentiation. We show that our
models can be used to quantify uncertainty in denoising and to improve the
mixing speed of Langevin dynamics via Ozaki discretization for sampling
synthetic data and natural images.
