Sliced Score Matching: A Scalable Approach to Density and Score
  Estimation
  Score matching is a popular method for estimating unnormalized statistical
models. However, it has been so far limited to simple, shallow models or
low-dimensional data, due to the difficulty of computing the Hessian of
log-density functions. We show this difficulty can be mitigated by projecting
the scores onto random vectors before comparing them. This objective, called
sliced score matching, only involves Hessian-vector products, which can be
easily implemented using reverse-mode automatic differentiation. Therefore,
sliced score matching is amenable to more complex models and higher dimensional
data compared to score matching. Theoretically, we prove the consistency and
asymptotic normality of sliced score matching estimators. Moreover, we
demonstrate that sliced score matching can be used to learn deep score
estimators for implicit distributions. In our experiments, we show sliced score
matching can learn deep energy-based models effectively, and can produce
accurate score estimates for applications such as variational inference with
implicit distributions and training Wasserstein Auto-Encoders.
