Efficient Learning of Generative Models via Finite-Difference Score
  Matching
  Several machine learning applications involve the optimization of
higher-order derivatives (e.g., gradients of gradients) during training, which
can be expensive in respect to memory and computation even with automatic
differentiation. As a typical example in generative modeling, score matching
(SM) involves the optimization of the trace of a Hessian. To improve computing
efficiency, we rewrite the SM objective and its variants in terms of
directional derivatives, and present a generic strategy to efficiently
approximate any-order directional derivative with finite difference (FD). Our
approximation only involves function evaluations, which can be executed in
parallel, and no gradient computations. Thus, it reduces the total
computational cost while also improving numerical stability. We provide two
instantiations by reformulating variants of SM objectives into the FD forms.
Empirically, we demonstrate that our methods produce results comparable to the
gradient-based counterparts while being much more computationally efficient.
