Understanding Minimum Probability Flow for RBMs Under Various Kinds of
  Dynamics
  Energy-based models are popular in machine learning due to the elegance of
their formulation and their relationship to statistical physics. Among these,
the Restricted Boltzmann Machine (RBM), and its staple training algorithm
contrastive divergence (CD), have been the prototype for some recent
advancements in the unsupervised training of deep neural networks. However, CD
has limited theoretical motivation, and can in some cases produce undesirable
behavior. Here, we investigate the performance of Minimum Probability Flow
(MPF) learning for training RBMs. Unlike CD, with its focus on approximating an
intractable partition function via Gibbs sampling, MPF proposes a tractable,
consistent, objective function defined in terms of a Taylor expansion of the KL
divergence with respect to sampling dynamics. Here we propose a more general
form for the sampling dynamics in MPF, and explore the consequences of
different choices for these dynamics for training RBMs. Experimental results
show MPF outperforming CD for various RBM configurations.
