Learning General Latent-Variable Graphical Models with Predictive Belief
  Propagation
  Learning general latent-variable probabilistic graphical models is a key
theoretical challenge in machine learning and artificial intelligence. All
previous methods, including the EM algorithm and the spectral algorithms, face
severe limitations that largely restrict their applicability and affect their
performance. In order to overcome these limitations, in this paper we introduce
a novel formulation of message-passing inference over junction trees named
predictive belief propagation, and propose a new learning and inference
algorithm for general latent-variable graphical models based on this
formulation. Our proposed algorithm reduces the hard parameter learning problem
into a sequence of supervised learning problems, and unifies the learning of
different kinds of latent graphical models into a single learning framework,
which is local-optima-free and statistically consistent. We then give a proof
of the correctness of our algorithm and show in experiments on both synthetic
and real datasets that our algorithm significantly outperforms both the EM
algorithm and the spectral algorithm while also being orders of magnitude
faster to compute.
