I'm trying to expand my experience with restricted Boltzmann machines to a more general class of graphical models and currently learning about belief propagation using message passing algorithms. One thing I can't understand is whether BP may be used to learn parameters of a probabilistic network or is just used to, well, propagate probability given single instance of data. Let me explain it by example.

For demonstration purposes I will use RBMs that I know well, although same principles should be applicable to most graphical models. RBM is a kind of bipartite graph with 2 kinds of nodes - evidence and hidden (both - Bernoulli random variables) - and parametrized by matrix of weights connecting these groups of nodes and 2 corresponding groups of bias terms:

enter image description here

The most common algorithm to learn RBM, i.e. estimate weights and biases, looks like this:

  1. Take batch of eviddata $\textbf{E}$.
  2. Make several loops of Gibbs sampling between evidence units $E$ and hidden units $H$ to obtain equilibrium distribution.
  3. Adjust weights and biases to minimize Kullback-Leibler divergence.
  4. Repeat.

After a number of iterations RBM is considered trained and we can use it to calculate $P(H|E)$, i.e. perform inference.

Now what about belief propagation?

I presume RBM may be transformed into a factor graph like the following:

enter image description here

And here's how I understand loopy belief propagation for this graph:

  1. Take a single vector of evidence data $E_i$ and pass it to factor nodes $\phi_i$.
  2. Pass beliefs from factor nodes $\phi_i$ to hidden $H_j$ and vice versa.
  3. Repeat (2) until convergence.

It sounds like inference of $P(H|E)$, but not like learning of network parameters.

I'm pretty much sure that my understanding of at least part of this process is incorrect, but being a newbie in LBF I need some extra help to figure out exactly what.

Here's a couple of concrete questions that I have most doubts about:

  • is it correct that in message passing we always take only a single vector of evidence data (i.e. not a batch)?
  • do we have network parameters analogous to weights and biases in standard RBM?
  • is the energy or any other cost function that we try to minimize/maximize?

Your Answer


Browse other questions tagged or ask your own question.