A Forest Mixture Bound for Block-Free Parallel Inference
  Coordinate ascent variational inference is an important algorithm for
inference in probabilistic models, but it is slow because it updates only a
single variable at a time. Block coordinate methods perform inference faster by
updating blocks of variables in parallel. However, the speed and stability of
these algorithms depends on how the variables are partitioned into blocks. In
this paper, we give a stable parallel algorithm for inference in deep
exponential families that doesn't require the variables to be partitioned into
blocks. We achieve this by lower bounding the ELBO by a new objective we call
the forest mixture bound (FM bound) that separates the inference problem for
variables within a hidden layer. We apply this to the simple case when all
random variables are Gaussian and show empirically that the algorithm converges
faster for models that are inherently more forest-like.
