SAME but Different: Fast and High-Quality Gibbs Parameter Estimation
  Gibbs sampling is a workhorse for Bayesian inference but has several
limitations when used for parameter estimation, and is often much slower than
non-sampling inference methods. SAME (State Augmentation for Marginal
Estimation) \cite{Doucet99,Doucet02} is an approach to MAP parameter estimation
which gives improved parameter estimates over direct Gibbs sampling. SAME can
be viewed as cooling the posterior parameter distribution and allows annealed
search for the MAP parameters, often yielding very high quality (lower loss)
estimates. But it does so at the expense of additional samples per iteration
and generally slower performance. On the other hand, SAME dramatically
increases the parallelism in the sampling schedule, and is an excellent match
for modern (SIMD) hardware. In this paper we explore the application of SAME to
graphical model inference on modern hardware. We show that combining SAME with
factored sample representation (or approximation) gives throughput competitive
with the fastest symbolic methods, but with potentially better quality. We
describe experiments on Latent Dirichlet Allocation, achieving speeds similar
to the fastest reported methods (online Variational Bayes) and lower
cross-validated loss than other LDA implementations. The method is simple to
implement and should be applicable to many other models.
