Handling the Positive-Definite Constraint in the Bayesian Learning Rule
  The Bayesian learning rule is a natural-gradient variational inference
method, which not only contains many existing learning algorithms as special
cases but also enables the design of new algorithms. Unfortunately, when
variational parameters lie in an open constraint set, the rule may not satisfy
the constraint and requires line-searches which could slow down the algorithm.
In this work, we address this issue for positive-definite constraints by
proposing an improved rule that naturally handles the constraints. Our
modification is obtained by using Riemannian gradient methods, and is valid
when the approximation attains a \emph{block-coordinate natural
parameterization} (e.g., Gaussian distributions and their mixtures). We propose
a principled way to derive Riemannian gradients and retractions from scratch.
Our method outperforms existing methods without any significant increase in
computation. Our work makes it easier to apply the rule in the presence of
positive-definite constraints in parameter spaces.
