SGB: Stochastic Gradient Bound Method for Optimizing Partition Functions
  This paper addresses the problem of optimizing partition functions in a
stochastic learning setting. We propose a stochastic variant of the bound
majorization algorithm that relies on upper-bounding the partition function
with a quadratic surrogate. The update of the proposed method, that we refer to
as Stochastic Partition Function Bound (SPFB), resembles scaled stochastic
gradient descent where the scaling factor relies on a second order term that is
however different from the Hessian. Similarly to quasi-Newton schemes, this
term is constructed using the stochastic approximation of the value of the
function and its gradient. We prove sub-linear convergence rate of the proposed
method and show the construction of its low-rank variant (LSPFB). Experiments
on logistic regression demonstrate that the proposed schemes significantly
outperform SGD. We also discuss how to use quadratic partition function bound
for efficient training of deep learning models and in non-convex optimization.
