Optimisation of Overparametrized Sum-Product Networks
  It seems to be a pearl of conventional wisdom that parameter learning in deep
sum-product networks is surprisingly fast compared to shallow mixture models.
This paper examines the effects of overparameterization in sum-product networks
on the speed of parameter optimisation. Using theoretical analysis and
empirical experiments, we show that deep sum-product networks exhibit an
implicit acceleration compared to their shallow counterpart. In fact,
gradient-based optimisation in deep tree-structured sum-product networks is
equal to gradient ascend with adaptive and time-varying learning rates and
additional momentum terms.
