HyperSPNs: Compact and Expressive Probabilistic Circuits
  Probabilistic circuits (PCs) are a family of generative models which allows
for the computation of exact likelihoods and marginals of its probability
distributions. PCs are both expressive and tractable, and serve as popular
choices for discrete density estimation tasks. However, large PCs are
susceptible to overfitting, and only a few regularization strategies (e.g.,
dropout, weight-decay) have been explored. We propose HyperSPNs: a new paradigm
of generating the mixture weights of large PCs using a small-scale neural
network. Our framework can be viewed as a soft weight-sharing strategy, which
combines the greater expressiveness of large models with the better
generalization and memory-footprint properties of small models. We show the
merits of our regularization strategy on two state-of-the-art PC families
introduced in recent literature -- RAT-SPNs and EiNETs -- and demonstrate
generalization improvements in both models on a suite of density estimation
benchmarks in both discrete and continuous domains.
