Tractable Regularization of Probabilistic Circuits
  Probabilistic Circuits (PCs) are a promising avenue for probabilistic
modeling. They combine advantages of probabilistic graphical models (PGMs) with
those of neural networks (NNs). Crucially, however, they are tractable
probabilistic models, supporting efficient and exact computation of many
probabilistic inference queries, such as marginals and MAP. Further, since PCs
are structured computation graphs, they can take advantage of
deep-learning-style parameter updates, which greatly improves their
scalability. However, this innovation also makes PCs prone to overfitting,
which has been observed in many standard benchmarks. Despite the existence of
abundant regularization techniques for both PGMs and NNs, they are not
effective enough when applied to PCs. Instead, we re-think regularization for
PCs and propose two intuitive techniques, data softening and entropy
regularization, that both take advantage of PCs' tractability and still have an
efficient implementation as a computation graph. Specifically, data softening
provides a principled way to add uncertainty in datasets in closed form, which
implicitly regularizes PC parameters. To learn parameters from a softened
dataset, PCs only need linear time by virtue of their tractability. In entropy
regularization, the exact entropy of the distribution encoded by a PC can be
regularized directly, which is again infeasible for most other density
estimation models. We show that both methods consistently improve the
generalization performance of a wide variety of PCs. Moreover, when paired with
a simple PC structure, we achieved state-of-the-art results on 10 out of 20
standard discrete density estimation benchmarks.
