Multivariate Density Estimation with Deep Neural Mixture Models
  Albeit worryingly underrated in the recent literature on machine learning in
general (and, on deep learning in particular), multivariate density estimation
is a fundamental task in many applications, at least implicitly, and still an
open issue. With a few exceptions, deep neural networks (DNNs) have seldom been
applied to density estimation, mostly due to the unsupervised nature of the
estimation task, and (especially) due to the need for constrained training
algorithms that ended up realizing proper probabilistic models that satisfy
Kolmogorov's axioms. Moreover, in spite of the well-known improvement in terms
of modeling capabilities yielded by mixture models over plain single-density
statistical estimators, no proper mixtures of multivariate DNN-based component
densities have been investigated so far. The paper fills this gap by extending
our previous work on Neural Mixture Densities (NMMs) to multivariate DNN
mixtures. A maximum-likelihood (ML) algorithm for estimating Deep NMMs (DNMMs)
is handed out, which satisfies numerically a combination of hard and soft
constraints aimed at ensuring satisfaction of Kolmogorov's axioms. The class of
probability density functions that can be modeled to any degree of precision
via DNMMs is formally defined. A procedure for the automatic selection of the
DNMM architecture, as well as of the hyperparameters for its ML training
algorithm, is presented (exploiting the probabilistic nature of the DNMM).
Experimental results on univariate and multivariate data are reported on,
corroborating the effectiveness of the approach and its superiority to the most
popular statistical estimation techniques.
