General Probabilistic Surface Optimization and Log Density Estimation
  In this paper we contribute a novel algorithm family, which generalizes many
unsupervised techniques including unnormalized and energy models, and allows us
to infer different statistical modalities (e.g. data likelihood and ratio
between densities) from data samples. The proposed unsupervised technique,
named Probabilistic Surface Optimization (PSO), views a model as a flexible
surface which can be pushed according to loss-specific virtual stochastic
forces, where a dynamical equilibrium is achieved when the pointwise forces on
the surface become equal. Concretely, the surface is pushed up and down at
points sampled from two different distributions. The averaged up and down
forces become functions of these two distribution densities and of force
magnitudes defined by the loss of a particular PSO instance. Upon convergence,
the force equilibrium imposes an optimized model to be equal to various
statistical functions depending on the used magnitude functions. Furthermore,
this dynamical-statistical equilibrium is extremely intuitive and useful,
providing many implications and possible usages in probabilistic inference. We
connect PSO to numerous existing statistical works which are also PSO
instances, and derive new PSO-based inference methods as demonstration of PSO
exceptional usability. Likewise, based on the insights coming from the
virtual-force perspective we analyze PSO stability and propose new ways to
improve it. Finally, we present new instances of PSO, termed PSO-LDE, for data
log-density estimation and also provide a new NN block-diagonal architecture
for increased surface flexibility, which significantly improves estimation
accuracy. Both PSO-LDE and the new architecture are combined together as a new
density estimation technique. In our experiments we demonstrate this technique
to be superior over state-of-the-art baselines in density estimation task for
multimodal 20D data.
