Maximal Causes for Exponential Family Observables
  Latent variable models represent observed variables as parameterized
functions of a set of latent variables. Examples are factor analysis or
probabilistic sparse coding which assume weighted linear summations to
determine the mean of Gaussian distribution for the observables. However, in
many cases observables do not follow a normal distribution, and a linear
summation of latents is often at odds with non-Gaussian observables (e.g.,
means of the Bernoulli distribution have to lie in the unit interval).
Furthermore, the assumption of a linear summation model may (for many types of
data) not be closely aligned with the true data generation process even for
Gaussian observables. Alternative superposition models (i.e., alternative links
between latents and observables) have therefore been investigated repeatedly.
Here we show that using the maximization instead of summation to link latents
to observables allows for the derivation of a very general and concise set of
parameter update equations. Concretely, we derive a set of update equations
that has the same functional form for all distributions of the exponential
family. Our results consequently provide directly applicable learning equations
for commonly as well as for unusually distributed data. We numerically verify
our analytical results assuming standard Gaussian, Gamma, Poisson, Bernoulli
and Exponential distributions. We point to some potential applications by
providing different experiments on the learning of variance structure, noise
type estimation, and denoising.
