{'fraction_non_alphanumeric': 0.0439321629452872, 'fraction_numerical': 0.01227202999829555, 'mean_word_length': 4.433780967816624, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the Convergence of the ELBO to Entropy Sums

The variational lower bound (a.k.a. ELBO or free energy) is the central objective for many established as well as many novel algorithms for unsupervised learning. Learning algorithms change model parameters such that the variational lower bound increases. Learning usually proceeds until parameters have converged to values close to a stationary point of the learning dynamics. In this purely theoretical contribution, we show that (for a very large class of generative models) the variational lower bound is at all stationary points of learning equal to a sum of entropies. For standard machine learning models with one set of latents and one set observed variables, the sum consists of three entropies: (A) the (average) entropy of the variational distributions, (B) the negative entropy of the model's prior distribution, and (C) the (expected) negative entropy of the observable distributions. The obtained result applies under realistic conditions including: finite numbers of data points, at any stationary points (including saddle points) and for any family of (well behaved) variational distributions. The class of generative models for which we show the equality to entropy sums contains many well-known generative models. As concrete examples we discuss Sigmoid Belief Networks, probabilistic PCA and (Gaussian and non-Gaussian) mixture models. The prerequisites we use to show equality to entropy sums are relatively mild. Concretely, the distributions of a given generative model have to be of the exponential family (with constant base measure), and the model has to satisfy a parameterization criterion (which is usually fulfilled). Proving the equality of the ELBO to entropy sums at stationary points (under the stated conditions) is the main contribution of this work.


Introduction
We consider probabilistic generative models and their optimization based on the variational lower bound. The variational bound is also known as evidence lower bound [ELBO; e.g., Hoffman et al., 2013] or (variational) free energy [Neal and Hinton, 1998]. For a given set of data points, a learning algorithm based on generative models usually changes the parameters of the model until no or only negligible parameter changes are observed. Learning is then stopped and parameters will usually lie very close to a stationary point of the learning dynamics, i.e., close to a point where the derivatives of the ELBO objective w.r.t. all parameters are zero.
In this theoretical study, we investigate the ELBO objective at stationary points. As our main result, we show that for many generative models (including very common models), the ELBO becomes equal to a sum of entropies at stationary points. The result is very concise such that it can be stated here initially. We will consider standard generative models of the form: where z and x are latent and observed variables, respectively, and where Θ denotes the set of model parameters. The distribution p Θ ( z) will be denoted the prior distribution and p Θ ( x | z) will be referred to as noise model or observable distribution. Given N data points x (n) , the variational lower bound is given by: where q (n) Φ ( z) denote the variational distributions used for optimization. The main result of this work states that under relatively mild conditions for the generative model in (1) the ELBO decomposes into a sum of entropies at all stationary points of learning. Concretely, we will show that at all stationary points (i.e., at all points of vanishing derivatives of F(Φ, Θ)) applies: The result does notably apply for any variational distribution (no conditions on q (n) Φ ( z)), at any stationary point (including saddle points), and for finitely many data points. In principle, the result can also be used for training (because for many generative models it is sufficient that only derivatives w.r.t. a subset of the parameters Θ vanish). In this contribution, we will focus on the derivation of the result itself, however, and on the conditions that have to be satisfied for a generative model in order for the result to apply. The essential condition is that the distributions p Θ ( z) and p Θ ( x | z) are within the exponential family of distributions but we will also require some conditions for the parameterization of the models for our proof. As examples of generative models for which the result applies, we explicitly discuss sigmoid belief networks [SBNs; Neal, 1992, Saul et al., 1996, probabilistic PCA [Roweis, 1998, Tipping andBishop, 1999] and mixture models [e.g. McLachlan and Peel, 2004]. But the result applies for a broader class of models which also includes prominent models such as Factor Analysis [e.g. Everitt, 1984, Bartholomew andKnott, 1987] or probabilistic sparse coding [e.g. Olshausen and Field, 1996]. Also hierarchical/deep models and other extensions of the basic graphical model structure (1) can be treated by using the result derived here or by using the here introduced treatment as basis for further generalizations.
The decomposition into entropies can be advantageous from the theoretical and practical perspectives. The entropy decomposition (3) can, for instance, be closed-form also in cases when the original ELBO objective is not (we will briefly discuss below). The computation of variational bounds and/or likelihoods can also become easier for other reasons, e.g., for probabilistic PCA the likelihood at stationary points can be computed from the parameter values alone (we discuss in Sec. 7).
Related work. There is a long-standing interest in stationary points of variational lower bounds. For instance, likelihoods, lower bounds, and their stationary points have been studied in detail for (restricted) Boltzmann machines [e.g. Welling and Teh, 2003], elementary generative models [e.g. Frey, 1999, Tipping andBishop, 1999] as well as for general graphical models , Yedidia et al., 2001, Opper and Saad, 2001. More recently, stationary points of deep generative models have been of increasing interest. Lucas et al. [2019], for instance, linked phenomena like mode collapse in variational autoencoders (VAEs) to the stability of stationary points of the variational lower bound; and Dai et al. [2018], analyzed a series of tractable special cases of VAEs to better understand convergence points of VAE learning. Notably both, Lucas et al. and Dai et al., emphasised the close connections of VAEs to principal component analysis [PCA; Roweis, 1998, Tipping andBishop, 1999]. Theoretical results on different versions of PCA [e.g. Tipping and Bishop, 1999, Xu et al., 2010, Holtzman et al., 2020 can consequently be of direct relevance for VAEs and visa versa.
Standard VAEs and standard generative models such as probabilistic PCA (p-PCA) or probabilistic sparse coding [e.g. Olshausen and Field, 1996] all assume Gaussian distributed observables (and standard VAEs and p-PCA also Gaussian latents). In such Gaussian cases, the relation between the integrals of the variational lower bound and entropies has been of interest previously Henniges, 2012, Damm et al., 2022]. Lücke and Henniges [2012] studied the ELBO and its equality to entropy sums when the following idealized conditions apply: (A) when the limit of infinitely many data points is considered, (B) when the model distribution can exactly match the data distribution, and (C) when a global optimum is reached. Lücke and Henniges [2012] also discussed relaxations of these rather rigid assumptions (their Sec. 6) but those relaxations made use of specific properties of p-PCA and sparse coding like models optimized using expectation maximization. Damm et al. [2022] report a result derived in parallel to the one presented in this work. They focus on standard VAEs, i.e., one specific (but very popular) class of generative models. For standard VAEs, the work shows equality to entropy sums also under realistic conditions (i.e., local optima, imperfect match of model assumptions and data, finite data sets etc.). Their derivations assume the standard choice of Gaussian observables and Gaussian latents. We here address the problem of more general distributions but we will not consider deep generative models. Our work is inspired by the results of Damm et al. [2022] and its earlier version Lücke et al. [2021]. In general, results as for VAEs show that convergence to entropy sums is a property that can apply irrespective of potentially intricate non-linear relations between latents and observables. For standard Gaussian VAEs one implication of equality to entropy sums is that closed-form expressions to compute the ELBO can be available at stationary points also in cases when the original ELBO is not closed-form [see Damm et al., 2022].

The Class of Considered Generative Models
We first define the class of generative models that we consider, i.e., the models and the properties that we require to show convergence to entropy sums. We will also discuss two concrete examples and a counter-example. We keep focusing on generative models of the kind introduced above (Eqn. 1). Such models will be sufficient to communicate the concepts the proof relies on but generalizations will be discussed in Sec. 7. First, we will require a more specific notation for the parameterization of the generative model, however.

Definition A. (Generative Model)
The generative models we consider have prior distribution p Ψ with parameters Ψ, i.e., all parameters are arranged into one column vector with scalar entries. Analogously, the noise distribution p Θ is parameterized by Θ such that: Arranging prior parameters and noise model parameters into column vectors will be important for further definitions and for the derivation of the main result. For our purposes, the definition of a generative model in terms of prior and noise model distributions will also be notationally more convenient than using the model's joint probability. In general, we will adopt a somewhat less abstract mathematical notation than could be applied. For instance, we will not use Lebesgue measure notation. The motivation is that the main steps of the derivation can be communicated using notations more common in the broader Machine Learning literature. Knowledge, e.g., on Lebesgue measures, Legendre transformations and duality principles are not essential for the here made definitions and the derivations of results. We will, however, briefly discuss more abstract notations in Sec. 7. Many standard generative models are of the form as described in Definition A. Examples are those named in the introduction. For our purposes, we will require that prior and noise distribution are of the exponential family of distributions, which is the case for most standard generative models of the form of Definition A. But there are exceptions that are not (at least not directly) covered, for instance, sparse coding with mixture of Gaussians prior [Olshausen and Millman, 2000]. We will denote the class of considered generative models as exponential family generative models.

Definition B. (EF Generative Models)
Given a generative model as given by Definition A, we say the generative model is an exponential family model (EF model) if there exist exponential family distributions p ζ ( z) (for the latents) and p η ( x) (for the observables) such that the generative model can be reparameterized to take the following form: where ζ( Ψ) maps the prior parameters Ψ to the natural parameters of distribution p ζ ( z), and where η( z; Θ) maps z and the noise model parameters Θ to the natural parameters of distribution p η ( x). Furthermore, we demand that the exponential family distributions p ζ ( z) and p η ( x) have a constant base measure.
The natural parameters of the prior can in principle also depend on Θ because reparameterizations are conceivable that also involve part of the noise model parameters for the reparameterized prior. However, we will use the more intuitive but slightly more restricted definition above. For our main result to apply, it will turn out that it is not sufficient for a generative model to be an EF generative model. An additional property for the mapping from standard to natural parameters is required. Relations of the property to concepts such as minimal exponential family representations [see, e.g., Wainwright and Jordan, 2008] and linear subspaces will briefly be discussed in Sec. 7 .

Definition C. (Parameterization Criterion)
Consider an EF generative model as given in Definition B with mappings to natural parameters given by ζ( Ψ) and η( z; Θ). The dimensionalities of the natural parameter vectors Ψ and η are K and L, respectively. Let us denote by I T (Ψ) the (transposed) Jacobian matrix of the mapping ζ( Ψ), i.e., Similarly, we denote the (transposed) Jacobian matrix of the mapping η( z; Θ) by J T ( z; θ ) , i.e., Importantly, the Jacobian (9) can be constructed by using just a subset θ of the parameters Θ. We then say that the EF generative model fulfills the parameterization criterion if the following two properties hold: (A) If for any vectorial function f from the sets of parameters Φ and Ψ to the (K-dim) space of natural parameters of the prior it holds: (B) If for any vectorial function g from parameter sets Φ and Θ and from the latent space Ω z to the (L-dim) space of the natural parameters of the noise model, there exists a subset θ of Θ such that: The subset θ has to be non-empty in order for the Jacobian to be defined.
The parameters Φ will later be the variational parameters but for now play the role of arbitrary additional parameters that are neither prior parameters Ψ nor noise model parameters Θ.
In the case when z is a discrete latent variable, the integral in (11) becomes a sum over all discrete states of z. For mixture models, z is usually replaced by c where c ∈ {1, . . . , C}. Part A of the parameterization criterion is usually trivially fulfilled but is required for completeness. For instance, the usual mappings from standard parameters to natural parameters of standard exponential family distributions (Bernoulli, Gaussain, Beta, Gamma etc.) can be shown to fulfill part A of the criterion. Part B is less trivial because of the integration over z, such that explicit derivations are required for a generative model to show that part B of the criterion holds.
Below, let us first discuss some examples of generative models. The examples aim at illustrating the parameterization criterion.
Example 1. (Simple SBN) Let us consider a first example, which will be a simple form of a sigmoid belief net [SBN; Saul et al., 1996].
and where Bern(z; π) = π z (1−π) 1−z is the standard parameterization of the Bernoulli distribution (π is the probability of z being one, p(z = 1) = π). The same parameterization is used for the two Bernoulli distributions of the noise model (with the sigmoidal function setting the probability). The prior parameters Ψ just consists of the parameter π ∈ (0, 1). The noise model parameters consists of v ∈ Ê and w ∈ Ê.
The generative model (12) and (13) is an EF generative model because it can be reparameterized as follows: p ζ (z) is now the Bernoulli distribution in its exponential family form with natural parameters ζ. Likewise, p η ( x) is the Bernoulli distribution (for two observables) with natural parameters η. The standard sigmoidal function (12) is the reason for the function to natural parameters, η(z; Θ), to take on a particularly concise form. For the generative model the parameterization criterion of Definition C is fulfilled. From the parameterization as EF model, it follows for part A that the Jacobian I T The left-hand-side of (10) is, therefore, equivalent to f (Φ, π) = 0 such that the right-hand-side of (10) is trivially satisfied. For the part B of the criterion, we use all parameters Θ to construct the Jacobian (i.e., ∂ ∂ θ = so essentially g d (z; Φ, Θ) = 0 (which we will not need explicitly, though). From (16) we conclude which shows that part B of the criterion is also fulfilled.

Example 2. (Simple Factor Analysis)
Let us consider as second example a simple form of factor analysis [FA; Everitt, 1984, Bishop, 2006.

The parameter vectors are Ψ = Ψ =α and Θ =
for which we again abbreviated the variances α 2 byα and σ 2 d byσ d . Although we use standard parameters (mean and variance) for the Gaussian distributions (and not natural parameters), the parameterization of the whole model is still somewhat unusual for factor analysis because the prior also has a parameter. However, the vector w is of unit (here Euclidean) norm such that the model parameterizes the same set of distributions as a more conventionally defined FA model. The reasons for the parameterization will become clearer later on.
The model (18) and (19) can be recognized as an EF generative model (Definition A) using the following natural parameters: For part A of the parameterization criterion we get where the last implication step is due toα > 0 andα being finite. We can therefore conclude: For part B of the criterion, we choose as subset of Θ the vector θ = is then a (2 × 4)-matrix and we obtain: We now drop the parameters Φ and Θ of g for readability. From Eqn. 23 we then conclude the equalities We can now derive: where the last step follows in virtue of equations (24). The parameterization criterion is consequently fulfilled.
Example 3. (Counter-Example: Rigid SBN) As an example of a model for which it can not be shown that the parameterization criterion is fulfilled, we go back to the simple SBN model (Example 1). However, the natural parameters of the noise model now use only one parameter v ∈ Ê as follows: Part A of the parameterization criterion remains satisfied. For part B of the criterion, θ = Θ = v is the only non-empty subset of Θ. The corresponding Jacobian is the (1 × 2)-matrix J T (z; θ) = (z, z). It consequently follows from We therefore obtain: But g 2 (z; Φ, Θ) is unequal to zero for arbitrary g(z; Φ, Θ). So, in this case, we can not conclude The counter-example shows that the parameterization criterion is not necessarily fulfilled for a generative model. Other counter-examples would be simple FA (Example 2) for which one sigma is fixed (e.g.σ 2 ). Also generative models with the Poisson distribution would be a counterexample but for another reason (because the Poisson distribution does not have a constant base measure). In general, however, one can observe the parameterization criterion to be satisfied for many (maybe most) conventional generative models. We will later discuss concrete models in more detail. But first we show that equality to entropy sums applies for all EF generative models that satisfy the parameterization criterion.

Equality to Entropy Sums at Stationary Points
Let us consider the optimization of a generative model of Definition A given a set of N data points x (n) . When considering the ELBO objective (2) we, as usual, assume the variational parameters Φ to represent a set of parameters different from Ψ and Θ. The variational distributions q (n) Φ are usually defined to approximate posterior distributions of the generative model well but here we will not require any conditions on the distributions (and we will not use a vector notation for Φ). Given distributions q (n) Φ ( z), the ELBO is given by: where the first and second line represent the most common standard forms. The only unusual part of our notation is the use of two separate symbols for prior and noise model parameters ( Ψ and Θ) and a notation that denotes them as vectors. For our purposes, and for generative models as defined in Definition A, we first decompose the ELBO into three summands as follows: We can now proceed by stating and proving our main result.
Theorem 1. (Equality to Entropy Sums) If p Ψ ( z) and p Θ ( x | z) is an exponential family generative model (Definition B) that fulfills the parameterization criterion of Definition C then at all stationary points of the ELBO (28) it applies that Proof The first summand of the ELBO in Eqn. 29 is already given in the form of an (average) entropy To prove the claim of Theorem 1, we will at stationary points show equality to entropies for the second and the third summand of Eqn. 29, separately.
F 2 (Φ, Ψ) at stationary points We have assumed that the generative model is an EF generative model. Therefore we can rewrite F 2 (Φ, Ψ) of Eqn. 31 using the reparameterization of p Ψ ( z) in terms of the exponential family distribution p ζ( Ψ) ( z): As p ζ ( z) is an exponential family distribution, its negative logarithm can be written as: where A is the partition function, T the sufficient statistics and h is the base measure. We will later require the derivative of the (negative) log probability w.r.t. Ψ. The derivative is given by: and where I T is the (transposed) Jacobian matrix defined in (8).
We will show that the summand F 2 (Φ, Ψ) will at stationary points be equal to the prior entropy. Therefore, we first rewrite − log p ζ( Ψ) ( z) using the prior entropy. By using the definition of the entropy and (35) we obtain: By combining (35) and (40), the negative logarithm of the prior probability becomes: where A ′ ( ζ) is defined as in Eqn. 37, and where the last two terms have cancelled because we assumed a constant base measure h( z). Inserting the negative logarithm (42) back into (34), we therefore obtain for the second summand of the ELBO: Our aim will now be to show that the second term of (44) vanishes at stationary points of F(Φ, Ψ, Θ). For all stationary points of F(Φ, Ψ, Θ) applies that the derivatives w.r.t. all parameters vanish. We have assumed the parameters Φ, Ψ and Θ to be separate sets of parameters. F 2 (Φ, Ψ) is, therefore, the only summand of F(Φ, Ψ, Θ) which depends on Ψ. By using (34) and (36) we obtain at stationary points: whereq Φ is defined in (43). Hence, at all stationary points (w.r.t. parameters Ψ) the following concise condition holds: where f (Φ, Ψ) is the same function as introduced in (45). We can now apply the parameterization criterion of Definition C as we assumed our generative model to fulfill the criterion. The criterion is formulated for arbitrary functions f with arbitrary parameters Φ and arbitrary values for Θ. So if the criterion is satisfied, it also applies for the f as defined in (45) with Φ being the variational parameters. Using the parameterization criterion we thus conclude from (49) that at stationary points applies: Using (44) we consequently obtain at all stationary points for the summand F 2 (Φ, Ψ): F 3 (Φ, Θ) at stationary points The proof part for F 3 (Φ, Θ) will be analog to that of F 2 (Φ, Ψ) above but slightly more intricate because the noise distribution is conditional on z. We have assumed that the generative model is an EF generative model. Therefore we can rewrite F 3 (Φ, Θ) of Eqn. 32 using the reparameterization of p Θ ( x | z) in terms of the exponential family distribution p η( z; Θ) ( x): As p η ( x) is an exponential family distribution, its negative logarithm is given by: Note that we will use the same symbols for sufficient statistics T ( x), partition function A( η) and base measure h( x) as for the prior in Eqn. 35. As prior and noise model distribution are (in general) different members of the exponential family, these entities (in general) differ, of course. But as they can be distinguished from context, we avoid the introduction of further symbols. We will later require the derivative of the (negative) log probability w.r.t. θ, i.e., w.r.t. a subset of the parameters Θ. The derivative is given by: where and where J T is the (transposed) Jacobian matrix defined in (9).
We will show that the summand F 3 (Φ, Θ) will at stationary points be equal to the noise model entropy. Therefore, we first rewrite − log p η( z; Θ) ( x) using the noise model entropy. By again using the definition of the entropy and (53) we obtain: where we abbreviated B( η ) : By combining (53) and (58), the negative logarithm of the noise distribution becomes: where A ′ η( z; Θ) is defined as in Eqn. 55, and where the last two terms cancel again because we assumed a constant base measure h( x) also for the noise model distribution. Inserting the negative logarithm (60) back into (52), we therefore obtain for the third summand of the ELBO: In analogy to the derivation for F 2 (Φ, Ψ), our aim will now be to show that the second term of (62) vanishes at stationary points of F(Φ, Ψ, Θ). For all stationary points of F(Φ, Ψ, Θ) applies that the derivatives w.r.t. all parameters vanish. We have assumed the parameters Φ, Ψ and Θ to be separate sets of parameters. F 3 (Φ, Θ) is therefore the only summand of F(Φ, Ψ, Θ) which depends on Θ. At stationary points we can consequently conclude for any subset θ of Θ that ∂ ∂ θ F 3 (Φ, Θ) = 0. By using (52) and (54) we obtain at any stationary point and for any subset θ: In expression (66) we recognize the function g( z; Φ, Θ) as defined in Eqn. 63. Therefore, expression (66) means that at all stationary points and for all subsets θ of Θ the following holds: As (67) applies for all subsets θ, it also applies for the specific subset θ of Θ that exists according to the parameterization criterion (Definition C, part B). Using the parameterization criterion (part B), we can consequently conclude from (67) that at all stationary points applies: By going back to (62) and by inserting (68), we consequently obtain at all stationary points for the summand F 3 (Φ, Θ):

Sigmoid Belief Networks
Let us consider as the first concrete generative model the Sigmoid Belief Network [SBN; Neal, 1992, Hinton et al., 2006. SBNs are amongst the first and most well-known generative models with an extensive body of literature; SBNs are also often considered as prototypical examples for Bayes Nets. Here we study SBNs in with one set of latents and one set of observables in their standard form. Deep SBNs can be treated analogously (see Sec. 7). We will formulate the theorem in standard notation to be more accessible without requiring the special notation we introduced in Sec. 2, i.e., we use one set Θ for parameters of prior and noise model. The separate parameter vectors Ψ and Θ will only be used for the proof.

Definition D. (Sigmoid Belief Nets)
The generative model of an SBN is defined as follows: and where Bern(z; π) = π z (1 − π) 1−z is again the standard parameterization of the Bernoulli distribution, and the same applies for Bern(x; ·) for the observables. The set of all parameters is Θ = ( π, W, µ).

Proposition 1. (Sigmoid Belief Nets)
A Sigmoid Belief Net (Definition D) is an EF generative model which satisfies the parameterization criterion (Definition C). It therefore applies at all stationary points:

Proof
The generative model (71) and (72) has by definition two different sets of parameters for prior and noise model Θ = ( Ψ, Θ), which we arrage into column vectors as follows: where Ψ has H entries and Θ has HD + D entries (for all weights and offsets). The generative model (71) and (72) is an EF generative model because it can be reparameterized as follows: and where the distributions p ζ ( z) and p η ( x) are now Bernoulli distributions in the exponential family form. For the parameterization criterion, the Jacobian matrices are given by: where we have chosen all parameters for θ (i.e., θ = Θ) for J T . As all π h ∈ (0, 1), the diagonal entries of the (H ×H)-matrix I T where z is a sum over all bit vectors z. We can therefore conclude: which means that also part B of the parameterization criterion is satisfied. By virtue of Theorem 1 it therefore applies: and (73) is just (83) expressed using the original parameter notation of Definition D.

Gaussian Observables and Probabilistic PCA
While Theorem 1 applies for a broad range of generative models, those models with Gaussian observable distributions are very common and, therefore, represent an important case. At the same time, the properties of Gaussian distributions will further simplify the result of Theorem 1. For these reasons, we will treat Gaussian observables explicitly in the following.

Definition E. (Gaussian observables with scalar variance)
We consider generative models as in Definition A where the observable distribution p Θ ( x | z) is a Gaussian. Concretely, we consider the following class of Generative models: and where µ( z; w) is a well-behaved function (with parameters w) from the latents z to the mean of the Gaussian.
For linear functions µ the vector w can be thought of as containing all entries of a weight matrix W and potentially all offsets.

Proposition 2. (Gaussian observables with scalar variance)
Consider the generative model of Definition E. If the prior p ζ( Ψ) ( z) = p Ψ ( z) is an exponential family distribution which satisfies part A of the parameterization criterion (Definition C) then at all stationary point applies: We need to show that part B of the parameterization criterion is fulfilled. We abbreviate the variance σ 2 again byσ, and we choose θ =σ as subset of Θ (i.e., we chose a single-valued vector θ). The transposed Jacobian J T ( z; θ) of Eqn. 9 is then a row vector. Now consider the natural parameters of a scalar Gaussian η = . The form of the natural parameters means that we can rewrite η( z; Θ) = 1 σ˜ η( z; w), where the second factor does not depend onσ. The transposed Jacobian J T ( z; θ ) now becomes: It therefore follows: because − 1 σ = 0 (which shows that the part B of the parameterization criterion is fulfilled). Consequently, Theorem 1 applies for the generative model.
The entropy of the Gaussian p Θ ( x | z) is given by D 2 log 2πeσ , i.e., it does not depend on z. The last term of (33) consequently simplifies to: The maybe most prominent generative model covered by Proposition 2 is probabilistic PCA [p-PCA; Roweis, 1998, Tipping andBishop, 1999], which has a Gaussian prior and a Gaussian noise model, i.e., both distributions are in the exponential family. Given Proposition 2, we only have to show that the parameterization criterion applies for the prior distribution. This can be shown similarly as was done for the noise distribution above. However, p-PCA first has to be parameterized similarly to Example 2 as we require a parameterized prior. In the standard parameterization of p-PCA (i.e., p Θ ( z) = N ( z; 0, 1) and p Θ ( x | z) = N ( x; W z + µ, σ 2 1)) and for the case when full posteriors are used as variational distributions, we then obtain: at all stationary points. The result can be shown to be consistent with the maximum likelihood solution of Tipping and Bishop [1999] by expressing M = W T W + σ 2 1 and σ 2 in terms of eigenvalues of the data covariance matrix. Also note the relation to variational autoencoders with linear decoder as discussed by [Damm et al., 2022]. While Definition A already covers many well-known models including probabilistic PCA or probabilistic sparse coding, other models such as Factor Analysis require a further generalization.

Definition F. (Gaussian observables with diagonal covariance)
We consider generative models as in Definition A where the observable distribution p Θ ( x | z) is a Gaussian with diagonal covariance matrix: where µ( z; w) is again a well-behaved function (with parameters w) from the latents z to the set the mean of the Gaussian.

Proposition 3. (Gaussian observables with diagonal covariance)
Consider the generative model of Definition F. If the prior p ζ( Ψ) ( z) = p Ψ ( z) is an exponential family distribution which satisfies part A of the parameterization criterion (see Definition C) then at all stationary point applies:

Proof
To show that part B the parameterization criterion is fulfilled, we consider as subset θ of Θ the vector of all variances σ 2 d (with abbreviationσ d = σ 2 d ) and the following natural parameters: Given θ as above, the transposed Jacobian (9) is a (D × 2D)-matrix given as follows: where A and B are (D × D)-matrices with entries: We now consider the vector g( z; Φ, Θ) = , which we take w.l.o.g. to consist of two concatenated vectors with D entries. For readability, we now consider the dependence of g( z; Φ, Θ) on the parameters Φ and Θ as implicit, i.e. we abbreviate g( z; Φ, Θ) = g( z) (and the same for g A and g B ). With respect to the parameterization criterion (Definition C) we now obtain: It follows that for each i = 1, . . . , D applies: As (− 1 σ i ) = 0 for all i = 1, . . . , D, we can conclude that for each individual i = 1, . . . , D the integral is zero. Therefore, also the sum over all i is zero and we obtain: Hence, the second part of the parameterization criterion is fulfilled and Theorem 1 applies. The entropy of the Gaussian p Θ ( x | z) is finally given by 1 2 d log 2πeσ d , i.e., it again does not depend on z. The last term of (33) consequently simplifies to: Notably, one has to consider the cases with scalar variance and diagonal covariance separately because scalar variance can in our setting not be treated as a special case of diagonal covariance. Using Proposition 3, we could now treat factor analysis [e.g. Everitt, 1984, Bartholomew andKnott, 1987] as a model with Gaussian prior and a linear mapping to observables.

Mixture Models
As the last example we consider mixture models with exponential family distributions as noise distributions. Such models are very closely related to Bregman clustering [e.g. McLachlan andPeel, 2004, Banerjee et al., 2005], and mixtures are a tool that is for machine learning and data science of similar importance as PCA-like approaches. In this section we follow the notational convention for mixture models and replace the latent variable z by c, where c takes on integer values. More concretely, we here consider the class of mixutre models defined as follows.

Definition G. (EF Mixture)
We consider generative models with a latent variable c and observables x. The latent can take on one of C different values, i.e. c ∈ {1, . . . , C}, each with probability π c > 0 where C c=1 π c = 1. The observable x is distributed according to a distribution p( x | Θ c ) with parameters depending on the value of the latent c. More formally, the generative model is given by: where Cat(c ; π) is the categorical distribution with standard parameterization. The set of all model parameters Θ is given by Θ = π, Θ 1 , Θ 2 , . . . , Θ C . We refer to the model (102) and (103) as an exponential family mixture (EF mixture) if the observable distribution p( x | Θ c ) can be written as an exponential family distribution p η( Θc) ( x) where η(·) maps an L-dim vector Θ c of standard parameters to an L-dim vector of natural parameters. the proof of the proposition on EF mixtures below, we will again us vectors for the parameters and will distinguish between prior parameters Ψ = (π 1 , . . . , π C−1 ) T and noise model parameters Θ = ( Θ 1 , . . . , Θ C ) T . Note that we will only consider (C − 1) prior parameters as π C can be taken to be determined by π 1 to π C−1 . We remark the technical but important difference between the mapping η(·) of Definition G and the mapping η(c; Θ) which takes the role of η( z; Θ) as used for Definitions B and C. η(·) is the usual mapping from standard parameters of an exponential family distribution to its natural parameters (see the just discussed examples Gaussian and gamma distributions). η(c; Θ) also maps standard parameters to natural parameters of the observable distributions (hence the same symbol) but, in contrast to η(·), the mapping η(c; Θ) has the vector of all noise model parameters as input. The Jacobians of the mappings are consequently different. The Jacobian of η(c; Θ) (computed w.r.t. all parameters Θ) is a (CL × L)-matrix, while the Jacobian of η(·) is a square (L × L)-matrix. This smaller (L × L)-Jacobian is important as condition for the following proposition. The large Jacobian of η(c; Θ) is important for the proposition's proof.

Proposition 4. (Mixture Models)
Consider an EF mixture model of Definition G where η(·) maps the standard parameters of the noise model to the distribution's natural parameters. If the Jacobian of η(·) is everywhere invertible, then the model is an EF generative model (Definition B) and the parameterization criterion (Definition C) is fulfilled. It then applies at all stationary points: By summing over c = 1, . . . , C − 1, we can conclude from (108) that: where we have used C−1 c=1 π c = π C . So we now know that the left-hand-side of (10) implies C−1 i=1 f i (Φ, Ψ) = 0 but we also still know that the left-hand-side implies (108). Using both we obtain that f c (Φ, Ψ) = 0 ∀c, i.e., the left-hand-side of criterion (10) implies for the categorical distribution that f (Φ, Ψ) = 0. The right-hand-side of (10), the term ζ T ( Ψ) f (Φ, Ψ), is consequently trivially zero as f (Φ, Ψ) is zero. Part A of the parameterization criterion is therefore fulfilled.
To construct the Jacobian J T (c; θ) for part B of the criterion, we use all parameters Θ, i.e. θ = Θ. The Jacobian is then the (CL × L)-matrix given by: From the left-hand-side of (11) for part B of the criterion we can, therefore, conclude: where for the last step we renamed c ′ to c. (110) is the transposed Jacobian of η(·). If that Jacobian is everywhere invertible (as we demanded), then (110) implies that g(c; Φ, Θ) is zero for all c.
We can therefore conclude that the term C c=1 η T (c; Θ) g(c; Φ, Θ), i.e. the right-hand-side of (11), is zero. Therefore, part B of the parameterization criterion is fulfilled.
As the model of Def. G with the assumption of Prop. 4 is an EF generative model that fulfills the parameterization criterion, (104) follows from Theorem 1.
The assumptions made in Proposition 4 are sufficient for (104) to apply. The assumptions are also sufficiently broad to apply for most mixtures but they can presumably weakened further. For a given mixture model, Proposition 4 is relatively easy to apply because the mapping from standard parameterization to natural parameters is usually known.
As example let us use a mixture of Gamma distribtuions. A mixture component c can then be written as exponential family distribution with L = 2 parameters given by: with Θ c = α c β c and η( Θ c ) = α c − 1 −β c . The base measure is constant, h(x) = 1, and A( Θ c ) = log Γ(α c ) − α c log β c is the partition function. The Jacobian of η(·) is the (2 × 2)matrix 1 0 0 −1 which is everywhere invertible. Similarly, mixture models based on other concrete observable distributions can be treated. Variational optimization can be used to accelerate training of mixture models [Neal andHinton, 1998, Lücke andForster, 2019] and (104) applies for the corresponding variational distributions. However, it is more common for mixture models to be trained without approximation usually by directly using expectation maximization [EM; Dempster et al., 1977]. In our setting, the exact (i.e. full) posterior can be considered as a special case of a variational distribution. Furthermore, using full posteriors means that the ELBO becomes equal to the likelihood L(Θ) at all stationary points. For mixture models optimized using EM, we can therefore conclude that at all stationary points of learning applies: . (112) Maybe as a small final consistency result, suppose that a mixture of scalar Gaussians converges such that one mixing proportion π c converges to one (and the others to zero). Then prior entropy and posterior entropy converge to zero. Furthermore, as the Gaussian entropy does not depend on the mean, we can conclude from (112) that it applies at the stationary point that where we assumed w.l.o.g. that it is the first cluster (c = 1) for which the mixing proportion becomes equal one (π 1 → 1). More concretely: L(Θ) = − 1 2 log 2πeσ 2 1 at stationary points. The same result can be obtained 'by hand' using the log-likelihood of a single Gaussian (with variance σ 2 1 ) and inserting the maximum likelihood solution for σ 2 1 . Equation (112) and Proposition 4 show that such convergences to entropies can be understood as (very) special cases of the much more general result of Theorem 1.

Discussion and Outlook
By considering the used definitions and the proof of Theorem 1, a number of aspects can be noted.
Theorem 1 and other standard generative models. Based on Theorem 1 further examples of standard generative models can be considered and treated similarly as shown for the three here considered types of generative models (Sections 4 to 6). For instance, other model with Gaussian observables can be considered with probabilistic sparse coding [e.g. Olshausen and Field, 1996] or factor analysis [e.g. Everitt, 1984, Bartholomew andKnott, 1987] being just two of many other examples. For each model, we have to show that all conditions required for Theorem 1 are fulfilled. The main challenge is usually the parameterization criterion but also the generative model first has to take on an appropriate form (such as being equipped with a parameterized prior). Further models that can be treated are concrete mixture models [McLachlan andPeel, 2004, Banerjee et al., 2005], i.e., mixtures with concrete examples for observable distributions such as Bernoulli, exponential or beta-distributions. In the case of mixtures, no further analytical steps are required at least for models that fulfill the requirements of Proposition G. Hence, concrete expressions for the ELBO (or for the log-likelihood) can be stated with only minor effort as entropies of the most common exponential family distributions are known, or can relatively easily be obtained [e.g. Nielsen and Nock, 2010]. More efforts may be required for less conventional mixtures or heterogeneous mixtures but applications of Theorem 1 will presumably be similar to the application to standard mixtures as treated here.
Non-constant base measures. By considering the proof of the main result (Theorem 1) the assumption of a constant base measure is not essential for the proof but the result would have to be modified. For instance, we could introduced an amended entropyH which includes base measure terms. However, we could also resort to a more abstract notation using Lebesgue integrals. By using the Lebesgue measure, as it is common for exponential families, the base measure becomes part of the Lebesgue measure and proofs can essentially be maintain as here described. For our purposes, we considered it more suitable to keep the more basic notation in order to connect to the standard notational conventions as used in most of the Machine Learning literature.
Deep generative models and earlier work. We have here considered standard (non-deep) generative models as natural first examples. But the treatment can, of course, in principle be further extended, e.g., towards deep generative models that are of considerable interest [see Bond-Taylor et al., 2022, for a review]. One canonical first example would be to consider variational autoencoders [Kingma andWelling, 2014, Rezende et al., 2014], and a second example would be represented by deep forms of SBNs [Neal, 1992, Hinton et al., 2006. Both types of generative models would require considerable effort to proof convergence to entropy sums. The here presented proofs show that already for non-deep standard generative models considerable effort is required which can be taken as evidence for considerably more extensive efforts in case of deep models. Especially for VAEs (which can be based on sophisticated and large neural networks) proving convergence to entropy sums can be a considerable challenge.
As detailed in the introduction, our investigation of entropy convergence for models based on exponential family distributions (Theorem 1) has its roots in two earlier papers Henniges, 2012, Damm et al., 2022]. Both of those contributions focus on models based on Gaussian distributions while the here presented results show convergence to entropy sums more generally: Theorem 1 shows that convergence to entropy sums is a very common feature of generative models, which does not require properties specific to the Gaussian distribution. Given the general result of Theorem 1, we can treat the Gaussian case as a specialization, of course. For Gaussian VAEs (especially VAEs with DNNs also for decoder variances) convergence to entropy sums can in this way be proven by showing that the parameterization criterion (Definition C) is fulfilled for such VAEs [see Damm et al., 2022]. In contrast, an earlier treatment of VAEs [Lücke et al., 2021] followed an approach in which properties of DNNs and Gaussians were much more entangled for proving entropy convergence of VAEs 1 .
In comparison to VAEs, deep SBNs represent another type of a deep generative model. The layers of deep SBNs consist of stochastic units, while the layers of the VAE generative model consists of the deterministic units of standard DNNs. Hence, for deep SBNs more than one set of latent variables has to be considered, which would require an extension of the models treated here. By considering the proof of the main result (Theorem 1), one observes that there is no need to restrict oneself to just one set of stochastic/latent variables, however. Indeed, it is relatively straight-forward to consider models with several sets of latents. Deep SBNs (but also other similar types of deep generative model) can therefore be expected to converge to sums of entropies with more than three entropy terms.
Parameterization criterion. The parameterization criterion requires further investigations. It is a sufficient condition, and the condition can directly be proven for important models and important classes of models. So in that sense, the criterion fulfills the purposes of this paper. However, the structure of the criterion suggests that it can be derived from a more concise, abstract property of a given generative model presumably closely related to reducability, minimal representations and linear subspaces. Also more constraint criteria such as facorizations may be well suited in order to show for some generative models that the parameterization criterion is fulfilled.
Conclusion. Future work will further investigate the theoretical implications of the result, and the classes of generative models for which results like Theorem 1 can be derived. Furthermore, different applications of the theorem will be investigated that leverage the convergence of the ELBO to entropy sums. The previously mentioned study [Damm et al., 2022] takes first steps in this direction by analyzing phenomena like mode collapse or by using entropy sums for fast model selection. Further steps currently considered do make use of the fact that equality to entropy sums applies also when only subsets of the model parameters are at (or close to) stationary points. Following such considerations, more global properties of the optimization landscape can be investigated and learning itself could be improved.