{'fraction_non_alphanumeric': 0.054316151531075337, 'fraction_numerical': 0.017126068886447236, 'mean_word_length': 3.952330257866099, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Consistency of the MLE under mixture models

The large-sample properties of likelihood-based statistical inference under mixture models have received much attention from statisticians. Although the consistency of the nonparametric MLE is regarded as a standard conclusion, many researchers ignore the precise conditions required on the mixture model. An incorrect claim of consistency can lead to false conclusions even if the mixture model under investigation seems well behaved. Under a finite normal mixture model, for instance, the consistency of the plain MLE is often erroneously assumed in spite of recent research breakthroughs. This paper streamlines the consistency results for the nonparametric MLE in general, and in particular for the penalized MLE under finite normal mixture models.


INTRODUCTION
A family of distributions, parametric or nonparametric, is regarded as a probability or statistical model. A parametric model can hence be a family of density functions in the form {f (x; θ) : θ ∈ Θ} where Θ is a subset of R d for some positive integer d. The measure with respect to which these densities are defined is µ, but this will be de-emphasized. In applications, statisticians must select one or a set of these distributions to characterize the uncertainty displayed in a random sample from a population.
A specific distribution family or model is often tentatively chosen in accordance with the scientific background of the application. For instance, the Poisson distribution/model is a textbook choice for the number of annual car accidents of a policy holder, but it may not be ideal because risk levels vary. It can therefore be helpful to divide the population into several subpopulations, each modeled by its own Poisson distribution. Thus, a finite mixture of Poisson distributions is a better choice for the pooled accident data. This leads to a generic finite mixture model in the form In this formulation, f (x; G) is a finite convex combination of the component density functions f (x; θ). Each f (x; θ j ) is the density for a subpopulation. The recipe for the mixture is summarized by G, specified by its cumulative distribution where I(·) is an indicator function. The model f (x; G) in the above definition has order m, even if some θ j are equal or some α j = 0. A finite mixture model of order m is hence also a degenerate order m + 1 mixture. For any c.d.f. G on Θ, the following integral is a well-defined density function. When G has only a finite number of support points, as in (1.2), f (x; G) reduces to the finite mixture model (1.1). When the form of G is unspecified, (1.3) is regarded as a nonparametric mixture model. The space of all the mixing distributions on Θ is denoted G, and the space of those with at most m support points is G m . It can be seen that G m ⊂ G m+1 . See Titterington, Smith, and Makov (1985), Lindsay (1995), and McLachlan and Peel (2000) for the general theory and applications of mixture models. Research into mixture models has a long history. The most cited early publication is Pearson (1894); he used a two-component normal mixture model for a biometric data set. This model was motivated by the apparent skewness in the ratio of forehead-width to body-length of 1000 crabs from Naples. It was suspected that the population contained two unidentified species. Pearson (1894) employed the method of moments for the parameter estimation and provided a detailed account of the numerical calculation in the absence of the modern computer. Nowadays, we would estimate the parameters by the maximum likelihood estimator (MLE). The MLE is favored for its asymptotic efficiency under regular parametric models. The unpleasant numerical obstacle is now a history.
Mixture distributions form a distinct class of nonregular statistical models. They are notorious for presenting statisticians with serious technical challenges as well as many pleasant surprises. Given an i.i.d. sample of size n from a mixture model, Hartigan (1985) showed that the likelihood ratio test statistic for homogeneity is stochastically unbounded defying the classical chisquared limiting distribution. Chen (1995) showed that the optimal rate for estimating the mixing distribution is O p (n −1/4 ) as compared with O p (n −1/2 ) for parameter estimation under regular models. Quinn, McLachlan and Hjort (1987) interpreted some of these abnormal results by the degenerated Fisher information when the order of the finite mixture model is unknown. At the same time, Lindsay (1983) showed that the nonparametric MLE of G has at most k support points, the number of distinct observed values. Given the order of the finite mixture model and an initial mixing distribution G (0) , Wu (1981) found that the famous EM-algorithm leads to a sequence of G (k) that converges to a local maximum of the likelihood function and it can be a locally consistent MLE Peter and Walker (1978). Both Lindsay (1983) on the geometric property and Wu (1981) on the algorithmic convergence place only nominal conditions on f (x; G).
There is also good news about the large-sample properties of the MLE. Kiefer and Wolfowitz (1956) and Pfanzagl (1988) proved that the nonparametric MLE of G is strongly imsart-sts ver. 2014/10/16 file: Consistency.tex date: July 6, 2016 consistent in an i.i.d. setting. Using quotient topology, Redner (1981) gave another consistency proof under finite mixture models.
These consistency results are encouraging, but careful discussions of the relationships between these proofs are lacking. Why are there multiple proofs for a single consistency result? How do these proofs relate to each other? Without a full understanding, researchers may cite a paper when its specific proof/result is not applicable to the target problem. This paper streamlines the consistency proofs of the nonparametric MLE under mixture models and crystallizes their applicability. Another topic is consistent estimation under the finite normal mixture model: the consistency of the MLE under this model is often erroneously assumed. This paper aims to popularize the consistent estimator based on the penalized likelihood. The ultimate goal of this exercise is to provide a solid basis for research into the large-sample properties of mixture models.
The remainder of the paper consists of two major sections. Section 2 addresses classical consistency results for the nonparametric MLE under a mixture model. Section 3 addresses results related to the consistency of the penalized (or regularized) MLE under the finite normal mixture model. Section 4 provides concluding remarks.

CLASSICAL CONSISTENCY RESULTS
Let x 1 , . . . , x n be a set of i.i.d. observations of size n from a nonparametric mixture model {f (x; G) : G ∈ G}. The likelihood function of G is given by The log-likelihood function of G is hence Both L n (G) and ℓ n (G) are functions defined on G. The nonparametric MLEĜ n of G is a c.d.f. on Θ such that Most rigorously,Ĝ is related to the sample in a measurable fashion, and it is potentially one of many possible global maximum points of ℓ n (·). In addition, G can be a limiting point of a sequence of mixing distributions G j such that ℓ n (G j ) → sup{ℓ n (G) : G ∈ G} as j → ∞. For ease of presentation, (2.2) is regarded as an unambiguous definition of MLE. When G is confined to G m , this becomes the MLE of G under finite mixture models. When m = 1 in G m , (2.2) defines the ordinary parametric MLE of θ under the model {f (x; θ) : θ ∈ Θ}.
As acknowledged by Kiefer and Wolfowitz (1956), their proof of the consistency ofĜ is a technical clone of that of Wald (1949). One key technical preparation for this proof is the following well-known inequality.
Lemma 2.1. (Jensen's inequality). Let X be a random variable such that E|X| < ∞ and let ϕ(t) be a convex function. Then Suppose X is a random variable with density function f (x; θ) in general and θ * is the true value of the parameter. Let Y = f (X; θ)/f (X; θ * ). It can be seen that where E * is the expectation with respect to the distribution with the specific θ * parameter value. Applying Jensen's inequality to Y and ϕ(t) = − log(t), we get The equality holds if and only if f (x; θ) ≡ f (x; θ * ), or it is equal except for a zeroprobability set of x in terms of f (x; θ * ). The expectation E * [log{f (X; θ * )/f (X; θ)}] is referred to as the Kullback-Leibler information between the two distributions.
The following lemma may seem trivial, but it is the basis of most proofs for generic consistency.
Then the MLEθ n of θ equals θ * almost surely as n → ∞.
By the strong law of large numbers, this inequality implies almost surely. By the definition of the MLE,θ n = θ * almost surely as n → ∞.
Surprisingly, most generic MLE consistency proofs are variations or novel upgraded versions of this lemma. These include the situations where Θ is a subset of the Euclidean space R d , the space of all mixing distributions G or G m , or any abstract space.
In Section 2.1, we first present a trivialized Wald theorem, Theorem 2.1. It is universal consistency result though not directly applicable to many practical models. However, the full Wald Theorem can be easily explained from this angle.
Theorem 2.1 is then used as basis to understand and prove the consistency results of Kiefer and Wolfowitz (1956); Pfanzagl (1988) in the context of mixture models subsequently presented in Sections 2.2-2.4.

Essentials of the Wald consistency proof
Let {f (x; θ), θ ∈ Θ} be the probability model under investigation, where Θ is an abstract but metric parameter space. For any subset B of Θ, define Denote the distance on Θ as ρ(·, ·). For any ǫ > 0, let be an open ball of radius ǫ centred at θ * . Let B c be the complement of B.
Suppose that for any ǫ > 0, there exists a finite number of subsets Then,θ n → θ * almost surely as n → ∞.
The proof is identical to that of Lemma 2.2. The technicalities of Wald (1949) amount to specifying conditions on the model that lead to the generalized Jensen's inequality (2.5). What are the conditions placed on f (x; θ) by Wald (1949)? Here is a reorganized and slightly altered list.
Clearly, if F (x; θ) = F (x; θ * ) for all x, there will be no stochastic difference between the data sets generated from these two distributions. Hence, consistent estimation of θ based only on data is not possible if θ = θ * . Identifiability is therefore a necessary condition for the consistent estimation of θ.
Here, |θ| can be the Euclidean norm or any norm appropriate in the context of the problem. Technically, continuity may not be required for a zero-probability set of x in terms of f (x; θ 0 ). The models used in applications are usually well behaved. Hence, this extra generality is not generally needed if one takes appropriate care and therefore is not included as part of the condition.
For the consistent estimation of a parameter, the distributions with close θ values must be similar. Therefore, the continuity condition is indispensable.
The use of these conditions is demonstrated in the following lemma. Proof. Note that log{f (X; B ǫ (θ))/f (X; θ * )} is a monotone increasing function of ǫ. The continuity condition (W2) ensures that lim f (X; B ǫ (θ)) = f (X; θ) as ǫ → 0 + , i.e., when ǫ decreases to 0. Thus, this condition validates the dominated convergence theorem in the following way: For the negative counterpart of this expectation, the famous Fatou's Lemma implies, under the continuity condition (W2), The monotonicity on the left-hand side in ǫ ensures that the limit exists, rather than merely lim inf. Hence, with the strict < 0 implied by the identifiability condition (W1). The above result shows that for each θ = θ * , there is a small enough ǫ θ such that E * log{f (X; B ǫ θ (θ))/f (X; θ * )} < 0.
Since f (x; θ) → 0 as |θ| → ∞, we can similarly show that there exists r such that E * log{f (X; B c r (θ * ))/f (X; θ * )} < 0. For any ǫ > 0, letB r (θ * ) be the closure of B r (θ * ). Then is bounded and closed and therefore compact. It is trivial to see that When a compact set is covered by the union of a collection of open sets, it is covered by a finite number of such sets. Applying this result here, we have a finite number of B ǫ θ (θ) whose union covers the compact set A. Let these be B 1 , . . . , B J−1 , and reserve B J for B r (θ * ). All the conditions in Theorem 2.1 are satisfied. Hence, the MLE is consistent under (W1)-(W4).
We have successfully established the consistency result of (Wald, 1949) stated as Lemma 2.3. The proof is markedly simplified and it is obtained without requiring When Θ is not a closed set, it is often possible to continuously extend the range of the density function f (x; θ) to the closure of Θ,Θ. The Wald Theorem may then be applicable to the expanded model. Take the Poisson distribution family as an example: for x = 0, 1, . . . and Θ = (0, ∞). It can be seen that Θ is not a closed set. However, by defining f (x; 0) = 0 for all x except for f (x; 0) = 1 when x = 0, we extend the model toΘ = [0, ∞). Consequently, the conditions for a consistent MLE are satisfied and the MLE is consistent. Of course, the consistency of the MLE can be easily established without utilizing the generic Wald Theorem.

Consistency of nonparametric MLE under mixture model: Kiefer-Wolfowitz (KW) approach
We now illustrate and understand the contribution of Kiefer and Wolfowitz (1956) through its connection to Theorem 2.1. To discuss the consistence of the nonparametric MLE, we need to choose a distance D(·, ·) on G such as where |θ| is interpreted as |θ 1 | + · · · + |θ d | and dθ as dθ 1 · · · dθ d when Θ ⊂ R d . Note that there are two d's involved: one is the dimension of θ and the other is the differential symbol for the integration.
At this moment, a generic notion of distance suffices. We say that G → G 0 if D(G, G 0 ) → 0. Suppose G * ∈ G is the true mixing distribution andĜ is an estimator. ThenĜ is strongly consistent when D(Ĝ, G * ) → 0 almost surely.
Suppose D KW (·, ·) is chosen as the distance on G. When f (x; θ) is confined to the Poisson distribution, (KW1) is satisfied. When f (x; θ) is the normal distribution, this condition is violated. When f (x; θ) is binomial, it is also violated in general. When G is reduced to G m , the normal mixture satisfies (KW1); the binomial mixture satisfies (KW1) when m is small. See Teicher (1961Teicher ( , 1963. For G m , a mixing distribution can be expressed by two vectors: one for the component parameter values and one for the corresponding mixing proportions. Suppose the Euclidean distance on this vector space is chosen. Then f (x; G) remains the same when the entries of the two vectors are permuted. This is the loss of identifiability due to label switching. Quotient topology as suggested by Redner (1981) can be used to avoid this problem. Label switching leads to technical difficulties for Bayesian analysis; see Frühwirth-Schnatter (2011) and Stephens (2000).
(KW2) Continuity: The component parameter space Θ is a closed set. For all x and any given G 0 , lim imsart-sts ver. 2014/10/16 file: Consistency.tex date: July 6, 2016 (KW3) Finite Kullback-Leibler Information: For any G = G * , there exists an ǫ > 0 such that (KW4) Compactness: The definition of the mixture density f (x; G) in G can be continuously extended to a compact spaceḠ while retaining the validity of (KW3).
Proof. Under the theorem conditions and following the proof of Theorem 2.1, there must exist a δ > 0 for each G = G * such that This leads to a finite open cover of the compact set B c ǫ (G * ) for any ǫ > 0. Hence, by the law of large numbers, almost surely. The arbitrariness of ǫ implies thatĜ n is within an infinitesimal neighborhood of G * almost surely as n → ∞ and is therefore consistent for G * .
The proof of the consistency result in Theorem 2.2 is not fundamentally different from that of Kiefer and Wolfowitz (1956). The simplicity comes from requiring high level conditions (KW3) and (KW4), and from not repeating some steps in Lemma 2.3. The current proof promotes the understanding of the essence of their proof. At the same time, (KW3) and (KW4) can be established by (a): introducing a specific distance on G; (b): extending the domain of f (x; G) in G continuously to a compactḠ; (c): verifying that Jensen's inequality hold onḠ. Going over these steps brings back the complexity. We illustrate this point sub- To compactify G, we introduce ρG as a subdistribution for any ρ ∈ [0, 1). LetḠ be G supplemented with all the subdistributions. Clearly, D KW (·, ·) is easily extended toḠ. Similarly, we naturally extend the range of Although we have technically expanded G toḠ, the likelihood cannot be maximized onḠ − G since if f (x; G) = 0 then f (x; ρG) < f (x; G) when ρ < 1. Hence, the MLE onḠ is always a proper distribution on Θ.
Since Θ is a closed subset of R d from condition (KW2), the limit of any Cauchy sequence in G in terms of D KW (·, ·) is a subdistribution on Θ. Hence,Ḡ is a closed set. With the addition of total boundedness,Ḡ is compact.
The extension from {f (x; G) : G ∈ G} to {f (x; G) : G ∈Ḡ} is largely symbolic. The real issues are continuity and (KW3) onḠ. For instance, the extension would fail on a normal mixture because the component density function is not defined at σ = 0. Otherwise, the continuous extension is feasible.
Here is the sufficient and likely also necessary condition for the continuity of f ( for all bounded and continuous functions h(·), according to one of many equivalent definitions. By condition (W2), lim |θ|→∞ f (x; θ) = 0. Thus, for given x, f (x; θ) is continuous and bounded on Θ. Therefore, this definition leads to Remark: Here the convergence in distribution includes subdistributions.

Generalized Jensen's inequality.
Technicality is inevitable when it comes to establishing the generalized Jensen's inequality for the mixture model onḠ. The user must decide whether or not (KW3) holds onḠ for each specific mixture model. Two examples will help to explain the issue. Proof: Let θ 0 be a support point of G * . There must be a positive constant δ such that The condition G * (M ) = 1 easily leads to the finiteness of both E * (X) and Since the probability mass function is bounded from above, we also have E This is (KW3) for all G ∈Ḡ.
Condition (KW3) is not always satisfied even when the component distribution is Poisson. Let G({A}) be the probability of θ ∈ A under mixing distribution G.
Example 2.2. Let G * be a mixing distribution such that G * ({log n}) = c{n(log n)(log log n) 2 } −1 with some normalizing positive constant c, for n = 20, 21, . . . Then under a Poisson mixture model, we have Consequently, for any ǫ > 0, Proof: The size of c in this example does not affect the proof, so we take c = 1. We choose this specific mixing distribution because E * {θ} = ∞. In addition, it can be seen that

It is seen that
The approximation is so precise that an error assessment is possible but unnecessary. Changing the variable via u = log t, we find The choice of log(20) as the lowest support point of G ensures that log(log(20)) > 1, which avoids some technicalities in the above inequality. Hence, Let δ x be a distribution with all the probability mass at x. It can be seen that Therefore, (1 − ǫ)δ 1 + ǫδ x ∈ B ǫ (1) for any x. Hence, for any x value, imsart-sts ver. 2014/10/16 file: Consistency.tex date: July 6, 2016 by the Stirling formula. Because the Stirling formula is very accurate, this implies that log{f (x; B ǫ (1))/f (x; G * )} ≥ log(ǫ/ √ 2π) + (1/2) log x.
Hence, for any ǫ > 0, we have The point is that condition (KW3) on a compactḠ places a severe restriction on the KW proof. Luckily, the Pfanzagl proof is free of this restriction.

Consistency of nonparametric MLE under mixture model: Pfanzagl approach
The most demanding condition in the consistency proof of Kiefer and Wolfowitz (1956) is (KW3), needed to validate the generalized Jensen's inequality (2.5). Unfortunately, (KW3) under a mixture model is hard to verify, as seen in the Poisson mixture example. The Pfanzagl approach requires merely (W2) in comparison.
Here is the inequality in Pfanzagl (1988) that takes over the role of Jensen's inequality.
Lemma 2.5. Let f (x) and f * (x) be density functions of any two distributions with respect to some σ-finite measure µ. For any u ∈ (0, 1), we have and equality holds if and only if f * (x) = f (x) almost surely with respect to the f * distribution.
Now consider the mixture model f (x; G) with its domain already extended tō G as in the last subsection.
Lemma 2.6. Assume that the mixture model is identifiable and (KW2) is satisfied onḠ. Then, for any G ∈Ḡ and G = G * , there exists an ǫ > 0 such that where G * is the true mixing distribution.
Proof: Note that f (x; B ǫ (G)) is a monotone increasing function of ǫ, or it decreases to f (x; G) for all x as ǫ → 0 + . In addition, given u ∈ (0, 1), Namely, this function has a finite lower bound, which enables the use of Fatou's Lemma. Hence, Consequently, there exists a positive ǫ value at which (2.6) holds.
This lemma is the counterpart of the generalized Jensen's inequality in the KW approach. It is challenging to verify the validity of the generalized Jensen's inequality under a general mixture model. In contrast, Lemma 2.6 is valid provided the parameterization is continuous and identifiable. Proof: For any δ > 0, let B c δ (G * ) be the distributions inḠ that are at least a distance δ from G * . BecauseḠ is compact, so is B c δ (G * ). Thus, the continuity condition implies that there exists a finite number of G k , k = 1, . . . , J, with corresponding ǫ k such that By the strong law of large numbers, (2.8) implies almost surely for k = 1, 2, . . . , J. Consequently, we have almost surely for each k = 1, 2, . . . , J. Combining this inequality with (2.7), we get almost surely. By interpreting the summation in terms of the log-likelihood function, we find ℓ n (uG * + (1 − u)G) > ℓ n (G) for all G ∈ B c δ (G * ) almost surely. Since the likelihood function at any G ∈ B δ (G * ) is smaller than the likelihood value at another mixing distribution uG * + (1− u)G imsart-sts ver. 2014/10/16 file: Consistency.tex date: July 6, 2016 that is a member ofḠ, the members of B c δ (G * ), all of which are at least a δdistance away from G * , cannot possibly attain the supremum of ℓ n (G). Hence, the nonparametric MLE must reside in the δ-neighborhood of G * almost surely. The arbitrarily small size of δ implies that D KW (Ĝ, G * ) → 0 almost surely as n → ∞. This completes the proof.
In this proof, Pfanzagl (1988) took tactical advantage of the linearity of the mixture model in mixing distributions: which is the density function of another mixture distribution.
There is a limitation in the Pfanzagl result. Consider the finite mixture model where G is replaced by G m for a given m. The Pfanzagl result is no longer applicable because uG * + (1 − u)G likely has more than m support points even if both G * and G have only m support points. In contrast, the KW proof leads to the consistency of the MLE under finite mixture models provided the corresponding conditions are satisfied. In addition, the KW conditions under finite mixture models are simple to verify, and they hold widely.
For finite mixture models, there is another widely cited paper on the consistency of the MLE.

Consistency of the MLE under finite mixture model: Redner approach
The generic results in Redner (1981) are not restricted to the finite mixture model: the paper examines models lacking full identifiability. Without identifiability, the parameter estimator is inconsistent in general. However, in many situations, the estimator may be regarded as consistent from a different angle. Suppose a probability model has its density function given by f (x; θ 1 , θ 2 ) = θ 1 exp(−θ 1 x) for x > 0. Of course, this is simply an exponential distribution with rate parameter θ 1 ; parameter θ 2 is irrelevant. Given a set of i.i.d. samples from a distribution in this model, the MLE of θ 1 is consistent, while there is no way to have θ 2 consistently estimated. At the same time, the consistent estimation of θ 2 is unnecessary: it has no role in this population. Let the distance between two vectors be ρ((θ 1 , θ 2 ), (η 1 , η 2 )) = |θ 1 − η 1 |.
Then the MLE would satisfy, almost surely, ρ((θ 1 ,θ 2 ), (θ * 1 , θ * 2 )) → 0 as the sample size n → ∞. Finite mixture models may not appear as trivial as in this example. The density function of the two-component normal mixture in mean parameter is given by The mixture with η = (α, θ 1 , θ 2 ) = (0.3, 1, 2) is identical to the mixture with η = (0.7, 2, 1). These two parameter vectors are apparently distinct in R 3 . Hence, identifiability is lost when the model is parameterized with this scheme. When η is regarded as a vector in R 3 equipped with the Euclidean distance, the MLE is not consistent.
Each η in the above model has a corresponding mixing distribution G. Let us define a distance for R 3 (with the first component in [0, 1]) as ρ(η 1 , η 2 ) = D KW (G 1 , G 2 ).
Since ρ(η 1 , η 2 ) = 0 does not lead to η 1 = η 2 in R 3 , ρ(·, ·) is not a mathematical distance. Regarding several distinct members of R 3 as identical leads to quotient topology. This turns out to be the core of Redner (1981).

Summary
The consistency result of Pfanzagl (1988) seems perfect except for the consistency of the MLE under finite mixture models. The proof of Kiefer and Wolfowitz (1956) contains conditions that are not user-friendly. However, these conditions become simple under finite mixture models. Hence, the two papers perfectly complement each other. Redner (1981) resorts to quotient topology to resolve nonidentifiability and thereby provides another consistency proof for the finite mixture model. Redner (1981), however, assumes that Θ is a compact subset of R d , making the result weaker.
These papers do not consider only mixture models, as this section may have suggested. In this section we have substantially streamlined the conditions and conclusions and provided additional insight in the context of mixture models.

CONSISTENCY UNDER FINITE NORMAL MIXTURE MODEL
One common requirement for the consistency of the MLE under a mixture model is that f (x; G) can be continuously extended toḠ. The first step of this extension is to have f (x; θ) continuously extended to include all θ on the boundary of Θ. This turns out to be impossible for the normal model with density function Because of this, none of the three approaches in the last section is applicable to normal mixture models. The above issue is not the only obstacle. The normal mixture model is not identifiable on G unless that space is reduced to G m for a prespecified m. This section is devoted to the consistent estimation of G under the finite normal mixture model.

Finite normal mixture model with equal variances
Consider the finite normal mixture model where the component distributions share an equal but unknown variance: imsart-sts ver. 2014/10/16 file: Consistency.tex date: July 6, 2016 Let G be the mixing distribution in component mean on Θ = R. The common variance σ is structural with parameter space R + . The log-likelihood function based on a set of i.i.d. samples is given by Here is a preliminary result similar to but much strengthened over that in Chen and Chen (2003). We no longer confine the mean parameter in a finite interval.
Letx be the sample mean, and s 2 It can then be seen that uniformly for all σ 2 > ∆. Let X be a random variable with the true finite normal mixture distribution. By the strong law of large numbers, s 2 n almost surely converges to var(X) as n → ∞. Hence, when log ∆ > log{var(X)} + (1/2), we have ℓ n (G; σ 2 ) < ℓ n (x, s 2 n ) almost surely for all σ 2 > ∆. This proves that the MLE for σ 2 is below this finite value ∆ almost surely. Next, because of the algebraic form of the normal density, regardless of the actual value of x. At the same time, Hence, for any G and σ 2 , there is another upper bound: Let M be an arbitrary positive number and denote the truncated θ value as Let θ = (θ 1 , . . . , θ m ) andθ = (θ 1 , . . . ,θ m ). The space ofθ is clearly compact given finite M . For any |x| ≤ M , we have

Applying (3.2) for |x i | > M and (3.3) for |x
Now we focus on the stochastic quantity Clearly, h(θ, x) ≤ M 2 and it is equicontinuous inθ for all x. Because the space ofθ is compact, by the uniform strong law of large numbers of Rubin (1956), almost surely and uniformly inθ. Because E * {h(θ, X)} is smooth inθ and it is clearly nonzero at eachθ, this implies where the infimum is over the compact space ofθ. Applying this result to (3.3), we find ℓ n (G, σ) ≤ −n{log σ + δ/σ 2 } almost surely for all σ and therefore (3.5) ℓ n (G, σ) − ℓ n (x, s 2 n ) ≤ −n{log σ − log(s n ) + δ/σ 2 − 1/2} almost surely. When σ 2 is small enough, the upper bound goes to negative infinity as n → ∞. Hence, the maximum value of ℓ n (G, σ) must be attained when σ > ǫ for some ǫ > 0. This completes the proof.
The key improvement of this lemma over that in Chen and Chen (2003) is that here the parameter space of θ is the noncompact R.
Based on this result, under the finite normal mixture model with equal variance, the effective component parameter space for (θ, σ 2 ) is R × [ǫ, ∆] from the asymptotic point of view. Restricting the space of σ in this way leads to a compact parameter space. On this component parameter space, we have lim |θ|→∞ f (x; θ, σ 2 ) = 0.
Hence, conditions (W2) and (W3) are satisfied after this restriction and the KW approach is applicable.

Finite normal mixture model with unequal variances
The unequal-variance assumption does not exclude the possibility that the true component variances are all equal. The density function is now given by In this case, the mixing distribution G is bivariate on R × R + and it mixes both mean and variance. The log-likelihood has the same symbolic form: Consider the case where m = 2. Let (θ 1 , σ 1 ) = (0, 1), α 1 = α 2 = 0.5. Let θ 2 = x 1 and σ 2 = 1/2k for k = 1, 2, . . . Let G k be the corresponding mixing distribution. This setup creates a sequence of mixing distributions {G k } ∞ k=1 . It can be seen that and that for i ≥ 2, Consequently, we have x 2 i − n log(2π).
Clearly, ℓ n (G k ) → ∞ as k → ∞. Hence, the limiting point of G k is one of the MLEs of G, which is inconsistent. There are some misconceptions in the literature. Since pr(X 1 = θ) = 0 for any given θ value, one may suggest that the probability of having a degenerate MLE is zero. This is false because θ = x 1 , in which x 1 is an observed value, is no longer random after the observation.
In applications, EM-algorithm can be used to locate many nondegenerate local maxima of ℓ n (G). The one with the largest likelihood value is a locally consistent MLE of G (Peter and Walker, 1978;Redner and Walker, 1981). One may also use a consistent estimator, possibly via method of moments, as an initial mixing distribution for the EM-algorithm. Gan and Jiang (1999) developed an approach to test for global maximum, which can be useful in the current context. Experience shows that such an estimator has good statistical properties, so we should not write off this practice.
The inconsistency conclusion may not be an obstacle in many applications. Nonetheless, it is more satisfactory to have a foolproof method with solid underlying theory that performs well in applications. In the literature there are two approaches to consistent estimation based on likelihood. One is the constraint MLE proposed by Hathaway (1985). Simply put, it reduces the component parameter space of σ. The result of the last subsection may be regarded as its simplest case.
This subsection focuses on the penalty method applied to σ. The penalty is also a prior on σ or a regularization measure. The first largely successful proof of consistency for the penalized MLE is in Ciuperca, Ridolfi, and Idier (2003), following its proposal in Ridolfi and Idier (1999). Chen, Tan, and Zhang (2008) provided a successful complete proof that is simplified and improved here.

Penalized likelihood.
The inconsistency of the MLE under the finite normal mixture model is largely due to nonregularity. Hence, regularizing the likelihood is a natural way to gain consistency of the altered MLE. The regularization is itself in the form of a penalized likelihood as follows: The mixing distribution is then estimated by one of the global maxima ofl n (G) over G m :G = arg maxl n (G).
The uniqueness is a natural consequence of the subsequent discussion. We denote the component means inG asθ j and so on.
Combining the two bounds leads to almost surely. Because n −1 is a high-order term compared to n −1 log n, it is absorbed into the latter.
The proof remains solid when ǫ depends on n. Because of this, ǫ is allowed to take an arbitrarily small value without invalidating the inequality. Technically, Lemma 3.2 leaves a zero-probability event for each value of ǫ on which the upper bound is violated. The union of these zero-probability events over ǫ does not have to be a zero-probability event. However, since sup θ {F n (θ + ǫ) − F n (θ)} is monotone in ǫ, this technicality is easily resolved.
Lemma 3.3. The upper bound in Lemma 3.2 after a minor alteration, holds uniformly for all ǫ > 0 almost surely.
This lemma shows that i.i.d. observations from a population with a bounded density function spread out evenly almost surely. This result extends a corresponding result in (Chen, Tan, and Zhang, 2008) to cover generic distribution F .
The first property allows for a simple discussion and straightforward numerical solution to the penalized MLE. The upper and lower bounds in P2 prevent the likelihood from being seriously inflated or deflated at any G. Property P3 requires the size of the penalty to be large enough to prevent σ j ≈ 0 in the penalized MLE of G. One possiblep n isp This penalty function goes to negative infinity when σ → 0 or σ → ∞. It is minimized when σ = 1. The upper-bound condition in P2 on [p n (σ)] + is certainly satisfied. The lower-bound condition in P2 is clearly satisfied. As for P3, when σ → 0 + , σ −2 → ∞ much faster than | log σ|. Hence, the penalty is much more severe than the required 4(log 2 n) log σ. In applications, σ should be replaced by σ/s n , where s n is the sample variance, to retain scale invariance. This penalty function also represents a prior Gamma distribution placed on σ −2 . This form of penalty is very convenient for the EM-algorithm popularly used for numerical computation and on Bayes analysis as in Redner, Hathaway and Bezdek (1987); Ridolfi and Idier (1999).

Consistency of the penalized MLE.
We now outline the proof for m = 2; the general case is similar and omitted. Let K * = E * {log f (X; G * )} and M = sup x f (x; G * ). Select a sufficiently small ǫ 0 such that Without loss of generality, we assume σ 1 ≤ σ 2 . Partition the mixing distribution space G 2 into The overall strategy is to show that the penalized MLE is not in Γ 1 or Γ 2 but in Γ 3 . The final conclusion follows from the fact that the finite normal mixture model on Γ 3 fits into the KW proof.
Step I. The following lemma says it all.
Proof: Define A j = {i : |x i − θ j | < |σ j log σ j |} for j = 1, 2. Partition the entries in ℓ n (G) to get where ℓ n (G; A) = i∈A log f (x i ; G). Denote the number of observations in set A as n(A). Since the mixture density function is bounded by σ −1 1 for G ∈ Γ 1 , ℓ n (G; A 1 ) ≤ −n(A 1 ) log(σ 1 ).
This completes the proof.
Remark: I have omitted "almost surely" in the proof for ease of presentation.
OnΓ 2 , σ 2 has a nonzero lower bound. Thus, g(x; G) is bounded although σ 1 = 0 is allowed. Without loss of generality, τ 0 is small enough such that the true mixing distribution G * ∈ Γ 2 . Hence, applying Jensen's inequality, we also have E * log{g(X; G)/f (X; G * )} < 0 for all G ∈Γ 2 . Using a slightly different symbol from ℓ, we define l n (G) = n i=1 log{g(x i ; G)} onΓ 2 . By the strong law of large numbers and the newly established Jensen's inequality, Further exploring this key conclusion leads to the following lemma.
Lemma 3.5. Consider a set of n i.i.d. observations from f (x; G * ) and the function g(x; G) defined by (3.11). Let l n (G) = n i=1 log{g(x i ; G)}. We then have for some δ(ǫ 0 ) > 0 almost surely.
Clearly, E * log g(X; G, ǫ) > −∞. Therefore, E * log{g(X; G, ǫ)/f (X; G * )} is well defined. Let ǫ → 0 + ; by the monotone convergence theorem we find Next, note thatΓ 2 is compact based on the distance D KW (·, ·). There is a finite number of G and ǫ such that and for each j = 1, . . . , J, This leads to the claim of this lemma: for some δ(ǫ 0 ) > 0 whose size depends on the size of ǫ 0 .
Let us connect l n (G) to ℓ n (G) on the spaceΓ 2 and refine this result to obtain the major result of this step. Lemma 3.6. As n → ∞, sup G∈Γ 2l n (G) −l n (G * ) → −∞.

Proof: Retain the definition
Therefore, the log-likelihood contribution of each observation in A 1 is log{f (x i ; G)} ≤ log(1/σ 1 ) + log{g(x i ; G)}.
For these x values, we have Consequently, The factor 1/σ 1 has been turned into exp(− log σ 1 ) in the first line of the above derivation. For a small enough ǫ 0 , exp − 1 4 (log σ 1 + 2) 2 + 1 ≤ 1 2ǫ 0 when σ 1 ≤ ǫ 0 . Hence, (3.13) leads to, for those x values not in the set A 1 , and therefore, In summary, when i ∈ A 1 , its log-likelihood contributions Combining the cases for the observations in and not in A 1 , we find ℓ n (G) ≤ n(A 1 ) log(1/σ 1 ) + log{g(x i ; G)}.
Hence, the proof of the lemma is reduced to showing that (3.14) sup for some δ > 2M τ 0 (log τ 0 ) 2 . This is implied by Lemma 3.5 when a sufficiently small τ 0 is chosen, after the choice of ǫ 0 .
The g(x; G) used here is more convenient than that in Chen, Tan, and Zhang (2008). The proofs so far have successfully excluded the possibility that the penalized MLE of G falls in Γ 1 ∪ Γ 2 . Finishing the consistency proof is a simple task.
Theorem 3.2. The penalized MLE of G is consistent:G → G * almost surely as n → ∞.
Once the mixing distribution is restricted to Γ 3 , the KW conditions are satisfied. Hence, the restricted MLE is consistent. On this space, the penalty is of size o p (n). Hence, the penalized MLE remains consistent. Note that the KW conditions are easy to verify for finite normal mixture models on Γ 3 .

Summary
The finite normal mixture model does not satisfy the KW conditions, and the MLE as defined by (2.2) together with subsequent remarks is inconsistent. The consistency of the penalized MLE has only recently been solidly proved. Yet these facts are often overlooked; we have made these results more accessible to researchers in various disciplines. The penalized MLE under a multivariate normal mixture has also been shown to be consistent by Chen and Tan (2009) with a minor correction by Alexandrovich (2014). Note that the proof of the multivariate case can be substantially simplified using the new techniques in this paper. Finally, being consistent is a minimum requirement in statistical data analysis. The proper estimation of the mixing distribution under a finite mixture model requires a very large sample size when the subpopulations are not well separated, as indicated by the simulation study of Chen and Tan (2009).

CONCLUDING REMARKS
We have discussed the consistency of the MLE under mixture models given i.i.d. observations.
When there are no restrictions on the mixing distribution, the nonparametric MLE is consistent under the minimum identifiability condition (KW1) and the continuity condition (KW2) based on Pfanzagl's proof. In addition, (KW2) is implied by (W2) and (W4).
The conclusion via the KW proof is most useful when applied to finite mixture models. In this case, the KW conditions are implied by Wald conditions (W2)-(W4) and (KW1). The MLE of the mixing distribution under a finite mixture is consistent.
Under the finite normal mixture model with equal variances, the MLE is consistent. Under the finite normal mixture model in both mean and variance, the MLE is not consistent. When a penalty satisfying P1-P3 is applied to the loglikelihood function, the penalized MLE is consistent.
It is curious that the consistency proof given by Pfanzagl for the nonparametric MLE under a mixture model requires a somewhat weaker set of conditions than its parametric counterpart given by Wald: (KW1), (W2), and (W4), but not (W3). Because Pfanzagl's result is not applicable to finite mixture models, it does not cover the result of Wald or that of KW. The KW proof makes that of Wald a special case. This paper is novel at streamlining the conditions, conclusions and the proofs related to the consistency of the MLE under mixture models. The proofs of Kiefer and Wolfowitz (1956); Pfanzagl (1988); Redner (1981); Wald (1949) are substantially simplified by connecting all of them with Theorem 2.1. The proofs focus on essential ideas and leaves complex conditions out for separate discussions. The Wald consistency result is strengthened by de-requiring E * | log f (X; θ * )| < ∞. The KW consistency conclusion is found most useful to finite mixture models. Its generic conclusion is less useful because its (KW3) condition is difficult to verify or not satisfied as testified by a Poisson mixture example. The paper follows the existing line of proofs for the consistency of the MLE under finite normal mixture models. When the variance is a structural parameter, the conclusion is strengthened by de-require the space of the mean parameter being compact. The consistency proof of the penalized MLE under finite normal mixture models is simplified though developing a more generic concentration inequality in Lemma 3.3 and introducing a more convenient function g(x; G, ǫ) in Lemma 3.5.