{'fraction_non_alphanumeric': 0.061481481481481484, 'fraction_numerical': 0.014757834757834758, 'mean_word_length': 3.633315733896515, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On mutual information, likelihood-ratios and estimation error for the additive Gaussian channel

This paper considers the model of an arbitrary distributed signal x observed through an added independent white Gaussian noise w, y=x+w. New relations between the minimal mean square error of the non-causal estimator and the likelihood ratio between y and \omega are derived. This is followed by an extended version of a recently derived relation between the mutual information I(x;y) and the minimal mean square error. These results are applied to derive infinite dimensional versions of the Fisher information and the de Bruijn identity. The derivation of the results is based on the Malliavin calculus.


Introduction
Let w t , 0 ≤ t ≤ T denote the standard d-dimensional Wiener process and w ′ t the related white noise. The white noise channel is, roughly speaking, defined by y ′ (t) = x ′ (t) + w ′ t , 0 ≤ t ≤ T where x ′ (t) is a signal process independent of the white noise process w ′ t . In the context of detection theory, the key entity is ℓ(y), the likelihood ratio, i.e. the Radon-Nikodym derivative of the measure induced by the {y ′ (t), t ∈ [0, T ]} process with respect to the measure induced by the white noise {w ′ t , t ∈ [0, T ]}. In the context of filtering theory the key entities are the causal and the non causal estimates, i.e. the conditional mean E(x ′ t |y ′ η , 0 ≤ η ≤ t) or E(x ′ t |y ′ η , 0 ≤ η ≤ T ) respectively. In addition to this pair of random entities, there are also averaged entities such as the averaged minimal errors which amount  where the expectation is w.r. to the P (x; y) measure, and also the relative entropy Eℓ(y). Relations between the likelihood ratio ℓ(y) and the causal conditional expectation were discovered in the late 60's and this was soon followed by a relation between the mutual information and the causal mean square error [10], [5], [9]. These relations which involved causal mean square errors were based on the Ito calculus. Similar problems for the non causal estimator were also considered [6], [8]. The formulation and results in the non causal case were restricted to the finite dimensional time discrete model of the Gaussian channel. Recently, however, Guo, Shamai and Verdú (GSV) [7] applied information theoretic arguments to derive new interesting results relating the mutual information with non causal estimation in Gaussian channels.
The Ito calculus which has proved to be a powerful tool for the relations associated with causal estimation could not be applied to problems related to non causal problems which explains the slow progress in the direction of relations for non causal estimates. However, the development of the Malliavin calculus, namely, the stochastic calculus of variation which was introduced in the mid 70's led in the early 80's to results which prove to be a very useful tool for the non causal type of problems.
The purpose of this paper is to apply the Malliavin calculus in order to derive the extension of the finite discrete time relations between non causal estimation and likelihood ratios to continuous time (section 4), and to prove an extended version of the results of [7] relating the mutual information with causal estimation error (section 5). The modelling of the additive Gaussian channel on the abstract Wiener space (in contrast to the d-dimensional Wiener process on the time interval [0, T ]) yield in sections 4 and 5 results of wide applicability, e.g. for the filtering and transmission of images and random fields. The relation of these results to the de Bruijn identity, causal filtering and the logarithmic Sobolev inequality are discussed in sections 6 and 7.
In the next section we define the Abstract Wiener Space which generalizes the classical d-dimensional Wiener process, and formulate the additive Gaussian channel which will be considered in the paper. Also, the problems considered in sections 4 and 5 are outlined in this section. Section 3 is a very short introduction to the Malliavin calculus. Section 4 presents the results relating likelihood ratios (R-N derivatives) with non-causal least square estimates cf. remark 2 in section 4 for possible applications of these results to nonlinear filering. In section 5 we derive an extended version of the GSV results. These results are applied in section 6 to consider the notions of Fisher information and the de Bruijn identities in an infinite dimensional setup. Section 7 deals with abstract Wiener spaces endowed with a time parameter. This enables the comparison of results for causal estimations with corresponding results for non-causal estimation. It is shown that a restricted form of the logarithmic Sobolev inequality follows directly from the results derived in this paper. 1 0 η i (s)dw s · e i (t) converges to w(t) in quadratic mean. We will denote the sequence of independent Gaussian, identically distributed (2.1) Now {e i (t), t ∈ [0, 1], i = 1, 2, · · · } can be considered as a C.O.N. base of an Hilbert ds ds. This space H is known as the Cameron Martin space. Note that the Wiener process which is continuous but not differentiable is not an element in H. The same notation goes over to the case of the d-dimensional In addition to the Banach space W we have to consider the space W * of all continuous functionals on W and it can be shown that W * is a dense subspace of H (cf. e.g. [14]). Hence for e ∈ W * , it also holds that e ∈ H and "e ∈ W * operating on w ∈ W " is the stochastic integral Cf. e.g. [17], or appendix B of [18] and the references therein, for further information on the AWS. Note that, unlike the classical case, the Abstract Wiener Space does not have any time-like parameter (this however can be added cf. section 7).
C. In order to introduce the general setup of the additive Gaussian channel, let (W, H, µ W ) be an abstract Wiener space and let (H, σ(H), µ X ) be a probability space on the Cameron-Martin Hilbert space H which is induced by an H-valued r.v. X. Let θ = (x, w), x ∈ H and w ∈ W , set Θ = {θ} and consider the combined probability space which is the space of the mutually independent 'signals' x and 'noise' w. Now, since H is continuously embedded in W we can identify x with its image in W and defined the additive Gaussian channel as where ρ is a free scalar 'signal to noise' parameter which will become relevant in Section 5. We will denote by X and Y the sigma fields induced on W by the r.v.'s x and y respectively. Note that y and w are W valued, x is H valued and we identify x with its image in W . In fact we will make throughout this paper, just for reasons of simplicity, the additional assumption that x is W * valued. As mentioned earlier, since W * ⊂ H ⊂ W we can also consider x to be H or W valued.
In section 4 we will be interested in the relation between two types of objects. The first class of objects is for e, e 1 , e 2 ∈ W * or globally The second class of objects are the likelihood ratio (the R-N derivative) between the measures induced by y and the one induced by w on W . This likelihood ratio will be denoted ℓ(w), w ∈ W . Note that if W is infinite dimensional then the measure induced by x is singular with respect to the measure induced by w, (since x ∈ H while w ∈ H).
In section 5 we will consider the relation between I(X; Y ) or rather dI(X; Y )/dρ and the non-causal filtering error: A related result for d(E log ℓ(w))/dρ is considered in section 6 and shown to be an extended version of the De Bruijn identity.

(a) The gradient
Let (W, H, µ) be an AWS and let e i , i = 1, 2, . . . be a sequence of elements in W * . Assume that the image of e i in H form a complete orthonormal base in H. Let f (x 1 , . . . , x n ) be a smooth function on R n and denote by f ′ i the partial derivative of f with respect to the i-th coordinate and let δe be as discussed in the previous section.
For cylindrical smooth random variables F (w) = f (δe 1 , . . . , δe n ), define . Therefore we set the following: It can be shown that this definition is closable in L p (µ) for any p > 1, which means that it can be extended to a wider class of functional as we will see below. We will restrict ourselves to p = 2, consequently the domain of the ∇ operation can be extended to all functions F (w) for which there exists a sequence of smooth cylindrical functions F m such that F m → F in L 2 and ∇F m is Cauchy in L 2 (µ, H). In this case set ∇F to be the L 2 (µ, H) limit of ∇F m . This class of r.v. will be denoted D 2,1 . It is a closed linear space under the norm Similarly let K be an Hilbert space and k 1 , k 2 , . . . a complete orthonormal base in K.
and denote by D 2,1 (K) the completion of ∇ϕ under the norm Note that this enables us to define recursively ∇ n F (w) for n > 1.

(b) The divergence (the Skorohod integral)
A few introductory remarks. Let v(x), x ∈ R n take values in R n , v(x) = 1 v i (x)ρ i , where the ρ i are orthonormal vectors in R n . Assume that the v i and F (x) are smooth and converge "quickly enough" to zero as |x| → ∞. Then the following "integration by parts formula" holds where div is the divergence: Note that the gradient and divergence are differential operations, and equation (3.5) deals with integration with respect to the Lebesgue measure on R n . In this subsection we are looking for an analog of the divergence operation on R n which will yield an integration by parts formula with respect to the Wiener measure.
Let u(w) be an H-valued r.v. in (W, H, µ), u will be said to be in dom 2 δ if E|u(w)| 2 H < ∞ and there exists a r.v. say δu such that for all smooth functionals f (δe 1 , . . . , δe n ) and all n the "integration by parts" relation is satisfied. δu is called the divergence or Skorohod integral. A necessary and sufficient condition for a square integrable u(w) to be in dom 2 δ is that for some γ = γ(u), for all smooth f . Note that while the definition of ∇f (at least for smooth functionals) is invariant under an absolutely continuous change of measure, this is not the case for the divergence which involves expectation in the definition. For non-random h ∈ W * , δh = h, w , setting f = 1 in (3.6) yields that Eδh = 0. It can be shown that if u ∈ D 2.1 (H) then u ∈ dom 2 δ. Also, for smooth f (w) it can be verified directly that and more generally under proper restrictions Consequently, if E|u| (c) Let (W, H, µ) be an abstract Wiener space and let µ 1 be another probability measure on the same space (W, σ{W }). Assume that µ 1 is absolutely continuous with respect to µ. Set ℓ(w) = dµ 1 dµ (w) and Q(w) = {w : ℓ(w) > 0} E 1 and E 0 will be used to denote the expectation with respect to the measures µ 1 and µ respectively. We will use the convention 0 log 0 = 0 throughout the paper.
Following the definition in 3(b), we define the divergence with respect to µ 1 to be as follows. The H-valued random variable u(w) will be said to be in dom 1 2 δ if there exists a r.v., say δu, which is L 2 under u 1 and such that for all smooth r.v.s f (w), it holds that The relation between δu and δu is given by the following lemma.

Relations between the estimation error and the likelihood ratio
Let (W, H, µ), (H, σ(H), µ X ), (Θ, F , P) and y(θ) = ρx + w be as in section 2. We will further assume that the H-valued r.v. x is actually W * valued, and exp α(x, h) H ∈ L 1 (µ X ) for all real α and all h ∈ W * . The measures induced by y and x on W will be denoted µ Y and µ X respectively. The conditional probability induced on W by y(θ) conditioned on x will be denoted by µ Y |X . Similarly, µ X|Y will denote the conditional probability induced on W * of x conditioned on y (cf. e.g. [4] for the existence of these conditional probabilities).
By the Cameron-Martin theorem (cf. e.g. [18]) and since x and w are independent, we have which by our assumptions belongs to L p for all p > 0. Hence, denoting by µ X (dx) the restriction of P to H: Proposition 4.1 Under these assumptions it holds that (a) a.s. µ W . Note that a denotes the conditional expectation conditioned on Y , equation (2.7). and (4.8) (c.f. also equations (6.3) and (6.4)).
Remark 2: (a) Consider the case where the abstract Wiener space is a classical Wiener space R n , then u ∈ H is of the form Rn ds < ∞. Further assume that E 2), given w ∈ W we can replace the integration with respect to µ X (dx) with a Monte-Carlo approximation. Similarly we can replace ∇ n ℓ(w), h 1 ⊗ · · · ⊗ h n by applying ∇ n to the integrand of equation (4.2) and then replacing again the integrand with a Monte-Carlo approximation. This can then be applied to derive a numerical approximation to n i−1 (h i , x) i.e. the non-adapted non-linear fitlering of n i=1 (h i , x). We will not follow these directions.
The following lemma will be needed in the proof of Proposition 4.1: Lemma 4.1 Assume that µ Y and µ Y |X are absolutely continuous with respect to µ W then for all bounded and measurable functions ψ on Θ

Proof of Lemma: Let
Then, by Fubini's theorem Since the conditional probability µ X|Y is regular (cf e.g. theorem 10.2.2 of [4]) we also have Proof of Proposition: From (4.1) and (4.2) and since by our assumptions we may (by dominated convergence) interchange the order of integration and differentiation Thus, by Lemma 4.1 proving (4.3). The same arguments also hold for repeated differentiation which yields (4.4). (4.5) follows directly from (4.4) since proving (4.5) and (4.6). From (4.4) we have and (4.7) follows.
We conclude this section with some results for δ x and δ x (cf. part (c) of section 3). By the assumptions of this section x ∈ dom 2 δ and x ∈ dom 1 2 δ. Therefore by (4.3) Note that L = δ∇ is the number operator, i.e. if α(w) is a square integrable r.v. of the Wiener space and α = n=1 I n , where I n is the Wiener chaos decomposition of x; then, formally, Lα = n=1 nI n . Therefore if Lα(w) ∈ L 2 and E(Lα(w)) = 0 then L −1 Lα is well defined, consequently it holds by equation (4.3) that where c is a normalizing constant. For δ x we have δ∇ℓ(w) .
Proof: By (4.3) Hence by Lemma 3.1 The GSV relation between the mutual information and the mean square of the estimation error Consider the setup and assumptions in the first paragraph of section 4. The mutual information between x and y is defined as E will denote expectation w.r. to the measure in (2.5), (cf. e.g. [16]). E 0 will denote expectation w.r. to the Wiener measure and E 1 will denote expectation w.r. to the measure on W induced by y (hence Ef (y) = E 1 f (w) = E 0 ℓ(w)f (w)).

An extended version of the De Bruijn identity
The Fisher information matrix J associated with a smooth probability density p(y 1 , . . . , y n ), y ∈ R n is defined as J = ∂ 2 log p(y 1 , . . . , y n ) ∂y i ∂y j 1≤i,j≤n and then the Fisher information which is defined by the r.h.s. of (6.1) satisfies: where E is the expectation with respect to the p density. The De Bruijn identity (cf. [3] or [7] and the references therein) deals with the case where y = x + √ tw where w = w 1 , w 2 , . . . , w n and the w j , j = 1, . . . , n are i.i.d. N(0, 1) and x is an R n random variable independent of w. It states that The Fisher information matrix cannot be extended directly to the case where y is infinite dimensional. However, the results of sections 4 and 5 yield some similar relations. Under the assumptions of section 5, comparing (5.1) with (5.3) we have which is "similar" to (6.2) and may be considered an extended De Bruijn identity.
Note that E 1 log ℓ(w) is the relative entropy of µ Y relative to µ W , also, note the difference between the ρ and the t parametrizations.
Comparing (5.1) with (4.8) yields E 1 trace ∇ 2 log ℓ(w) = ρ dI(x; y) dρ which is different from (6.1) by the ρ 2 E|x| 2 H term. Note that the validity of (6.4) is restricted to the case where ℓ(w) is induced by a signal plus independent noise model and not for any ℓ(w) which is a negative r.v. and whose expectation is 1, cf. the concluding lines of the next section.
7 Adding a "time parameter" to the abstract Wiener space Given an abstract Wiener space (W, H, µ) we can introduce the notion of continuous time on this space as follows. Let {π θ , 0 ≤ θ ≤ 1} be a continuous , strictly increasing, resolution of the identity on H with π 0 = 0, π 1 = I. Set F θ = σ{δπ θ h, h ∈ H} and F · will denote the filtration induced by F θ on [0, 1]. An H-valued r.v. u(w) will be said to be adapted to F · if (u, π θ , h) H is F θ measurable for all h ∈ H and every θ ∈ [0, 1], (c.f. section 2.6 of [18] for more details). Let D 2 (H) denote the class of H-valued u(w) such that E 1 |u| 2 H < ∞. The class of adapted square integrable random variables is a closed subspace of D 2 (H) and will be denoted by D a 2 (H). We will denote by u the projection of u ∈ D 2 (H) on D a 2 (H), i.e. (this corresponds to the dual predictable projection in martingale theory). Since x is independent of w, and since x = E 1 (x|σ(y)), then i.e. if x is not measurable on the σ-field induced by y, project, first, x on the σ-field generated by y and then project on D a 2 (H), which is the same as replacing u with x in (7.1). Then (cf. e.g. [18]) By the same arguments as in [5] or [9] and by the assumptions of proposition 5.1 without the restriction on ℓ(w) to be generated by a signal plus independent white noise. This seems to be a delicate problem; the left hand side of (7.7) can be shown to be equal to the left hand side of (7.6) without the restriction that the signal and noise are independent ( [9]). However it is not clear if the right hand side of (7.7) and (7.6) are equal.