{'fraction_non_alphanumeric': 0.057602623891764466, 'fraction_numerical': 0.01557935735150925, 'mean_word_length': 3.936503921072603, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Feedback Capacity of Parallel ACGN Channels and Kalman Filter: Power Allocation with Feedback

In this paper, we relate the feedback capacity of parallel additive colored Gaussian noise (ACGN) channels to a variant of the Kalman filter. By doing so, we obtain lower bounds on the feedback capacity of such channels, as well as the corresponding feedback (recursive) coding schemes, which are essentially power allocation policies with feedback, to achieve the bounds. The results are seen to reduce to existing lower bounds in the case of a single ACGN feedback channel, whereas when it comes to parallel additive white Gaussian noise (AWGN) channels with feedback, the recursive coding scheme reduces to a feedback"water-filling"power allocation policy.


I. INTRODUCTION
The feedback capacity [1] of additive colored Gaussian noise (ACGN) channels has been a long-standing problem in information theory, generating numerous research papers over the years, due to its significance in understanding and applying communication/coding with feedback. In general, we refer to the breakthrough paper [2] and the references therein for a rather complete literature review; see also [3], [4] for possibly complementary paper surveys. Meanwhile, papers on this topic have also been coming out continuously after [2], which include but are certainly not restricted to [5]- [17]. Most of the aforementioned works, however, focused merely on the feedback capacity of a single ACGN channel, so to speak, whereas when it comes to parallel ACGN channels, the corresponding results have been lacking in general. One exception is the recent paper [11] which generalized the computational approach in [10] to multi-antenna ACGN channels.
In this paper, we establish a connection between a parallel of ACGN channels with feedback and a variant of the multi-input multi-output (MIMO) Kalman filter for colored Gaussian noises. In light of this, we obtain lower bounds on feedback capacity for parallel ACGN channels by examining the algebraic Riccati equations associated with the Kalman filter. Meanwhile, the Kalman filtering systems, which are essentially feedback (closed-loop) systems, naturally provide recursive coding schemes, in terms of feedback power allocation policies, to achieve the lower bounds. In addition, the lower bounds are shown to be consistent with existing feedback capacity results when it comes to a single ACGN channel. It is also seen that in the special case of parallel additive white Gaussian noise (AWGN) channels, the recursive coding reduces to a feedback "water-filling" solution.
In a broad sense, in this paper we utilize a control-theoretic approach towards this problem as in, e.g., [5]- [7], [10]- [14], [16], [18]- [20]; see also [21] and the references therein. Note also that although the organization of this paper resembles that of [16] to a certain extent, the results are not trivial generalizations of those therein, as evidenced by the results themselves as well as their proofs.
The rest of the paper is organized as follows. Section II provides the preliminary background on feedback capacity and Kalman filter. In Section III, we present the main results of this paper. Concluding remarks are given in Section IV.
II. PRELIMINARIES In this paper, we consider real-valued continuous random variables and discrete-time stochastic processes they compose. All random variables and stochastic processes are assumed to be zero-mean for simplicity and without loss of generality. We represent random variables using boldface letters. The logarithm is defined with base 2. A stochastic process {x k } is said to be asymptotically stationary if it is stationary as k → ∞, and herein stationarity means strict stationarity [22]. Note in particular that, for simplicity and with abuse of notations, we utilize x ∈ R and x ∈ R n to indicate that x is a real-valued random variable and that x is a real-valued n-dimensional random vector, respectively.
The following definitions of entropy and entropy rate are adapted from, e.g., [23].
Definition 1: The differential entropy of a random vector x with density p x (x) is defined as The entropy rate of a stochastic process {x k } is defined as

A. Feedback Capacity
Consider a parallel of n additive colored Gaussian noise channels given by where {x k } , x k ∈ R n denotes the channel input, {y k } , y k ∈ R n denotes the channel output, and {z k } , z k ∈ R n denotes the additive noise which is assumed to be stationary colored Gaussian. The feedback capacity C f of such a channel with power constraint P is given by [1], [23] It is worth mentioning that in the case of n = 1, it was shown in [2] that the optimal channel input process {x k } is stationary and in the form of while satisfying E x 2 k ≤ P . This has also been generalized to the case of n parallel channels by [11]. For the purpose of this paper, however, it suffices to consider the original definition in (1).

B. Kalman Filter
We now give a brief review of (a special case of) the MIMO Kalman filter [24], [25]; note that hereinafter the notations are not to be confused with those in Section II-A. Particularly, consider the Kalman filtering system depicted in Fig. 1, where the state-space model of the plant to be estimated is given by Herein, x k ∈ R n is the state to be estimated, y k ∈ R n is the plant output, and v k ∈ R n is the measurement noise, whereas the process noise, normally denoted as {w k } [24], [25], is assumed to be absent. The system matrix is A ∈ R n×n while the output matrix is C ∈ R n×n , and we assume that A is anti-stable (i.e., all the eigenvalues are unstable with magnitudes greater than or equal to 1) while the pair (A, C) is observable (and thus detectable [26]). Suppose that {v k } is white Gaussian with covariance V = E v k v T k 0 and the initial state x 0 is Gaussian with covariance E x 0 x T 0 0. Furthermore, {v k } and x 0 are assumed to be uncorrelated. Correspondingly, the Kalman filter (in the observer form [26]) for (2) is given by where x k ∈ R n , y k ∈ R n , e k ∈ R n , and u k ∈ R n . Herein, K k denotes the observer gain [26] (note that the observer gain is different from the Kalman gain by a factor of A; see, e.g., [25], [26] for more details) given by where P k denotes the state estimation error covariance as In addition, P k can be obtained iteratively by the Riccati equation Additionally, it is known [24], [25] that when (A, C) is detectable, the Kalman filtering system converges, i.e., the state estimation error {x k − x k } is asymptotically stationary. Moreover, in steady state, the optimal state estimation error variance T attained by the Kalman filter is given by the (non-zero) positive semi-definite solution [25] to the algebraic Riccati equation whereas the steady-state observer gain is given by In fact, by letting x k = x k − x k and y k = y k − z k = y k − Cx k , we may integrate the steady-state systems of (2) and (3) into an equivalent form: as depicted in Fig. 2, since all the sub-systems are linear.

III. LOWER BOUNDS ON FEEDBACK CAPACITY OF PARALLEL ACGN CHANNELS AND RECURSIVE CODING
The approach we take in this paper to obtain lower bounds on the feedback capacity of parallel ACGN channels is by establishing the connection between a parallel of ACGN channels with feedback and a variant of the Kalman filter for colored Gaussian noises. Towards this end, we first present the following variant of the Kalman filter.

A. A Variant of the Kalman Filter
Consider again the Kalman filtering system given in Fig. 1. Suppose that the plant to be estimated is still given by only this time with an auto-regressive moving average where Equivalently, {v k } may be represented [27] as the output of a linear time-invariant (LTI) filter F (z) driven Herein, we assume that F (z) is stable and minimum-phase.
We may now generalize the method of dealing with colored noises as employed in [16] (which in turn was developed based on [25]; see detailed discussions in [16]), to the case of MIMO Kalman filtering systems.
Proposition 1: and |λ | ≥ 1, = 1, . . . , n. Denote Then, (7) is equivalent to where Proof: Note first that since F (z) is stable and minimumphase, the inverse filter is also stable and minimum-phase. As a result, it holds ∀ |z| ≥ 1 that i.e., the region of convergence must include, though not necessarily restricted to, |z| ≥ 1. Consequently, for |z| ≥ 1, we may expand Accordingly, we may also rewrite Meanwhile, since A is anti-stable (and thus invertible), we have x k−i = A −i x k . As a result,

Note then that
Moreover, since T T is invertible and the eigenvalues of T T ⊗ I n×n are given by the n copies of the eigenvalues of T T , it thus follows that T T ⊗ I n×n is invertible and Accordingly, In addition, Meanwhile, we have already shown that ∀ |z| ≥ 1, As such, since |λ | ≥ 1, = 1, . . . , n, we have 3. The steady-state integrated Kalman filter for colored noises.
As a result, To sum it up, we have This completes the proof.
Meanwhile, the Kalman filter for (12) is given by where x k ∈ R n , y k ∈ R n , e k ∈ R n , and u k ∈ R n . Furthermore, when A, C is detectable, the Kalman filtering system converges, i.e., the state estimation error {x k − x k } is asymptotically stationary. Moreover, in steady state, the optimal state estimation error covariance T attained by the Kalman filter is given by the (non-zero) positive semi-definite solution to the algebraic Riccati equation whereas the steady-state observer gain is given by Again, by letting x k = x k −x k and y k = y k − z k = y k − Cx k , we may integrate the steady-state systems of (12) and (14) into as depicted in Fig. 3. In addition, it may be verified that the closed-loop system given in (17) and Fig. 3 is stable [25], [26].
In fact, we may design matrix C specifically to render matrix C an identity matrix.
and |λ | ≥ 1, = 1, . . . , n. If then Proof: It is known from the proof of Proposition 1 that is invertible, and it can be verified that its inverse is given by Hence, if (19) holds, then This completes the proof.
Note that when C = I n×n , the pair A, C is always observable (and thus always detectable [26]). To see this, note that if C = I n×n , then the observation matrix for A, C becomes  which has a row rank of n, indicating that A, C is observable [26], regardless of what A is. As such, in this case the Kalman filtering system always converges, whereas (15) reduces to and (16) reduces to

B. Feedback Capacity of Parallel ACGN Channels
We now proceed to obtain lower bounds on feedback capacity as well as the corresponding recursive coding schemes, based upon the discussions in the previous sub-section. We first examine the solution to the algebraic Riccati equation given by (15) when A and C are designed specifically.
Theorem 1: Suppose that in (8), and U v is an orthogonal matrix. If and where and then the (non-zero) positive semi-definite solution to (15) is given by Proof: Note first that the eigenvalues of Therefore, according to Proposition 1 and Proposition 2, when C is given by (26), it follows that C = I n×n while (22) holds.
On the other hand, it may be verified that P = U v Λ x U T v is the (non-zero) positive semi-definite solution to (22) as (Note also that clearly P = 0 is the other positive semi-definite solution to (22), which is not relevant herein though.) This completes the proof. Note that herein Λ and P can respectively be rewritten as and Note also that in the special case when {v k } is a white noise, i.e., when F i = G j = 0 in (8), (26) reduces to On the other hand, we may obtain an equivalent form of the system in Fig. 3 as given by (17).
Proposition 3: The system in Fig. 3 is equivalent to that in Fig. 4, where K (z) is dynamic and is given by Fig. 4. The steady-state integrated Kalman filter for colored noises: Equivalent form. Herein, K is given by (16). More specifically, the system in Fig. 4 is given by which is stable as a closed-loop system. Proof: Note first that the system of Fig. 3 is equivalent to the one of Fig. 5, since K = F (z) K (z). In addition, it is known from the proof of Proposition 1 that As such, since Consequently, the system of Fig. 5 is equivalent to that of Fig. 6. Moreover, since all the sub-systems are linear, the system of Fig. 6 is equivalent to that of Fig. 7, which in turn Fig. 6. The steady-state integrated Kalman filter for colored noises: Equivalent form 3. equals to the one of Fig. 4; note that herein F (z) is stable and minimum-phase, and thus there will be no issues caused by cancellations of unstable poles and nonminimum-phase zeros. Meanwhile, the closed-loop stability of the system given in (34) and Fig. 4 is the same as that of the system given by (17) and Fig. 3, since they are essentially the same feedback system.
As a matter of fact, in the system of Fig. 4, or equivalently, in the system of Fig. 8, we may view as a feedback channel [2], [5] with additive colored Gaussian noise {v k }, whereas {−y k } is the channel input while {e k } is the channel output. Note that in Fig. 8, may be viewed as a particular class of feedback coding. On the other hand, with the notations in (35), the feedback capacity is given by (cf. the definition in (1)) As such, if A and C are designed specifically as in Theorem 1, then (36) naturally provides a class of sub-optimal feedback coding scheme, by which the corresponding h ∞ (e )−h ∞ (v) that can be achieved is thus a lower bound of (37).
In this view, the following lower bound of feedback capacity can be obtained.
Theorem 2: Suppose that in (8), and U v is an orthogonal matrix. Then, a lower bound of the feedback capacity with power constraint P is given by where P 1 , . . . , P n satisfy Herein, where Proof: To start with, suppose that A and C are specifically designed as in Theorem 1. In this case, it is known from Theorem 3 that the system in (34) is stable. Note then that and thus is stable. Accordingly, since {v k } is stationary Gaussian, {−y k } is also stationary Gaussian. On the other hand, it holds that and as a consequence (cf. discussions in [5], [18]), where the first equality may be referred to [21], [22] while the third equality follows as a result of the Bode integral or Jensen's formula [21], [28]. Note that herein we have used the fact that F −1 (z) is stable and minimum-phase, (A, C) is detectable (thus the set of unstable poles of C (zI − A) is exactly the same as the set of eigenvalues of A with magnitude greater than or equal to 1; see, e.g., discussions in [29]), and is stable. Consequently, according to the definition of feedback capacity given in (37), it holds that when the corresponding is less than the power constraint P , i.e., when (see Theorem 1) Note that herein we have used the fact that {−y k } is stationary. In particular, we may pick the allocation P 1 , . . . , P n that maximizes This completes the proof.
Note that the solution to (39) is essentially a power allocation policy with feedback. Note also that he lower bound in Theorem 2 is equal to max a1≥1,...,an≥1 where a 1 , . . . , a n satisfy Herein, C is given by (41), where We now consider the case of independent parallel channels. Corollary 1: Suppose that in (8), and while G j = diag (g j1 , . . . , g jn ) , j = 1, . . . , q, which essentially model a parallel of independent ARMA noises. Then, a lower bound of the feedback capacity with power constraint P is given by where P 1 , . . . , P n satisfy Herein, Proof: Note first that in this case, U v = I and hence As such, if where a 1 , . . . , a n are given by (53), then similarly to the procedures in the proof of Proposition 1, it can be obtained that  Meanwhile, in this case it holds for = 1, . . . , n, that and Thus, , and the power constraint becomes Similarly, if Λ = −diag (a 1 , . . . , a n ) , then it can be obtained that , and the power constraint becomes This completes the proof. Equivalently, the lower bound in Corollary 1 can be rewritten as max a1≥1,...,an≥1 where a 1 , . . . , a n satisfy We next consider some special cases in which Theorem 2 (or Corollary 1) can be characterized more explicitly, including a parallel of AWGN channels (Example 1) and a single ACGN channel (Example 2).
Example 1: In the special case when {v k } is a white Gaussian noise with covariance V , i.e., when F i = G j = 0 in (8), a lower bound of the feedback capacity with power constraint P is given by where P 1 , . . . , P n satisfy As a matter of fact, the lower bound is tight in this case [23] and the optimal power allocation solution is given by the classical "water-filling" policy [23] as where ζ > 0 satisfies It is also worth mentioning that the lower bound in (57) can equivalently be rewritten as (cf. also discussions after Theorem 2 for the general case) max a1≥1,...,an≥1 where a 1 , . . . , a n satisfy n =1 a 2 − 1 V = P .
Correspondingly, the optimal "allocation" solution is given by where This provides an alternative perspective to look at the waterfilling allocation, while also displaying more clearly the connections with lower bounds in other cases, e.g., that of the subsequent Example 2. Example 2: For another special case, consider the scalar case of n = 1. In this case, (8) where { v k } , v k ∈ R is white Gaussian with variance σ 2 v > 0. Accordingly, Theorem 2 reduces to that a lower bound of the feedback capacity with power constraint P is given by where P satisfies Herein, It may then be verified that this lower bound can equivalently be rewritten as (cf. also discussions after Theorem 2 or Corollary 1) where a satisfies In fact, (66) coincides with the lower bound in [16] given as max a∈R log |a| , where a satisfies whereas this in turn reduces to the results in, e.g., [18] (see detailed discussions in [16], which also relates to the formulae in, e.g., [2]). Note that (34) and Fig. 8 essentially provide a recursive coding scheme to achieve the lower bound in Theorem 2. This is more clearly seen in Fig. 9, where L (z) is given by (36). More specifically, the recursive coding algorithm is given as follows.
Theorem 3: Suppose the optimal solution to (39) is given by P 1 , . . . , P n . Then, one class of recursive coding scheme to achieve the lower bound in Theorem 2 is given by Fig. 9. The steady-state integrated Kalman filter as a feedback coding scheme Herein, A = U v ΛU T v and C is given by (41), where Λ is given by In the case of parallel AWGN channels, Theorem 3 reduces to a recursive water-filling scheme.
Example 3: In the special case when {v k } is a white noise, i.e., when F i = G j = 0 in (8), the coding scheme of (75) reduces to Herein, A = U v ΛU T v and C = I n×n , where Λ is given by and ζ > 0 satisfies n =1 max 0, ζ − V = P .
Note that this is essentially a feedback ("closed-loop") waterfilling power allocation scheme, which is potentially more "robust" than the classical "open-loop" water-filling policy; cf. results in [30] for instance. We will, however, leave detailed discussions on this topic to future research.

IV. CONCLUSION
In this paper, from the perspective of a variant of the Kalman filter, we have obtained lower bounds on the feedback capacity of parallel ACGN channels and the accompanying recursive coding schemes in terms of power allocation policies with feedback. Possible future research directions include investigating the tightness of the lower bounds, as well as the special cases in which more explicit solutions (cf. water-filling) to the feedback power allocation policies may be derived.