{'fraction_non_alphanumeric': 0.06868259306505857, 'fraction_numerical': 0.019047895164096024, 'mean_word_length': 3.9374463212138564, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the existence of the stabilizing solution of generalized Riccati equations arising in zero-sum stochastic difference games: the time-varying case

In this paper, a large class of time-varying Riccati equations arising in stochastic dynamic games is considered. The problem of the existence and uniqueness of some globally defined solution, namely the bounded and stabilizing solution, is investigated. As an application of the obtained existence results, we address in a second step the problem of infinite-horizon zero-sum two players linear quadratic (LQ) dynamic game for a stochastic discrete-time dynamical system subject to both random switching of its coefficients and multiplicative noise. We show that in the solution of such an optimal control problem, a crucial role is played by the unique bounded and stabilizing solution of the considered class of generalized Riccati equations.


Introduction
This paper is devoted to the study of a globally defined solution of a large class of stochastic gametheoretic Riccati difference equations, namely the bounded and stabilizing solution. It is well known that the study of stochastic Riccati equations when compared to their deterministic counterparts offers some very specific challenges that could not be tackled mutatis mutandis. Several important results have been already reported in the literature in this context. We refer here for example to [6,1,5] (and the references therein). The results reported in the afore mentioned references are related to Riccati equations with sign definite quadratic terms. The study of global solutions of such sign definite nonlinear matrix equations has nowadays reached an advanced maturity state. In this study we are interested by stochastic Riccati equations with sign indefinite quadratic terms. Such matrix equations are encountered when dealing with stochastic dynamic games and will be referred to as stochastic game theoretic Riccati equations in the rest of the paper. Unlike their sign definite counterpart, stochastic game theoretic Riccati equations have received less attention and there still exist substantial open problems in the characterization of globally defined solutions of such nonlinear matrix equations (see [13,16] and the reference therein). In this study, we are interested by a particular globally defined solution, namely the bounded and stabilizing solution. We establish some fundamental properties regarding the solutions of the considered class of matrix difference equations such as uniqueness, monotonicity, comparison as well as existence theorems. Our main contributions along this direction are summarized as follows: • Uniqueness: Usually the uniqueness of the bounded and stabilizing solution of a Riccati type equation is proved employing some additional properties of this solution such as its maximality or its minimality in a class of admissible solutions. Such arguments cannot be used in the case of Riccati equations without definite sign. For the sign indefinite generalized Riccati equations addressed in this work, we propose an original proof for the uniqueness of the bounded and stabilizing solution. This proof does not rely on other properties of the stabilizing solutions than its boundedness. The main ingredient used here is the exponential stability of adequately defined positive operators on ordered Hilbert spaces. As a consequence of the uniqueness property it is obtained that the bounded and stabilizing solution is a periodic sequence if the coefficients of the Riccati equation under consideration are periodic sequences.
• Existence: We believe that existence conditions of the stabilizing solution for this type of nonlinear matrix equations has not been reported yet in the literature and represents one of our major contributions. The lack of results regarding such a class of Riccati equations is due to the absence of the sign information of the quadratic part of the equation. Our proposed existence conditions are characterized by the non emptiness of an adequately defined matrix set (A Σ ) and the stochastic detectability of an adequately defined auxiliary dynamic stochastic system, respectively. We will show that the non emptiness of A Σ is also a necessary condition (and hence not conservative) while the stochastic detectability condition is a stochastic counterpart of a standard existence condition in deterministic Riccati equations theory.
As we already mentioned above, the class of Riccati equations we are dealing with in this work are closely related to stochastic dynamic games. Hence, in the second part of this paper, we will address the problem of infinite-horizon zero-sum two players LQ dynamic game for a stochastic discrete-time dynamical system subject to both random switching of its coefficients and multiplicative noise. We state and solve both the so called full state-feedback and full information feedback cases. Following a Riccati equation approach, we show that in the solution of such control problems, a crucial role is played by the unique bounded and stabilizing solution of the considered class of generalized Riccati difference equations. We construct optimal strategies in a feedback-loop form by virtue of the bounded and stabilizing solution to the considered Riccati equations verifying specific sign conditions for each admissible strategy. As a byproduct, our proposed solution to this control problem inherits the numerical tractability of the proposed existence conditions for the bounded and stabilizing solution of the game theoretic Riccati equations. We believe that no former results establishing saddle point relations for such games exist in the literature.
This paper is organized as follows: In Section 2 we give the problem setting. Section 3 addresses the uniqueness property of the stabilizing solution of the considered class of Riccati equations. In Section 4 the existence conditions are given. In Section 5 we state and solve the problem of infinite-horizon zero-sum LQ stochastic dynamic game.
Notations. N = {1, 2, ..., N } where N ≥ 1 is a fixed natural number. A T stands for the transpose of the matrix A and T r[A] denotes the trace of a matrix A. The notation X ≥ Y (X > Y , respectively), where X and Y are symmetric matrices, means that X − Y is positive semi-definite (positive definite, respectively). In block matrices, ⋆ indicates symmetric terms: The expression M N ⋆ is equivalent to M N M T while M ⋆ is equivalent to M M T . Consider the following space of matrices M N n,m = R n×m × · · · × R n×m . In the case n = m we shall write M N n instead of M N n,n . We introduce the following convention of notations: • If B = B(1), · · · , B(N ) ∈ M N n,m and D = D(1), · · · , D(N ) ∈ M N m,p then C = BD ∈ M N n,p where C = C(1), · · · , C(N ) , C(i) = B(i)D(i), 1 ≤ i ≤ N .
As usual, S n ∈ R n×n demotes the subspace of symmetric matrices of size n×n and S N n = S n ×· · ·×S n . S N n is a finite dimenional real Hilbert space with respect to the inner product: for all X = (X(1), X(2), ..., X(N )), Y = (Y (1), Y (2), ..., Y (N )) ∈ S N n . Throughout the paper E [·] stands for the mathematical expectation and E [·|θ t = i] denotes the conditional expectation with respect to the event {θ t = i}.
2 Problem setting

Model description
Consider the following nonlinear difference equation on the space S N n : 1 ≤ i ≤ N , for all X = X(1), · · · , X(N ) ∈ S N n . In (2) M(t) = (M (t, 1), ...., M (t, N )) ∈ S N n , R(t) = (R(t, 1), ...., R(t, N )) ∈ S N m , L(t) = (L(t, 1), ..., L(t, N )) ∈ M N n,m . Regarding the coefficients of the equation (2) we make the assumption: Since the discrete-time backward nonlinear equation (2) contains as special cases a large number of discrete time Riccati type equations arising in diverse linear quadratic control problems in both the deterministic and stochastic frameworks, we shall call this type of equations generalized discrete time Riccati equations (GDTRE).
Remark 2.1 The GDTRE (2) is associated to a quadruple and P t ∈ R N ×N is the stochastic matrix satisfying the assumption H1).
The GDTRE (2) occurs in connection with a LQ control problem described by the controlled system and the quadratic performance criterion where x u (t) is the solution of the initial value problem (IVP) described by the controlled system (5), , · · · , w r (t)) T is a sequence of independent random vectors and the triple ({θ t } t≥0 , {P t } t≥0 , N) is a time non-homogeneous Markov chain defined on a given probability space (Ω, F, P) with the finite states set N = {1, · · · , N } and the sequence of transition probability matrices {P t } t≥0 . Regarding the processes {θ t } t≥0 and {w t } t≥0 , the following assumptions are made: is a sequence of independent random vectors with the following properties: I r being the identity matrix of size r.
The initial probability distributions π 0 = (π 0 (1), ..., π 0 (N )) satisfying the assumption H3) b) will be called admissible initial probability distributions of the Markov chain. For each t ≥ 0, we denote H t = F t ∨ G t , and for each t ≥ 1 we denoteH t = F t−1 ∨ G t andH 0 = σ(θ 0 ). Here, H t is the smallest σ-algebra containing the σ-algebras F t and G t andH t is the smallest σalgebra containing the σ-algebras F t−1 and G t , respectively. In the following, ℓ 2H {t 0 , τ ; R m } stands for the space of all finite sequences {y t } t 0 ≤t≤τ of m-dimensional random vectors with the properties that The norm induced by the usual inner product on each of these Hilbert spaces are: We define the class U ad (t 0 , x 0 ) of admissible controls which consists in all stochastic processes, u = {u(t)} t≥0 ∈ ℓ 2H {0, ∞; R m } with the property that x u (·) ∈ ℓ 2H {0, ∞; R n }.
On the other hand, according with the assumptions H1) b) and H3) b) one may prove inductively that π t 0 (i) :

The stabilizing solution of GDTRE
If F(t) = F (t, 1), · · · , F (t, N ) with F (t, i) ∈ R m×n , we introduce the following Lyapunov type operator L F (t) : S N n → S N n , defined by: is exponentially stable (ES), where L Fs (t) is defined as in (7) taking F s (t, i) instead of F (t, i) with: Remark 2.3 According with Definition 3.1 and Theorem 3.4 from [6] we obtain that under assumptions H1) b), H2) and H3) a), X s (·) is a stabilizing solution of the GDTRE (2) if and only if it satisfies (8) and additionally the system of discrete time linear stochastic equations is exponentially stable in mean square with conditioning of first kind (ESMS-CI) i.e. there exist β ≥ 1, α ∈ (0, 1) such that the solutions of this system satisfy It is worth pointing out that we do not know a priori neither an initial value nor a boundary value of a stabilizing solution of the GDTRE (2). That is why, it becomes an important problem to derive necessary and sufficient conditions or at least sufficient conditions that guarantee the existence of a stabilizing solution of this kind of Riccati equations. In the derivation of some conditions for the existence of a bounded and stabilizing solution of some GDTRE of type (2) an important role is played by the sign of the matrices Thus the problem of the existence and uniqueness of the bounded and stabilizing solution of a GDTRE of type (2) satisfying a sign condition of the form for all (t, i) ∈ Z + × N was studied in Chapter 5 from [6] in the finite dimensional case and in [15] in the infinite dimensional case. The problem of the existence and uniqueness of the bounded and stabilizing solution of a GDTRE of type (2) satisfying a sign condition of the form for all (t, i) ∈ Z + × N was studied especially with the stochastic H ∞ control problem in the so called Bounded Real Lemma (see e.g. [14,7] for the case of homogeneous Markov chain and [12,3,4] for the case of nonhomogeneous Markov chain). The goal of the present work is to provide a set of conditions that guarantee the existence and uniqueness of the bounded and stabilizing solution of a GDTRE of type (2) satisfying the sign condition: where I = diag{−I m 1 , I m 2 } and Inertia[·] denotes the inertia of a matrix. A condition of type (15) means that the matrix R(t, X s (t + 1), i) has m 1 negative eigenvalues and m 2 positive eigenvalues with Often we shall call GDTRE of type (2) with indefinite sign of quadratic part, the Riccati equations satisfying a sign condition of type (15). First, in the next section, we prove the uniqueness of the bounded and stabilizing solution of a GDTRE of type (2). Further, we shall study the problem of the existence of the bounded and stabilizing solution of this kind of discrete-time Riccati equations satisfying the sign condition (15). Finally, we shall show how one may use the bounded and stabilizing solution of a GDTRE of type (2) to construct an equilibrium strategy in a linear quadratic discretetime dynamic game.

Uniqueness conditions
Before stating and proving the main result of this section, we discuss several issues related to the discrete-time perturbed Lyapunov equations. First, let us remark that ifL(t) : D N d → S N d is a discrete-time Lyapunov type operator defined bŷ then its adjoint operator with respect to the inner product (1) is given bŷ for all X = (X(1), ..., X(N )) ∈ S N d . Employing Theorem 2.12 from [6] in the case of the operator (16) we obtain: is exponentially stable.
n → S N n , j = 1, 2 be the Lyapunov type operators described by we define the extended Lyapunov operator L e (t) : are exponentially stable then the discrete-time linear equation on S N 2n is exponentially stable too.
has the uniform positive and bounded solution given byZ(t) = (Z(t, 1), ...,Z(t, N )) wherẽ Applying the implication (ii) → (i) from Corollary 3.1 in the case of the equation we obtain the desired conclusion. Thus the proof ends.
We are now in position to prove the main result of this section.
Theorem 3.4 Assume that the assumption H1) is fulfilled. Under this condition the following hold: i) the (GDTRE) (2) has at most one bounded on Z + and stabilizing solution; ii) if there exists an integer p ≥ 1 such that for all t ∈ Z + , i, j ∈ N, then the unique bounded and stabilizing solution of (2) if any is a periodic sequence of period p.
Proof 3.5 i) Let us assume by contrary that the GDTRE (2) has two bounded and stabilizing solutions {X ℓ s (t)} t∈Z + , ℓ = 1, 2. Let F ℓ s (t, i) be the corresponding stabilizing feedback gain associated to the solution X ℓ s (·) via (10). By direct calculations one obtains that the equation (2) verified by X ℓ s (·) may be written in the form: where Q(t, i) were introduced by (4). Let Y(t) = (Y (t, 1), ..., Y (t, N )) be defined by for all (t, i) ∈ Z + × N. We set Based on this choice of the matricesÂ k (t, i) we construct the Lyapunov type operatorL e (t) and its adjoint operatorL * e (t) defined as in (16) and (17), respectively. On the other hand, let L F ℓ s (t) : S N n → S N n be the linear operators defined as in (7) for F(t) replaced by F ℓ s (t) = (F ℓ s (t, 1), ..., F ℓ s (t, N )). Since X ℓ s (·) is a stabilizing solution of (2) it follows that the discretetime linear equation is exponentially stable. Applying Lemma 3.2 we deduce that the discrete-time equation on S N 2n is exponentially stable. LetZ(t) = (Z(t, 1), ...,Z(t, )) be defined bỹ By direct calculation, involving (17), (20), (21) we obtain that {Z(t)} t≥0 is a bounded solution of a discrete-time backward equatioñ Applying Theorem 2.5 from [6] to the special case of the equation (22) we deduce that this equation has a unique bounded solution namelyZ(t) = 0 ∈ S N 2n , t ∈ Z + . This allows us to conclude that X 1 Thus we have shown that part i) from the statement holds.
We setX(t) = (X(t, 1), ...,X (t, N )) defined byX(t, i) = X s (t + p, i), t ∈ Z + . By direct calculation involving the periodicity of the coefficients one shows that {X(t)} t≥0 is a bounded solution of (2). Let LF(t) be the linear operator defined as in (7) for F(t) replaced byF(t) = (F (t, 1), ...,F (t, N )),F (t, i) being computed as in (10) for X s (t, i) replaced byX(t, i). Employing the periodicity property of the coefficients of (2) one obtains that n be the linear evolution operators defined by L Fs (·) and LF(·), respectively, where I S N n is the identity operator on S N n . Based on (23) we getT From the definition of the stabilizing solution is exponentially stable, which means thatX(·) is also a bounded and stabilizing solution of the GDTRE (2). From the uniqueness of the bounded and stabilizing solution of (2) we conclude that Thus the proof ends.
Remark 3.1 In the case when the bounded and stabilizing solution X s (·) of a GDTRE of type (2) satisfies a sign condition of type (13) or (14) one shows that in this case X s (·) is the bounded and maximal solution (bounded and minimal solution), respectively of the considered GDTRE, which guarantee the uniqueness of the bounded and stabilizing solution.
In the case when the bounded and stabilizing solution of a GDTRE of type (2) satisfies a sign condition of type (15) we cannot say if this solution is maximal or minimal and, therefore, in this case we cannot obtain the uniqueness of the bounded and stabilizing solution based on the uniqueness of the maximal or minimal solution of this kind of GDTRE. The proof of Theorem 3.4 from above is not based on the property of the sign of the matrices defined in (12).

Existence conditions
In this section we deal with the problem of the existence of the bounded and stabilizing solution of a GDTRE (2) satisfying a sign condition of type (15). We shall provide some conditions which guarantee the existence of the bounded and stabilizing solution of this kind of Riccati equations with the desired sign condition.

Preliminary remarks
Consider the following partitions of the coefficients of (2): and: Correspondingly we obtain the partitions: In the rest of the paper, the following assumption regarding the weight matrices M (t, i), R(t, i) and L(t, i) is made: with ρ j > 0, j = 1, 2 given constant scalars.
In this work, we are interested by solutions X(·) : I ⊂ Z + → S N n of (2) satisfying the following sign conditions: for all t ∈ I, 1 ≤ i ≤ N , δ k > 0, k = 1, 2, being constants.
be the partition of the matrix R(t, X(t + 1), i) compatible with the partition (25), R(t, X(t + 1), i) being computed as in (12) with X s (t + 1) being replaced by X(t + 1). So, the left part of (32) is the (2, 2)-block of the matrix R(t, X(t + 1), i), while the left part of (31) is the Schur complement of the (2, 2)-block of the same matrix.
In the sequel, we shall denote D R the set of solutions {X(t)} t∈I of the GDTRE (2) satisfying the conditions (31) and (32) and we call them admissible solutions. Employing Lemma 2 in Chapter 4 from [11], some useful properties of admissible solutions can be derived: Assume that the Assumptions H1) a) and H4) are fulfilled. LetX(t) = (X(t, 1), ...,X (t, N )), t ∈ Z + be a bounded solution of (2). IfX(·) ∈ D R then the matrices R(t,X(t + 1), i) satisfy a sign condition of type (15). IfX(·) and the coefficients of (2) are periodic sequences of period p and additionallyX(t, i) ≥ 0 for all (t, i) ∈ Z + × N, then the following are equivalent: ii) the matrices R(t,X(t + 1), i), (t, i) ∈ Z + × N satisfy a sign condition of type (15).
According to the result stated in Corollary 4.1, to derive conditions that guarantee the existence of a bounded and stabilizing solution of (2) satisfying a sign condition of type (15) we shall provide conditions that allows us to obtain the existence of a bounded and stabilizing solution of (2) satisfying sign conditions of type (31)-(32). We shall see that the conditions of type (31)-(32) are more adapted to the definition of some admissible strategies in a zero sum discrete-time LQ dynamic game.
Remark 4.2 In the sequel, we will also give conditions for the existence of a bounded and stabilizing solution of (2) satisfying sign conditions of type: together with (32), for all t ∈ I, 1 ≤ i ≤ N , δ 3 > 0 being a constant. Such solution is involved in the definition of another admissible strategy in a zero sum discrete-time LQ dynamic game. Such a strategy appears for example in [13] where further details could be found for the interested reader.
In order to obtain (34) we may take This factorization, adapted to the indefinite sign of R(t, i) + Π 3 (t)[X(t + 1)](i), will play a crucial role in the derivation of the main results.
We will now define and characterize the set A Σ that will play a key role in the proof of our main results. To this end, and according with the partition (25), we define Hence (5) and (6) is rewritten as follows: where x u 1 (t) is the solution of (35) corresponding to u 1 (t) and: To the above system (35) and the corresponding quadratic functional (36) we associate the following Riccati-type difference equation of type (2): where: for all X ∈ S N n . Since the Riccati equation (38) is of type (2) it follows that the notion of stabilizing solution of (38) may be introduced as in Definition 2.1. Furthermore, according to Theorem 3.4 the GDTRE (38) may have at most one bounded and stabilizing solution. In the following, to a GDTRE (2) defined by a given quadruple of sequences we associate the set A Σ , which consists of all pairs of feedback gains (K(·), i) The zero solution of the stochastic linear system: is ESMS-CI (see Definition 3.1 from [6] for details).
Remark 4.4 i) In order to obtain necessary and sufficient conditions that allow us to decide if the set A Σ is empty or not, one can apply Theorem 5.6 in [6] to the Riccati difference equation: obtained from (38) by taking Y (t, i) = −X(t, i), (t, i) ∈ Z + × N and noticing thatX(t) is the bounded and stabilizing solution of (38) satisfying the sign condition (41) if and only ifỸ(t) is the bounded and stabilizing solution of (42) satisfying the sign condition: Such a necessary and sufficient condition is formulated as a convex optimization problem via an LMIs (Linear Matrix Inequalities) setting.
ii) It follows from the exponential stability in the mean square sense with conditioning of system (40) and sign conditions (28) and (29) one can show that if (K(·), W(·)) ∈ A Σ then the bounded and stabilizing solution of (38) is positive semi-definite.
Among the subsets of A Σ we distinguish the subset A Σ 0 that consists of all feedback gains (K(·), W(·)) ∈ A Σ such that W (t, i) = 0 for all (t, i) ∈ Z + × N. The bounded and stabilizing solution of the GDTRE (38) with W (t, i) = 0 will be denoted byX K (t) = (X K (t, 1), · · · ,X K (t, N )). The sign condition (41) reduces in this case to: Note that Remark 4.4 could be specialized in a straightforward way to the set A Σ 0 and the corresponding stabilizing solutionX K (t) of the GDTRE (38). Hence by combining the sign condition (44) with (28) and (29) one shows easily thatX K (t) verifies the sign conditions (31) and (32).

Main results
Before introducing the main existence conditions, let us first start by stating and proving a comparison theorem for the considered class of generalized Riccati equations. Such a result shows the monotonicity of the admissible solutions (in the sense of conditions (31)-(32)) of the Riccati difference equation (2). This result will play a key role in the proof of the proposed existence conditions. Theorem 4.2 Let X 2 (·) : I 2 ⊂ Z + → S N n be an admissible solution of a Riccati difference equation of type (2) associated to the quadruple 2 n be a solution of (2) associated to the quadruple 1 that verifies the sign condition (32). A(t), B(t), P t are as in Remark 2.1 while: is an admissible solution it follows that it satisfies the sign conditions of type (31)-(32). Therefore, and taking into account Remark 4.3, the following factorization takes place: Appendix A, we deduce that the Riccati difference equation solved by X l (·), l = 1, 2, may be rewritten in the form: Taking: and employing (47) and (49)-(50) one obtains that: t → X 2 (t) − X 1 (t) solves the backward difference equation: where H(t) = H(t, 1), · · · , H(t, N ) with: The assumption Q 2 (t, i) ≥ Q 1 (t, i) together with (48) yields: Q 2 Γ (t, i) − Q 1 Γ (t, i) ≥ 0, t ∈ I 1 ∩ I 2 , 1 ≤ i ≤ N . Hence (52) allows us to conclude that H(t, i) ≥ 0, t ∈ I 1 ∩ I 2 , 1 ≤ i ≤ N . So, by induction, the proof is completed.
We are now in position to prove the following result. Under these assumptions, the following hold: (X(t, 1), · · · ,X(t, N )), t ∈ Z + being an arbitrary admissible globally defined positive semidefinite solution of (2).
ii) It follows immediately from Theorem 4.2.
iv) The proof of 0 ≤ X τ 1 (t, i) ≤ X τ 2 (t, i) is a direct consequence of the Comparison Theorem (Theorem 4.2). The existence of µ > 0 follows from Lemma C.3. (i) and the boundedness property ofX KW (·).
v) The existence of the limitX(t, i) = lim τ →∞ X τ (t, i) follows from iv) above. The minimality and admissibility properties are direct consequences of assertion ii) and iii).
vi) The periodicity property can be proved using the same steps as in in the proof of Theorem 1 from [2].
Let us now introduce the following auxiliary system: The main result of this section is given as follows: d) the auxiliary system (54) is stochastically detectable: then: i)X(·) defined in v) from Proposition 4.4 coincides with the unique admissible stabilizing solution X s (·) of (2).
In order to prove the above assertions, we need first to prove some auxiliary results.
Proof 4.8 Using Lemma A.1. in Appendix A with: one shows that the Riccati difference equation (2) can be rewritten as: for (t, i) ∈ Z + × N. After some algebraic manipulations, one shows that (58) can be rewritten as: is the solution of (56) with x(t 0 ) = x 0 . Using conditional expectation properties, one can show that: for all τ ≥ t 0 , whereL(t, θ t ) = L 2 (t, θ t ) + Γ T 2 (t, θ t )R 22 (t, θ t ). SinceX(·) is a semi-positive bounded solution we deduce that: for some positive constant c > 0. It follows from the detectability assumption that there exist gain matrices H(·) : Z + → M N n,p such that: is ESMS. Note also that (56) can be rewritten as: where: (t, i) ∈ Z + × N. Hence reasoning as in the proof of Corollary 3.9 in [6] one deduces that there exist a positive constant ν not depending upon (t, x 0 , i) ∈ Z + × R n × N such that: for all (t, x 0 , i) ∈ Z + × R n × N. Using the implication ii) → i) from Theorem 3.2 in [8], one obtains that system (56) is ESMS-CI. Using Theorem 3.6 from [6] one shows that under assumption c) from the statement of Theorem 4.6 from above, that the stability notions ESMS-CI and ESMS are equivalent. This ends the proof.   (10) with X s (·) being replaced byX(·). Then there exists c > 0 not depending upon t 0 , such that: whereΦ(t, t 0 ) is the fundamental random matrix solution of the system: Proof 4. 10 We shall show that for each i ∈ N we have: Assume by contrary that there exists i 0 ∈ N such that the supremum in (68) is not finite. Hence, following step by step the reasoning in the proof of Lemma 3 from [9] one shows that there exist sequences τ k , t k , τ k > t k and k 0 ≥ 1 such that: is the fundamental random matrix solution of the system: and x k ∈ R n , |x k | = 1. This leads to: The condition (71) written for the initial distributionπ 0 yields (using assumption c) from the statement of Theorem 4.6): Letũ τ k 1 (t) = F τ k 1 (t,θ t )x τ k (t), t ∈ [t k , τ k ], hence it follows from Lemma C.1. ii) that: for all k ≥ k 0 , i ∈ N and u 2 ∈ ℓ 2H {t k , τ k ; R m 2 }. Let (K(·), W(·)) ∈ A Σ be arbitrary but fixed and let u k 2 be defined by u k It follows from Proposition B.4. that: for some positive constants α 0 and α 1 which do not depend upon k. We deduce from the above developments that:

is in contradiction with the result given in Lemma C.3. Thus we have obtained that (66) is true for the Markov chain
for all t 0 ∈ Z + , i ∈ N, whereΦ(t, t 0 ) is the fundamental random matrix solution of the system obtained from (67) written forθ t instead of θ t . On the other hand from the representation theorem (Theorem 3.1 in [6]) and Remark 3.2 from [6] we deduce that for all t ≥ t 0 ≥ 0, i ∈ N and for arbitrary Markov chain ({θ t } t≥0 , {P t } t≥0 , N), where T * F (t, t 0 ) is the adjoint operator of the linear evolution operator on S N n defined by the discrete time linear equation (9) written forF (t, i) instead of F s (t, i) and H(t) = (H(t, 1), · · · , H(t, N )) with H(t, i) =F T 1 (t, i)F 1 (t, i). Now (77) and (78) alow us to conclude that (66) holds for any Markov chain ({θ t } t≥0 , {P t } t≥0 , N). Thus the proof is complete.
Remark 4.6 In [6] the representation formula (3.4) is proved for H(θ t ) and not for H(t, θ t ). One can show that an equality of type (3.4) from [6] takes place also when H(i) is replaced by H(t, i), t ≥ t 0 , i ∈ N. Indeed we chose t 1 ≥ t 0 arbitrary but fixed. Applying (3.4) from [6] we obtain for all t ≥ t 0 , i ∈ N. Particularly, (79) is also true for t = t 1 which leads to the desired equality.
Proof of Theorem 4.6 i) LetX(t, i) = lim τ →∞ X τ (t, i), (t, i) ∈ Z + × N. It follows from Proposition 4.4 thatX(·) is a bounded on Z + positive semi-definite solution of equation (2). Let us now show thatX(·) is a stabilizing solution of (2). First note that the system: (whereF (t, i) is defined as in (10) with X s (·) being replaced byX(·)) can be rewritten as: where: It follows from Lemma 4.9 that E Hence using the same reasoning as in the proof of Lemma 4.7 one shows thatX(·) is a stabilizing solution of (2). The unicity ofX(·) follows from Theorem 3.4.
ii) The proof is the same as for the proof of point vii) in Proposition 4.4.
This ends the proof.

Remark 4.7
i) The condition that A Σ is not empty is also necessary. Indeed, one can prove that if the Riccati equation (GRDE) has a positive semidefinite and stabilizing solution, then the set A Σ is not empty. To this end, one can takesK(t, i) andW (t, i) as defined in (105)-(106) and prove that (K(t, i),W (t, i)) ∈ A Σ .
ii) The stochastic detectability of the auxiliary system (54) may be tested by the solvability of a set of LMIs conditions (The reader can refer to [6] for example). This shows, in addition with remark 4.4 i), the numerical tractability of the proposed set of existence conditions thanks to the LMI formalism.

A zero-sum stochastic LQ difference game
In this section, we show the relation between the bounded and stabilizing solution of GDTRE (2) and the design of the feedback gains of an equilibrium strategy in the case of a zero-sum stochastic LQ difference game on infinite time horizon described by a controlled system of type (5) and a quadratic performance criterion of type (6). Let us first rewrite (5) and (6) In game theory terminology, the inputs u k : Z + → R m k are usually called strategies available for player k, k = 1, 2. These are functions with the property that u T 1 (·) u T 2 (·) T are in a subset of U ad . Such a subset of U ad will be called the set of admissible strategies. The shape of this set depends on the type of available informations for player k in order to compute the value of u k (t). In this section, we shall consider two different types of admissible strategies. Before detailing these two different strategies, we give here for the reader convenience a definition of an equilibrium strategy of the zero-sum difference game described above.
Roughly speaking, from (85) one sees that the aim of player 1 is to maximize the performance J (x 0 , ·, ·) while the goal of player 2 is to minimize the value of J (x 0 , ·, ·).

The full state-feedback case
In this case, the admissible strategies are constructed in a linear state-feedback form as follows: We denote F ad the set of the pairs (F 1 (·), F 2 (·)) where F k (·) = (F k (·, 1), · · · , F k (·, N )) with t → F k (t, i) : Z + → R m k ×n are bounded and matrix valued sequences with the property that the closedloop system: is ESMS. In order to implement a strategy of the form (86), we assume that each player has access to the current Markov state θ t and the current system state x(t). One can easily verify that the strategies of type (86) lie in U ad .
Remark 5.2 Note that conditions for the existence of a bounded and stabilizing solutionX(·) of GDTRE (2) satisfying the sign conditions (89) and (90) are given by Theorem 4.6.
One can see that the strategies of the form: are of type (99) if t → K(t, i) : Z + → R m 2 ×n , t → W (t, i) : Z + → R m 2 ×m 1 are bounded matrix valued sequences with the property that the linear system of type (40) is ESMS.
Following similar arguments as in the proof of Lemma 5.2 one can states the following Lemma which will be used in the derivation of the main result of this section.
The next result provides an equilibrium strategy of the zero-sum LQ differential game described by the system (83), the quadratic functional (84) and the class of admissible strategiesŨ ad .
Appendix A.
, t ∈ Z + are gain matrices such that Γ j (t, i) ∈ R m j ×n , j = 1, 2, then the Riccati difference equation (2) solved by X(·), may be rewritten in the form: is the adjoint of the Lyapunov type operator defined by: The proof can be done by direct calculation. The details are omitted.

Appendix B.
In this appendix, we will state and prove several preliminary results that relate the set A Σ to the GDTRE (38) and the functional (36) respectively.

Proof. Let
With these notations, we have thatX KW (·) verifies the sign conditions (31) and (32) if and only if G 22 (t, i) ≥ δ 2 I m 2 and: First, it follows from Remark 4.4 ii) that for any (K(·), W(·)) ∈ A Σ the unique bounded and stabilizing solution of the corresponding GDTRE (38) satisfies the sign condition G 22 (t, i) ≥ δ 2 I m 2 . Furthermore, the property (41) leads to: We set: One sees that (116) may be rewritten as: for all t ∈ Z + . Using (117), one can show by direct calculations that: Hence the proof is complete.
Appendix C.
In this Appendix, we will state and prove several technical results that will be used in the proof process for the existence conditions of the bounded and stabilizing solution of (2). These results rely partly on the material given in Appendix B. Let: (147) Let t 0 ∈ I(τ ), t 0 ≤ τ be fixed and x Fτ (t, t 0 , x 0 ) be the solution of the initial value problem (IVP): and: ũ τ 1 (t) = F τ 1 (t, θ t )x Fτ (t, t 0 , x 0 ) u τ 2 (t) = F τ 2 (t, θ t )x Fτ (t, t 0 , x 0 ).
Now using Lemma A.1 in Appendix A with: one shows that the Riccati difference equation (2) can be rewritten as: The conclusion follows by induction using assumption H4), (165) and (167).