{'fraction_non_alphanumeric': 0.05413913584944823, 'fraction_numerical': 0.01598228802712146, 'mean_word_length': 4.173228346456693, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Escaping High-order Saddles in Policy Optimization for Linear Quadratic Gaussian (LQG) Control

First order policy optimization has been widely used in reinforcement learning. It guarantees to find the optimal policy for the state-feedback linear quadratic regulator (LQR). However, the performance of policy optimization remains unclear for the linear quadratic Gaussian (LQG) control where the LQG cost has spurious suboptimal stationary points. In this paper, we introduce a novel perturbed policy gradient (PGD) method to escape a large class of bad stationary points (including high-order saddles). In particular, based on the specific structure of LQG, we introduce a novel reparameterization procedure which converts the iterate from a high-order saddle to a strict saddle, from which standard random perturbations in PGD can escape efficiently. We further characterize the high-order saddles that can be escaped by our algorithm.


I. INTRODUCTION
In this paper, we revisit the linear quadratic Gaussian (LQG) control, one of the most fundamental problems in control theory, from a modern optimization view. In brief, we focus on a continuous-time linear timeinvariant (LTI) systeṁ where x(t) ∈ R n , u(t) ∈ R m , y(t) ∈ R p are the state, control input, and measurement (output) vector at time t, respectively, and w(t), v(t) are white Gaussian noises with intensity matrices W 0 and V ≻ 0, respectively. The goal is to design a controller (i.e., policy) based on partial measurements y(t) to minimize a quadratic cost x T Qx + u T Ru dt . (2) A special case is the linear quadratic regulator (LQR) [1], where we have direct access to the state x (i.e., y(t) = x(t), v(t) = 0, ∀t ∈ R in (1)). It is known that the optimal policy for the LQR is in the form of static state feedback u(t) = Kx(t), where K ∈ R m×n is a constant matrix that can be obtained by solving a Riccati equation [2]. On the other hand, when the state is not directly observed, the policy that minimizes (2) is a dynamical controller of the form [3] ξ(t) = A K ξ(t) + B K y(t), where the optimal parameters K * := (A * K , B * K , C * K ) can be obtained by solving two Riccati equations [3], [4] (see Section II-A). Algorithms for solving Riccati equations are well-studied, including iterative algorithms [5], algebraic solution methods [6], and semidefinite optimization [7]. All these methods are model-based and explicitly rely on the system model (1). Recently, policy gradient methods have achieved impressive results for many challenging problems [8]. These methods directly optimize the quadratic cost (2) as a function of the policy class K = (A K , B K , C K ) via gradient descent or its variants. They can be further made model-free, bypassing an explicit estimation of the model (1). The flexibility of model-free control has stimulated a growing interest in investigating foundations of policy gradient methods for classical control problems [9]- [20].
While it is guaranteed to obtain the optimal controller for LQR or LQG via classical model-based methods, such optimality guarantee is more difficult when using policy gradient methods since the cost (2) is typically nonconvex in the policy space. For LQR, recent work has shown that although the LQR cost is nonconvex, it is gradient dominant and coersive, and has a unique stationary point under very mild conditions, rendering the convergence of policy gradient methods to the globally optimal controller [9]- [12]. On the other hand, the LQG cost is neither gradient dominant nor coersive, and there may exist spurious saddle points [20], making it challenging for policy gradient to find the optimal controller.
Saddle points do not always destroy the performance of policy gradient methods. Suitable perturbed policy gradient methods are able to escape strict saddle points whose Hessian has at least one strictly negative eigenvalue [21]- [23]. However, it is shown in [20,Theorem 4.2] that the Hessian of the LQG cost at a saddle point can even degenerate to zero. We denote the saddle point whose Hessian does not give escaping directions as a high-order saddle point. Perturbed policy gradient methods [21]- [23] may thus get stuck and take an exponential number of iterations to escape high-order saddle points.
All the (strict or high-order) saddle points of LQG discussed in [20] are due to a loss of controllability and/or observability for the controller (A K , B K , C K ) in (3) (i.e., non-minimal controllers). Indeed, any stationary point corresponds to a full-order minimal controller cannot be saddle and it is instead globally optimal [20]. Further, many intrigue landscape properties of LQG are brought by a classical notion of similarity transformations that induces a symmetry structure [20]. In this paper, we raise a natural question of whether this induced symmetry structure allows us to reveal more information about high-order saddles of LQG such that suitable perturbed policy gradient methods can escape those points. We provide a positive answer to this question.
In particular, we first show that any stationary point after model reduction remains to be stationary. This gives a classification of the stationary points: all bad (suboptimal or saddle) stationary points after model reduction become lower-order and form new stationary points with the same LQG cost. We then reveal an intriguing transfer function G(s) at any stationary point is not identically zero, we can perturb (A K , B K , C K ) to get a new stationary point with the same LQG cost, which is a strict saddle with probability one. Standard perturbed policy gradient (PGD) methods [21], [22] can thus escape this new strict saddle. We emphasize that our PGD method include perturbations on two parts: 1) a novel structural perturbation on the stationary point (A K , B K , C K ); 2) a standard random perturbation on gradients [22]. This combination enables escaping a large class of bad stationary points (including high-order saddles) in LQG.
The rest of this paper is organized as follows. We present the problem statement in Section II. Our main results on characterizing stationary points and Hessians are presented in Section III. Section IV shows empirical performance of our perturbed policy gradient method. We conclude the paper in Section V. Technical proofs and auxiliary computations are postponed to the appendix.

A. Review of LQG control
The classical LQG control problem is defined as where J(u) is defined in (2) with Q 0 and R ≻ 0. In (4), the input u(t) depends on all past observation y(τ ) with τ < t. We make the following standard assumption. The optimal solution to (4) is a dynamical controller in the form of (3), in which ξ(t) ∈ R q is the controller internal state, and A K ∈ R q×q , B K ∈ R q×p , C K ∈ R m×q specify the dynamics of the controller. While q can be any positive number, one does not have to use q > n and the optimal controller has q = n, given by algebraic Riccati equations (AREs) [3,Thm. 14.7]. Precisely, let P, S be the unique positive semidefinite solutions to the following AREs Then, the parameters of an optimal controller to (4) are is not unique in the state-space domain. Any similarity transformation leads to another equivalent optimal controller (they correspond to the same transfer function in the frequency domain).

B. Problem Statement
In this paper, we embrace the spirit of [9]- [12], [20] and view the LQG problem (4) from a modern optimization perspective. We consider the policy class (A K , B K , C K ) in (3), and the closed-loop matrix becomes The set of internally stabilizing policies [3,Chapter 13] is Let J q (K) : C q → R denote the corresponding LQG cost (2) for each stabilizing policy in C q . It is known [20,Lemmas 2.3 & 2.4] that this function J q (K) is real analytic on C q and admits efficient computation.
where X K and Y K are the unique positive semidefinite solutions to the following Lyapunov equations where A cl is defined in (7) and Lemma 1 works for stabilizing controllers of any order q. In this paper, we are mainly interested in characterizing the full-order case C n . Now, given the state dimension n, we can formulate the LQG problem (4) into a constrained optimization problem An important notion of dynamical controllers is mini- is controllable and (C K , A K ) is observable. As revealed in [20], the optimization landscape of (10) is more complicated than that of LQR: 1) the feasible region C n can have at most two disconnected components; 2) the cost function J n (K) is not coersive and not gradient dominant, and it can have suboptimal saddle points [20,Theorem 4.2]. Two nice features are 1) all sub-optimal saddle points correspond to non-minimal controllers and 2) all stationary points that correspond to minimal controllers in C n are globally optimal [20,Theorem 4.3].
Naive policy gradient methods can thus get stuck around sub-optimal saddle points. In this paper, we aim to provide further landscape characterizations of (10) and introduce a perturbed policy gradient method to escape bad stationary points of (10). In particular, we first show that any stationary point of the LQG problem (10) after model reduction remain to be stationary, and then characterize the second-order behavior of J n (K) on a non-minimal stationary point. This motivates the design of our perturbed policy gradient method.

III. STATIONARY POINTS AND THEIR HESSIANS
The LQG problem (4) has an inherent symmetry structure induced by the notion of similarity transformation. Let GL q denote the set of q × q invertible matrices. Given q ≥ 1 such that C q = ∅, the following map T q : GL q × C q → C q represents similarity transformations It is well-known that similarity transformations do not change the behavior of dynamical controllers and thus the LQG cost (2) is invariant with respect to T q (T, K), i.e., we have J q (K) = J q (T q (T, K)) , ∀K ∈ C q , T ∈ GL q .

A. Classification of stationary points
The symmetry via similarity transformations brings rich and complicated landscape properties of (4). Here, we show that the underlying symmetry also allows a classification of stationary points of LQG (4). The lemma below gives an explicit relationship among the gradients of J q (K) at K and T q (T, K).
As expected, a direct consequence of Lemma 2 is that a stationary point K of J q remains to be stationary over C q after any similarity transformation. We can further derive a classification of the stationary points of J n over the set of full-order controllers C n . (4), and letK = 0Ĉ K B KÂK ∈ C q be a minimal realization of K, where q ≤ n is the order of its minimal realization. Then, the following dynamical controller with any stable matrix Λ ∈ R (n−q)×(n−q) is a stationary point of (10). If q = n (i.e. K itself is minimal), then K is globally optimal.
The fact ofK (13) being stationary of J n (K) seems to be expected, sinceK and K correspond to the same transfer function in the frequency domain and K is in a higher dimensional space C n than C q . The technical proof is not difficult, which combines the classical Kalman decomposition with Lemma 2 and a result in [20,Theorem 4.1]. We provide the details in the appendix. The second part that K is globally optimal if q = n has been proved in [20,Theorem 4.3].
Theorem 1 shows all stationary points that correspond to non-minimal controllers admit a standard parameterization as we defined in (13), which splits the controller state ξ ∈ R n into 1) the controllable/observable (associated withÂ K ,B K ,Ĉ K blocks) part and 2) noncontrollable/non-observable (associated with Λ) part. Furthermore, Theorem 1 indicates that all bad stationary points of (4) after model reduction are in the same form of (13). Thus, policy gradient methods only need to escape those bad saddle points of the form (13). This motivates our results in the next section.
Remark 1 (Non-minimal globally optimal controllers). Note that a controller in the form of (13) might still be globally optimal to (4); See Example 2 below. This happens when the solutions is not observable or both. We conjecture that a random LQG instance should have (6) being minimal with probability one. An exact characterization is left for future work.

B. Hessian of stationary points
Once a policy gradient method reaches a stationary point, if the stationary point corresponds to a minimal controller, it has found a globally optimal solution to (4). If the stationary point does not correspond to a minimal controller, we can bring it into the form of (13), for which we have the following characterization of its hessian.
be the unique positive semidefinite solutions to the Lyapunov equations (9a) and (9b) witĥ where A cl is defined in (7) with theK above, and C cl : The following statements hold. (14) is globally optimal in C n , then the function (15) is not a zero function, thenK is a strict saddle point (the Hessian of J n (K) atK is indefinite) with probability one when randomly choosing a stable and symmetric Λ ∈ S n−q . 3) Let Z be the set of zeros of G(s), i.e.,Z = {s ∈ C | G(s) = 0} . Given a stable and symmetric Λ ∈ S n−q , let eig(−Λ) denote the set of (distinct) eigenvalues of −Λ. If eig(−Λ) Z, then the Hessian of J n (K) atK is indefinite.

1) IfK in
Proof. Statements 1) and 2) are direct consequences of Statement 3). We give simple arguments below. (14) is globally optimal in C n , then the Hessian of J n (K) atK must be positive semidefinite. If G(s) is not identically zero, then its zero set Z is a set of finite points due to the fundamental theorem of algebra 1 . Then, there exists a symmetric Λ ∈ S n−q such that eig(−Λ) Z, and thus its Hessian atK is indefinite. This is contradicted withK being globally optimal.

3) ⇒ 1): IfK in
3) ⇒ 2): If G(s) is not an identically zero function, then its zero set Z is a set of finite points. When choosing a stable and symmetric Λ ∈ S n−q randomly, we have eig(−Λ) Z holds with probability one. Thus, K is a strict saddle point with probability one.
The proof of Statement 3) exploits the bilinear property of the Hessian and the non-controllable/nonobservable property to identify a two-by-two hessian block in which the diagonal entries are always zero. Using the Hessian calculation in [20,Lemma 4.3], we then prove that if eig(−Λ) Z, then the off-diagonal entries are non-zero. The Hessian of J n (·) atK is thus indefinite. Details are presented in the appendix.
Our Theorem 2 includes the recent result [20, Theorem 4.2] as a special case in which the authors only consider a zero controller K = 0. Our main proof in the appendix, however, is motivated by that in [20,Theorem 4.2] with more complicated and careful calculations.
If the transfer function G(s) is not identically zero, thenK in (14) is a strict saddle point with probability one when randomly choosing Λ. Thus, we can apply the perturbed policy gradient method for "escaping saddle" [22], so that the policy gradient iterations do not get stuck around these sub-optimal saddle points. We note that when G(s) is not identically zero,K in (14) may still have a zero Hessian (i.e., high-order saddle) if Λ is chosen such that eig(−Λ) ⊆ Z; an explicit example is given Example 3 below. Therefore, our proposed perturbed policy gradient method for the LQG problem (4) includes perturbations on Λ as well as on the gradients. More details are given in Section IV. Remark 2 (Sufficiency of G(s) ≡ 0 for global optimality and its interpretation). Theorem 2 holds with q = n, so G(s) ≡ 0, ∀s ∈ C is also true when K comes from the Riccati equations. In this case, we expect that G(s) in (15) should have a nice control-theoretic interpretation. It is interesting to further investigate whether G(s) ≡ 0, ∀s ∈ C is sufficient (or some other suitable conditions are needed) to certify the global optimality ofK.

Thus, we have
This result that G(s) being identically zero is expected from Theorem 2 since K ⋆ is globally optimal.
We then consider [20,Example 7] for which the globally optimal LQG controller is non-minimal in C n .

Example 2. Consider an LQG instance with matrices
The globally optimal controller from (6) is given by It is easy to verify that (C * K , A * K ) is not observable. The Hessian of J 2 (K) at K ⋆ ∈ C 2 is positive semidefinite with eigenvalues as λ 1 = 581.5529, λ 2 = 7.1879, λ 3 = 0.2592, λ 4 = · · · = λ 8 = 0. (See Appendix D for details). Four zero eigenvalues are expected, due to the symmetry by similarity transformations, and the other zero is caused by the unobservablility of (C * K , A * K ). Consider two reduced-order controllers both of which are globally optimal. Thus, the following two full-order controllers are globally optimal as well. From Theorem 2, we expect G(s) ≡ 0 for bothK 1 andK 2 . For both of them, we can compute (details are in Appendix D) that Thus, we have the expected result from Theorem 2 that   (15) is .
IV. PERTURBED POLICY GRADIENT METHOD Inspired by Theorems 1 and 2, we introduce a novel perturbed policy gradient method that combines a structural perturbation on Λ in (14) with a standard perturbation on gradients [21], [22]. Numerical results confirm that our perturbed policy gradient method can escape high-order saddles more efficiently, than either vanilla policy gradient or standard perturbed policy gradient [22].

A. Algorithm setup
Recent work has established that variations of gradient descent can escape strict saddle-points -points at which the minimum eigenvalue of the Hessian is strictly negative. For example, stochastic gradient descent [21], gradient descent with appropriate random perturbation [22] or with cubic regularization sub-oracle [23] are proven to escape strict saddles and visit an approximate local minimum in polynomial time with high probability.
Our method combines the standard perturbed gradient descent [22,Algorithm 2] with an additional oracle of random structural perturbation on Λ. Our perturbed policy gradient descent is listed in Algorithm 1. We note that Algorithm 1 is a prototype algorithm in the sense that some quantities (e.g., order-reduction, gradient and Hessian Lipschitz constants) of the LQG problem require more investigations. Convergence conditions and further quantitative analysis of our algorithm are also left for future work.
The high-level ideas are described below. • When the gradient of a controller K t is close to zero, we check whether it is minimal, i.e., the smallest Hankel singular value of the controllability/observability matrix is bounded away from zero (for the connection with Hankel singular values and controllability/observability, please refer [3, Chapter 7]).
• If K t is controllable and observable (i.e., minimal), it is close to be globally optimal by Theorem 1. We terminate the algorithm.
• If K t is non-minimal, we perform a minimal realization (e.g., Kalman decomposition or balance realization) to get a controller in the form of (13).
• We then choose a symmetric and stable Λ randomly. From Theorem 2, we expect that the resulting controller is close to a strict saddle point.
• We apply a random perturbation on the gradients. The random perturbation is i.i.d. Gaussian variables, with small magnitudes, added to each entry of A K , B K , C K such that the controller is still stabilizing. We run a few gradient descent iterations afterwards. We expect that these gradient descent iterations will escape from the strict saddle point.
• We terminate the algorithm when the algorithm reaches the predefined number of steps T .

Algorithm 1 Perturbed policy gradient
Require: 1) Loss J(K) with its gradient. 2) Thresholds g th , ι. 3) Constant T , τ , step size η. 4) Function λ Han,min (K) that returns the minimum singular value of the Hankel matrix of K. 5) Function reduce_order(K) that finds the approximate order of K. 1: Set t = 0, t perturb = −τ −1 and initialize a stabilizing controller K 0 . 2: while t ≤ T do 3: if ∇J(K t ) ≤ g th and λ Han,min (K t ) ≥ ι then else if ∇J(K t ) ≤ g th and λ Han,min (K t ) ≤ ι and t − t perturb > τ then 6:K t , q t ← reduce_order(K t ) where q t is the order after model reduction; 7: Λ t ← λI n−qt with λ < 0 randomly selected; 8: (14) (Theorem 2); 9: K t ← K t + ξ t with ξ t uniformly sampled from B Kt (r); 13: t ← t + 1; 14: end while Algorithm 1 can escape a large class of (but not all) high-order saddles at which G(s) in (15) is not identically zero. When Algorithm 1 terminates, it is likely to produce an approximately global minimum or return a point at which the transfer function G(s) in (15) is close to zero. In the later case, the point may not be globally optimal, and this is related to the sufficiency of G(s) ≡ 0 for global optimality in Remark 2.

B. Numerical results
We implement Algorithm 1, and consider Example 3 for numerical comparison with three other algorithms: 1) Vanilla policy gradient; 2) Standard perturbed policy gradient [22] (with no perturbation on dynamics Λ, i.e., no Lines 6-8 in Algorithm 1); 3) Perturb the dynamics Λ but with no perturbation on gradients (i.e., no Line 9 in Algorithm 1.). The globally optimal controller from (6) for the LQG instance in Example 3 is As discussed in Example 3, this initial point is close to a high-order saddle A K = −0.5I 2 , B K = 0, C K = 0. We add a perturbation to the first iteration and run gradient descent with the fixed step size. The perturbations are different, as discussed at the beginning of this section. The results are shown in Figure 1: the left sub-figure shows the suboptimality gap, and the right one shows the norm of graidents at each iteration. Our Algorithm 1 implements both perturbations: 1) identifying an onedimensional Λ as in the standard form (14) and change it randomly, and 2) randomly perturb all variables with a small quantity 0.01. As shown in Figure 1, our Algorithm 1 can escape this high-order saddle faster than the other three algorithms, including standard PGD in [22] (in which no perturbation on Λ was applied).

V. CONCLUSIONS
We have proposed a novel PGD algorithm (cf. Algorithm 1) to escape high-order saddles of LQG. Our PGD algorithm combines the inherent structure of LQG control with standard perturbation on gradients. We have shown the structure of all stationary points after model reduction (cf. Theorem 1). We have also introduced a reparameterization procedure with an intriguing transfer function G(s) at any stationary point (cf. Theorem 2). If G(s) ≡ 0, we can certify that the high-order saddle can be made as a strict saddle by the reparameterization. Numerical simulations confirmed that Algorithm 1 combining the reparameterization with random perturbation on gradients can accelerate the speed of escaping high-order saddles. Ongoing and future directions include quantitative analysis of Algorithm 1. We are also interested in the sufficiency of G(s) ≡ 0 (or other conditions are needed) for global optimality of LQG (see Remark 2).

APPENDIX
In this appendix, we complete the proofs to Theorems 1 and 2, and present some further details on the Hessian computations in Examples 1 to 3.

A. Proof of Theorem 1
Our proof for Theorem 1 is built on the classical Kalman decomposition for linear time-invariant (LTI) systems and a recent result in [20,Theorem 4.1]. We first recall the celebrated Kalman canonical decomposition.
Theorem 3 (Kalman decomposition [3, Theorem 3.10]). Given any LTI system of the forṁ there exists a nonsingular coordinate transformationx = T x such that where the vectorx co is controllable and observable,x cō is controllable but unobservable,xc o is observable but uncontrollable, andxcō is uncontrollable and unobservable. Moreover, the transfer matrix from u to y is given by The dimension ofx co is the same as the dimension of the minimal realization of G(s). The result in [20,Theorem 4.1] states that any stationary point of J q can be augmented to be a stationary point of J q+q ′ for any q ′ > 0 over C q+q ′ with the same LQG cost.
Proof of Theorem 1: Let K = 0 C K B K A K ∈ C n be a stationary point of LQG (4). According to Theorem 3, we can perform a Kalman decomposition using a similarity transformation T , i.e., where the system (Ã K ,B K ,C K , 0) is in the standard Kalman decomposition of the form (19). By Lemma 2, we further have Therefore, the new pointK in the Kalman decomposition remains to be a stationary point of J n in C n . This is expected sinceK and K corresponds to the same transfer function.
Denote the controllable and observable components inK = 0C K B KÃK ∈ C n asK co = 0C K,cõ B K,coÃK,co ∈ C q , where q is the dimension of controllable and observable states (i.e., the dimension of its minimal realization). Since K is a stationary point of J n (K) over C n , the directional derivatives of J n (K) atK co are all zero. Therefore, it is not difficult to see thatK co is a stationary point of J q (K) over C q . Now, given an arbitrary minimal realization of K asK = 0Ĉ K B KÂK ∈ C q , there exists a unique similarity transformation T q ∈ R q×q [3, Theorem 3.17] such that Lemma 2 ensures that this minimal realizationK ∈ C q is a stationary point of J q (K) sinceK co is stationary. Applying Theorem 4 with q ′ = n − q leads to the desired result in (13). The statement that K is globally optimal when q = n has been shown in [20,Theorem 4.3] where the authors have shown that all minimal full-order stationary points are in the form (6) up to a similarity transformation. This completes the proof.

B. Proof of Theorem 2
Before presenting the proof, we recall a few definitions. We define a linear space Let K be any controller in C n ⊂ V n , and we use Hess K : V n × V n → R to denote the bilinear form of the Hessian of J n at K, so that for any ∆ ∈ V n , we have as ∆ F → 0. Obviously, Hess K is symmetric in the sense that Hess K (x, y) = Hess K (y, x) for all x, y ∈ V n . We recall [20,Lemma 4.3] that presents the hessian calculations of the LQG problem (4) by solving three Lyapunov equations.
Hess K (∆, ∆) = 2 tr 2 0 where X K and Y K are the solutions to the Lyapunov equations (9a) and (9b), and X ′ K,∆ ∈ R (n+q)×(n+q) is the solution to the following Lyapunov equation with We are now ready to prove the main technical result in Theorem 2. Our proof is motivated by that of [20,Theorem 4.2] with more complicated and careful computation. The main idea is to exploit the bilinear property of the Hessian and the non-controllable/non-observable property to identify a two-by-two hessian block HessK(∆ (1) , ∆ (1) ) HessK(∆ (1) , ∆ (2) ) HessK(∆ (1) , ∆ (2) ) HessK(∆ (2) , ∆ (2) ) ∈ S 2 in which the diagonal entries are always zero. Using the Hessian calculation in Lemma 3, we then prove that if eig(−Λ) Z, then the off-diagonal entries are non-zero. The Hessian of J n (K) atK is thus indefinite.
If there exists a direction ∆ = such that HessK(∆, ∆) = 0, then HessK must be indefinite since there is a two-by-two block which has zero diagonal entries and non-zero off-diagonal entries. The rest of the proof is show that if Λ is symmetric and eig(−Λ) Z with Z defined in Theorem 2, we can indeed find a direction ∆ such that HessK(∆, ∆) = 0. Computation of HessK(∆, ∆): Consider the controllerK and a direction∆ as where Λ is an (n − q) × (n − q) stable symmetric matrix.
We let XK and YK be the unique positive semidefinite solutions to the Lyapunov equations Note that (24a) reads as Then, the solutions XK and YK to the Lyapunov equations (24a) and (24b) are in the form of is the unique positive semidefinite solution to the left-upper Lyapunov equation in (25), i.e., which is the same as (9a), and Y op ∈ S n+q + is the unique positive semidefinite solution to (9b).
We next use Lemma 3 to compute the hessian HessK(∆,∆). For this computation, we note that Therefore, by Lemma 3, we can see that where X ′ K,∆ is the solution to the following Lyapunov equation In (27), the matrix M 1 (XK,∆) is defined as where we have used the definition in (16), i.e., For notational convenience, we define Then, the solution to the Lyapunov equation (27) becomes Now we have For the second part in (26), we have Then, we have where we have definedC

Substituting (29) and (30) into (26) leads to
Let λ ∈ eig(−Λ)\Z. Since λ / ∈ Z, there exists some i, j such that where e (p) i ∈ R p is the i-th element of the standard basis in R p and e (m) j ∈ R m the j-th element of the standard basis in R m .
We consider a symmetric Λ ∈ S n−q , then it can be diagonalized with real eigenvalues We then choose a special direction∆ leading to which is not zero by (32). We thus have an indefinite 2-by-2 hessian block in (23). This completes the proof.
We have shown the following two non-minimal full-order controllers are also globally optimal to the LQG problem in Example 2. The Hessian at these two controllers are Then, we can compute (CX op + VB T K )(sI − A T cl ) −1 Y opB = 26.5s + 56.5 (s + 1) 2 26.5s + 56.5 (s + 1) 2 .