{'fraction_non_alphanumeric': 0.039824229944594225, 'fraction_numerical': 0.010232024964442652, 'mean_word_length': 4.3392270202878835, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Lyapunov stabilizability of controlled diffusions via a superoptimality principle for viscosity solutions

We prove optimality principles for semicontinuous bounded viscosity solutions of Hamilton-Jacobi-Bellman equations. In particular we provide a representation formula for viscosity supersolutions as value functions of suitable obstacle control problems. This result is applied to extend the Lyapunov direct method for stability to controlled Ito stochastic differential equations. We define the appropriate concept of Lyapunov function to study the stochastic open loop stabilizability in probability and the local and global asymptotic stabilizability (or asymptotic controllability). Finally we illustrate the theory with some examples.


Introduction
We consider a N -dimensional stochastic differential equation where W t is a standard M -dimensional Brownian motion. Since the 60th, it was developed a stochastic Lyapunov method for the analysis of the qualitative properties of the solutions of stochastic differential equations, in analogy to the deterministic Lyapunov method. The main contributions are due to Hasminskii (see the monograph [21] and references therein) and Kushner (see the monograph [25] and [27]). They introduced the notion of stability in probability and asymptotic stability in probability. This means that the probability the trajectory leaves a given neighborhood of the equilibrium is decreasing to 0 as the initial data is approaching the equilibrium. If, moreover, the trajectory is asymptotically approaching the equilibrium with probability decreasing to 0 as the initial data is approaching the equilibrium the system is asymptotically stable in probability. Finally if for every initial data the trajectory is asymptotically approaching the equilibrium almost surely, the system is asymptotically stable in the large. The stochastic analog of deterministic Lyapunov functions V are twice differentiable continuous functions, which are positive definite and proper and satisfy the infinitesimal decrease condition with l ≥ 0 for mere Lyapunov stability and l > 0 for x = 0 for asymptotic stability, where a := σσ T /2. By the Dynkin formula, this differential inequality implies that the stochastic process V (X t ), where X t is the solution of the stochastic differential equation starting from x, is a positive supermartingale. This fact translates, in the stochastic setting, the requirement on the Lyapunov function to decrease along the trajectories of the dynamical system. There is a large literature on this kind of stochastic stability: we refer to the cited monographs and to [30], see also references therein. We recall here also the work of Florchinger [19,20] and Deng, Krstić, and Williams [14] on feedback stabilization for controlled stochastic differential equations by the Lyapunov function method.
In this paper we extend the Lyapunov method for stochastic differential equations essentially in two directions. First of all we consider controlled stochastic differential equations in IR N dX t = f (X t , α t )dt + σ(X t , α t )dW t , moreover we allow the Lyapunov functions to be merely lower semicontinuous. The nonexistence of smooth Lyapunov functions is well known in the deterministic case, see [2] for stable uncontrolled systems and the surveys [35,2] for asymptotically stable controlled systems. Also in the stochastic case, the assumption of smoothness for Lyapunov functions is not necessary and would limit considerably the applicability of the theory and the possibility of getting a complete Lyapunov-characterization of the stabilizability in probability by means of a converse theorem. Kushner proved in [26] a characterization of asymptotic uniform stochastic stability (for uncontrolled systems) by means of only continuous Lyapunov functions (here, however, the infinitesimal decrease condition is not given with a differential inequality but in terms of the weak generator of the process). For stability in probability, Hasminskii provided a C 2 Lyapunov function under the assumption of strict nondegeneracy of the diffusion: this result cannot be extended to possibly non-degenerate diffusions. Converse theorems in the controlled case will appear in the Ph.D. thesis by the author [10]. In particular we prove that the existence of a local Lyapunov function is also necessary for the stability in probability. Hence we show that if the system (CSDE) is uniformly asymptotically stabilizable in probability then there exists a local strict Lyapunov function, which is continuous.
We define then a Lyapunov function for the stability in probability as a lower semicontinuous, positive definite, proper function V , continuous at 0 and satisfying in viscosity sense the differential Hamilton-Jacobi-Bellman inequality This means that if there exists a Lyapunov function, then for every initial data we can find an admissible control which keeps the whole trajectory near the equilibrium with probability decreasing to zero as the starting point of the trajectory approaches the equilibrium. Moreover, the existence of a strict Lyapunov function implies also that, for every ε > 0, there exists an admissible control driving the trajectory asymptotically to the equilibrium with probability greater than 1 − ε. The same proof provides the global versions as well: if V satisfies (2) in IR N \ {0} then (CSDE) is also (open loop) Lagrange stabilizable, i.e. has the property of uniform boundedness of trajectories, and if V is strict then the system is (open loop) asymptotically stabilizable in the large. We also give sufficient conditions for the stability of viable (controlled invariant) sets more general than an equilibrium point.
The main tool to provide such a result is a superoptimality principle for lower semicontinuous bounded viscosity supersolutions V of the Hamilton-Jacobi-Bellman equation (2). The new point is that this superoptimality principle holds as an equality and then gives a representation of such V as value functions of an appropriate obstacle control problem. A similar approach has been exploited in the deterministic case by Barron and Jensen (see [9]) for globally asymptotically stable systems affected by disturbances and by Soravia ([38], [37]) for stable systems with competitive controls.
Precisely we prove that every bounded LSC viscosity supersolution V of (2) can be represented as This representation formula is important on its own, since it refers to Hamilton-Jacobi-Bellman equations for which it is not expected uniqueness of solutions. In particular this formula permits to give a characterization of the minimal nonnegative LSC viscosity supersolution of the equation (2) as the value function The representation formula (3) is obtained by introducing a suitable sequences of obstacle problems, solved in the viscosity sense by V . The conclusion comes from a uniqueness results for viscosity solutions of such problems and from an approximation procedure. The delicate point is the proof of a suboptimality principle for the min-max value function (on the right of equality (3)). To get this principle, we have to choose appropriately the controls we are allowing for our problem. Actually the class of controls on which we minimize a certain given functional has to satisfy two key properties in order to get a dynamic programming principle. They are the stability under concatenation and stability under measurable selection. For the definition of the classes of controls and for the compactness and measurable selection results we are going to use, we refer mainly to the article by Haussmann and Lepeltier [22] (see also the article by El Karoui and others [16] and the book by Stroock and Varadhan [40, ch 12]). For related results on the existence of optimal controls for stochastic problems we refer to the article by Kushner [28].
There is a large literature on dynamic programming and superoptimality and suboptimality principles for viscosity solutions of second order Hamilton-Jacobi-Bellman equations, starting from the papers by P.L.Lions [29] (see also the book [17]). We recall here the recent work by Soner and Touzi on dynamic programming for stochastic target problems ( [32], [33]). We refer also to the paper by Swiech [41] on superoptimality and suboptimality principles for value functions of stochastic differential games (see also the paper by Fleming and Souganidis [17]). In this paper we are extending to the stochastic case some results obtained by Soravia in [38] (see also [39] and [3]). He provides, in the general context of differential games, a representation formula for supersolutions of first order Isaacs equations. This gives a superoptimality principle which holds as an equality and which refers to equations which in general have not unique solutions. This result is then applied to the Lyapunov characterization of the stabilizability to an equilibrium set of a deterministic dynamical system with competitive controls by means of Lyapunov functions which are only required to be continuous on the boundary of the equilibrium set.
In the last section, we present a simple application of our Lyapunov method. We consider an asymptotically controllable deterministic system and we study under which conditions it remains stable if we add to it a stochastic perturbation. By the Lyapunov characterization of asymptotic controllability provided by Clarke, Ledyaev, Rifford, Stern ( [12]) and Rifford ([31]), we know that the unperturbed system admits a Lyapunov function, which is Lipschitz continuous and semiconcave except possibly at the origin. We study therefore under which perturbations this function remains a Lyapunov function also for the perturbed system. In particular we get a small intensity condition on the diffusion matrix σ, depending on the semiconcavity constant of the Lyapunov function and on qualitative properties of the stable trajectories of the deterministic systems.
We conclude with some additional references. We recall that there are other notions of stochastic stability. Kozin introduced the exponential almost sure stability of uncontrolled stochastic system. The stability in mean square and the p-stability were studied by means of Lyapunov functions (we refer to the monograph [21]). In the controlled case, in previous papers Bardi and the author (see [5]) characterized by means of appropriate Lyapunov functions the almost sure stabilizability of stochastic differential equations. This is a stronger notion of stochastic stability, never verified for nondegenerate processes. Indeed a system is almost surely stabilizable if it behaves as a deterministic stabilizable system and remains almost surely in a neighborhood of the equilibrium point. Turning to deterministic controlled systems, a complete Lyapunov characterization of the asymptotic stabilizability (called asymptotic controllability) has been proved by Sontag and Sussmann (see the articles [34], [36] and the review paper [35]). The infinitesimal decrease condition of the Lyapunov function along the trajectories of the system is expressed in terms of Dini directional derivatives, contingent directional derivatives and proximal subgradients. There is a large literature on the stabilization of deterministic controlled system by the Lyapunov function method: we refer to the monograph [2], to the papers [11], [31], see also the references therein.
The paper is organized as follows. In Section 2 we introduce the stochastic control problems and recall the definitions and the basic properties of the controls we are using. Section 3 is devoted to the proof of the representation formula (3) for bounded continuous viscosity solutions of the differential inequality (2) and then to the extension to lower semicontinuous functions. Section 4 contains the definitions of stabilizability in probability, asymptotic stabilizability and Lyapunov functions; in Section 5 we apply the results obtained in Section 3 to prove local and global versions of the Lyapunov theorems. We show that, given a control Lyapunov function or a strict control Lyapunov function, the system (CSDE) is respectively stabilizable or asymptotically stabilizable in probability. In Section 6 we introduce the notion of controlled attractor and we discuss the generalization of the direct Lyapunov method to the case of stabilization of viable sets. Finally in Section 7 we present some examples illustrating the theory.

Stochastic control setting
In this section we introduce the stochastic control problem and recall the definitions and the basic properties of the controls we are using.
We consider a controlled Ito stochastic differential equation: We assume that α t takes values in a given compact set A ⊆ IR M , f, σ are continuous functions defined in IR N × A, taking values, respectively, in IR N and in the space of N × M matrices, and satisfying for all x, y ∈ IR N and all α ∈ A We define a(x, α) : We recall here the definition of admissible controls that we are allowing for our control problems. For precise definitions we refer to [22] and [16] (see also references therein). Actually in these articles the problem is formulated in terms of solutions of the martingale problem, but it is also showed that there is an equivalent formulation in terms of solutions of (CSDE). We are relaxing the control problem by using weak controls, that is, admitting all the weak solutions of (CSDE). We have not assigned a priori a probability space (Ω, F , F t , P) with its filtration. So when we introduce a control we mean that we are at the same time choosing also a probability space and a standard Brownian motion B t on this space. Actually under the hypothesis (4) it can be shown that the space of strong controls is not empty and that, under suitable assumptions on the cost functional (which are essentially the lower semicontinuity with respect to the x variable), the strong problem and the weak problem have the same value ( [16,Theorem 4.11]). Definition 1 (Strict controls, Definition 2.2 [22]). For every initial data x ∈ IR N , a strict control is a progressively measurable A-valued process (α t ) t≥0 such that there exists a IR N valued, right continuous, almost surely continuous, progressively measurable solution X α t to (CSDE) (see also [16,Definition 1.4]). We denote with A x the set of strict controls for x ∈ IR N .
The class of strict controls can be embedded, as in the deterministic case, in a larger class of admissible controls. We denote by M (A) the set of probability measures on A endowed with the topology of weak convergence. We note that it is a separable metric space.
Definition 2 (Relaxed controls, Definition 3.2 [22]). For every initial data x ∈ IR N , a relaxed control is a progressively measurable M (A)-valued process (µ t ) t≥0 such that there exists a IR N valued, right continuous, almost surely continuous, progressively measurable solution X µ t to (CSDE) (see also [16,Definition 2.4]). We denote with M x the set of relaxed controls for x ∈ IR N .
We choose now a canonical probability space for our control problem. By means of this canonical space we can give a formulation of the optimization problem in a convex compact setting. The most natural canonical space for the strict control problem seems to be the space of the trajectories X . of (CSDE). It is the space of continuous functions C from [0, +∞) to IR N with its natural filtration. In order to give a control on this space, it is sufficient to specify the probability measure on C (which is the law of the process X . ) and the progressively measurable function α. Rather than working with this canonical space we consider the space of trajectories (X . , µ . ) for µ relaxed control. Let V the space of measurable functions from [0, +∞) to M (A) with its canonical filtration. We denote with M (V) the set of probability measures on V endowed with the stable topology (this is a topology introduced by Jacod and Menin, for precise definition we refer [22, section 3.10] and references therein). The canonical space for the relaxed control problem will be the product space C × V with the product filtration. We call canonic relaxed control or control rule a relaxed control defined in this canonical space (see [22,Definition 3.12] and also [16,Definition 3.2]). In order to identify a canonic relaxed control, it will be sufficient to specify the probability measure on the space C × V: the canonic relaxed controls can be considered as measures on the canonical space.
In the following we will consider a cost functional where l is a continuous, nonnegative function and V is a LSC, nonnegative function. The functional for every x and t, the lower continuity assumptions required in [22] on the cost functional. Then, since the supremum of LSC maps is LSC, we get that also the functional J(x, α) satisfies the same lower semicontinuity assumptions. We list here the results obtained in [22] that we are going to use. The crucial assumption for all of them, besides the right choice of the class of admissible controls and the lower semicontinuity of the cost functional, is the convexity assumption (5).
The class of control rules is the class on which it is possible to formulate a dynamic programming principle and to show the existence of an optimal control. The key result is Proposition 5.2 in [22]: for every initial data x, the class of optimal control rules admissible for x is convex and compact.
We have the following theorem stating the existence of an optimal control.
Theorem 3 (Theorem 4.7 and Corollary 4.8 [22]). Under the convexity assumption (5) and the other assumptions listed above, for every initial data x ∈ IR N there exists an optimal control rule for the control problem inf α J(x, α).
Moreover the infimum of the cost functional computed on the class of control rules coincides with the infimum of the cost functional computed on the class of strict controls: In particular the optimal control can be chosen strict.
The two crucial properties on the control space to get a dynamic programming principle are the stability under measurable selection and the stability under concatenation (see [33]). They are satisfied by the class of control rules. We consider a measurable set valued map from IR N to the space of probability measures on the canonical space C × V, with convex compact values. Then, by a standard measurable selection theorem (see [16,Theorem 5.3]) this map has a measurable selector. In [22,Lemma 5.5] (see also [40, ch 12] and [16, Theorem 6.3, 6.4]) it is proved that this measurable selector is an admissible control rule. Moreover, in [22,Lemma 5.8] (see also [16,Theorem 6.2]) it is shown that if we take an admissible control and then at some later stopping time we switch to an ε-optimal control from then on, the concatenated object is still admissible.
Finally we observe that all these results remain valid if we consider instead of the trajectories of (CSDE) in IR N , the trajectories of this system stopped at the exit time from a given open set (see [22]).

Superoptimality principles
In this section we prove a representation formula for bounded LSC viscosity supersolutions of Hamilton-Jacobi-Bellman equations. We start proving the result for continuous functions and then, by a standard approximation procedure, we extend it to LSC bounded functions. The representation formula is given first in the whole space IR N and then also in a localized version. We are adapting to the second order case the proof of optimality principles for viscosity supersolutions of first order Hamilton-Jacobi equations given by Soravia in [38] and [39]. This requires the use of stochastic control instead of deterministic control.
We consider the following Hamilton-Jacobi-Bellman equation where l : IR N → IR is a nonnegative bounded continuous function.
Theorem 4 (Representation formula for viscosity supersolutions). Consider a bounded LSC function V : IR N → IR. If V is a viscosity supersolution of the Hamilton-Jacobi-Bellman equation (7) in IR N , then it can be represented as where the infimum is taken over all strict admissible controls.
Proof. Without loss of generality, we can reduce to the case V ≥ 0 by an appropriate translation. Since V is LSC, bounded and nonnegative, we can consider an increasing sequence of continuous, nonnegative, bounded functions V k such that If V is continuous, we choose V k = V for every k. Now for every k ≥ 0, we introduce the following obstacle problem in IR N with unknown W and obstacle V k : Obviously V is a bounded LSC viscosity supersolution of the problem (9) for every λ ≥ 0 and every k ≥ 0. For λ > 0 fixed, define The plan of the proof is the following. First of all we show that the value function L λ,k is a bounded discontinuous viscosity subsolution of the obstacle problem (9): this means that the upper semicontinuous envelope L ⋆ λ,k is a viscosity subsolution of (9). Then, by the comparison principle for bounded discontinuous viscosity solutions of Isaacs equations we get that, for every . From this, we can conclude, sending λ to 0 and k to +∞, that V satisfies the superoptimality principle (8).
By the definition and the boundness of V k , we get that L λ,k is bounded. We want to prove that its upper semicontinuous envelope L ⋆ λ,k is a viscosity subsolution of the obstacle problem (9). To get this result it is sufficient to check that L ⋆ λ,k is a viscosity subsolution of the Hamilton-Jacobi-Bellman equation . This result can be obtained by standard methods in the theory of viscosity solutions if we prove a local suboptimality principle for L ⋆ λ,k on such points x (see [13] and [17]).
First of all we need the following technical lemma whose proof we postpone to the end.
This is a local suboptimality principle. This inequality, by a standard argument in the theory of viscosity solution (for the detailed argument see for example [7], see also [17]), implies that L ⋆ λ,k is a viscosity subsolution of equation (7) at the points x such that L ⋆ λ,k (x) > V k (x). From this we deduce that L ⋆ λ,k is a viscosity subsolution of (9). By the comparison principle obtained for Isaacs operators and bounded discontinuous viscosity solutions by Ishii in [24, Theorem 7.3], we get that V (x) ≥ L ⋆ λ,k (x) for every λ > 0 and k ≥ 0. In particular, for T > 0 and k fixed, we get Therefore for every T > 0 and k ≥ 0 Now we want to pass to the limit for k → +∞. For x fixed and every k ≥ 0 we consider an admissible control α k such that By the definitions recalled in Section 2 we can associate to each couple (X α k . , α k ) a control rule P k . By the compactness of the space of control rules, we can extract a subsequence of control rules, which we continue to denote with P k , to some control rule P in the stable topology. This control rule is the measure of a trajectory of (CSDE) (X . , µ) driven by a relaxed control µ. Since the convergence in the stable topology implies in particular the weak convergence of the measures P k to the measure P , we get immediately that for every t ∈ [0, T ] where the expected value on the left and right hand side is computed, respectively, using the measures P k and P .
Recalling now that V = sup k V k where V k are continuous functions, it is easy to show that V can be obtained as Moreover, since the convergence in the stable topology implies also the convergence in probability of X α k t to X t , we get that, for t ∈ [0, T ] fixed, we can extract a subsequence X α k t which converges to X t almost surely with respect to the measure P . Then along this subsequence By the Fatou lemma and the definition of stable convergence we deduce that, for each t ∈ [0, T ], along a subsequence lim inf where the expected value is computed respectively using the measures P k and P . To summarize, for every t ∈ [0, T ] we get from (12): So for every T > 0 there exists a control rule for which Now, by the statement (6) in Theorem 3, we obtain Now it remains only to let T → +∞. For ε > 0, consider an ε optimal control α for (13): in Considering X α T as starting point of the trajectory we obtain by (13) Let β be an ε 2 2 optimal control rule for V (X α T ) (we can choose a measurable selection of the optimal controls rules for V (X α T )(ω)), moreover the control rule obtained concatenating this selected control and α is still an admissible control rule. We can then proceed recursively and conclude by induction that we can construct an admissible control rule P such that By the statement (6) in Theorem 3 and recalling that ε is arbitrary, we obtain which is the desired formula.
We give here the proof of the technical Lemma 5 Proof. of Lemma 5. If the statement were not true, for every sequence x n → x such that L λ,k (x n ) → L ⋆ λ,k (x) and L λ,k (x n ) > V k (x n ) for every n we could find t n ≤ 1 n such that for By definition of L λ,k , for every ε n > 0 and every control α, there exists T (ε n , α) such that By the inequality (15), we can choose a sequence ε n → 0 and controls β n for which Therefore for every control α we obtain from the inequalities (16) and (17) E xn e −λtn L ⋆ λ,k (X βn n (t n )) + tn 0 l(X βn n (s))e −λs ds + ε n ≤ L λ,k (x n ) − ε n < < E xn e −λT (εn,α) V k (X α n (T (ε n , α))) + T (εn,α) 0 l(X α n (s))e −λs ds .
We claim now that for every n there exists α such that T (ε n , α) ≤ t n . Assume by contradiction that there exists N such that, for every α admissible, T (ε N , α) > t N , in particular T (ε N , α N ) > t N for every control α N which for t ≤ t N coincides with β N : by the previous inequality we get We choose now an admissible control rule which is ε N /2 optimal for L ⋆ λ,k (X βN N (t N )) (we operate a measurable selection between the ε N /2 optimal control rules for L ⋆ λ,k (X βN N (t N ))(ω)) and the concatenated control with β N is still an admissible control rule P N which is the measure associated to the couple (X N (.), µ N ). Therefore we get where the expected value is computed with respect to the measure P N , and the inequality we obtain gives a contradiction. Therefore there exists for every n an admissible control rule P n such that T (ε n , µ n ) ≤ t n : choosing α = µ n in the inequality (16) we get L λ,k (x n ) − ε n ≤ E xn e −λT (εn,µn) V k (X n (T (ε n , µ n ))) + T (εn,µn) 0 l(X n (s))e −λs .
For every n we call A n = {ω| X n (T (ε n , µ n )) ∈ B(x n , ( 1 4 √ n )} and B n = Ω \ A n : since for every n the trajectory is a Markov process and the drift and the diffusion of this control problem are equi-Lipschitz and equi-bounded in the compacts with respect to the control it is possible to show (we refer to [15, pp 284,285] for the proof) that P(B n ) ≤ KO( 1 3 is uniform with respect to the initial data x n and to the control µ n . Therefore we get from which, since V k is continuous and t n ≤ 1/n, letting n → +∞, we deduce in contradiction with our assumption.
Remark. The previous result can be proved in more general situations: consider a bounded, nonnegative, LSC viscosity supersolution V : where k : IR N → IR is a Lipschitz continuous nonnegative function. The proof of Theorem 4 applies directly and we obtain the representation formula We can prove also a localized version of the Theorem 4.

Corollary 6.
Consider an open set O ⊆ IR N . For every δ > 0, consider the set O δ := {x ∈ O | d(x, ∂O) > δ} and denote with τ α δ the stopping time at which the sample function of the process X α t reaches the boundary ∂O δ : we denote with τ α δ (t) the minimum between τ α δ and t. Assume that V : O → IR is a bounded nonnegative function. If V is a LSC viscosity supersolution of the Hamilton-Jacobi-Bellman equation (7) in O, then it can be represented, for every δ, x ∈ O δ , as Proof. We fix δ > 0 and a smooth cut off function 0 ≤ ξ ≤ 1 such that ξ(x) = 0 for x ∈ IR N \ O and ξ(x) = 1 for x ∈ O δ . We consider the stochastic controlled differential equation in IR N : Observe that for x ∈ O δ , the solution (X ′ ) α to (CSDE) ′ coincides a.s. with the solution X α to (CSDE) up to time τ α δ . We define the process X α τ α δ (t) obtained by stopping the process (X ′ ) α t at the instant it reaches the boundary of O δ : it has a Ito stochastic differential and it is still a strong Markov process with continuous trajectories (see for example [21,Lemma 3.3

.1] and references therein).
We extend V outside O as a bounded nonnegative LSC function that we continue to denote V . So it is immediate to show that V is a viscosity supersolution in IR N of the equation: We can apply Theorem 4 to V . Indeed it is sufficient to define We can repeat the proof in Theorem 4 (all the results in [22] hold also for the stopped process Y α ) and we get that L ⋆ λ,k is a viscosity supersolution of the obstacle problem (9) in IR N . So again repeating the same arguments of Theorem 4 we get that V satisfies the following representation formula for x ∈ O δ : with l ≥ 0: since the constant function U ≡ 0 is always a subsolution, it is interesting to characterize the minimal nonnegative supersolution. From a control point of view, the natural solution seems to be the value function of the infinite horizon control problem with running cost l If V ∞ is well defined and bounded, then it is possible to show that it is LSC, by an argument based on the properties of the class of admissible relaxed controls. Moreover, by standard methods in the theory of viscosity solutions (see [18], [13]), it is possible to show that V ∞ is a viscosity supersolution of the previous Hamilton-Jacobi-Bellman equation.
In this case, an easy application of the previous theorems gives that every bounded, nonnegative, viscosity supersolution V of the Hamilton-Jacobi-Bellman equation in IR N satisfies therefore V ∞ is the minimal nonnegative viscosity supersolution of the equation.
Remark (Representation formula for viscosity subsolutions). The counterpart of Theorem 4 for viscosity subsolutions is straightforward from classical suboptimality principles: let U : IR N → IR be an upper semicontinuous bounded viscosity subsolution of the Hamilton-Jacobi-Bellman equation then the function U can be represented as:

Stability in probability and Lyapunov functions
We begin this section with the notion of both Lyapunov and asymptotic stability in probability. They were introduced by Hasminskii and Kushner (see [21] and [25]) in the case of uncontrolled stochastic differential equations. We present their natural extension to the case of controlled diffusions.
Definition 7 (stabilizability in probability). The controlled system (CSDE) is (open loop) stabilizable in probability at the origin if for all ε, k > 0 there exists δ > 0 such that for every |x| ≤ δ there exists a control α . ∈ A x such that the corresponding trajectory X . verifies This is equivalent to assume that for every positive k The system is (open-loop) Lagrange stabilizable in probability, or it has the property of uniform boundedness of trajectories, if for each ε > 0, R > 0 there is S > 0 such that, for any initial point x with |x| ≤ R, This is equivalent to assume that for every R > 0 Remark. The stabilizability in probability implies that the origin is a controlled equilibrium of (CSDE), i.e., ∃ α ∈ A : f (0, α) = 0, σ(0, α) = 0.
In fact, the definition gives for any ε > 0, for k > 0 fixed, an admissible control such that the corresponding trajectory starting at the origin satisfies P(sup for any λ > 0 and any real function l such that 0 ≤ l(r) ≤ 1 for any r and l(r) = 0 for r ≤ k. Then inf α.∈A0 E 0 +∞ 0 l(|X t |)e −λt dt = 0. Theorem 3 implies that the inf is attained: therefore for any k > 0 there is a minimizing control which produces a trajectory satisfying a.s. |X t | ≤ k for all t ≥ 0. So inf α.∈A0 E 0 +∞ 0 |X t |e −λt dt = 0 for any λ > 0. Again Theorem 3 implies that the inf is attained, and the minimizing control produces a trajectory satisfying a.s. |X t | = 0 for all t ≥ 0. The conclusion follows from standard properties of stochastic differential equations.
Regarding the Lagrange stabilizability, we observe that, using standard properties of diffusions under the regularity assumptions (4), it is possible to prove (see [15,21]) that for every fixed T > 0 and R > 0 Nevertheless the Lagrange stabilizability is a stronger condition since it requires that The controlled diffusion is said to be asymptotically stabilizable in probability if the equilibrium point is not only stabilizable but also an attracting point for the system, locally around the equilibrium point.
Definition 8 (asymptotic stabilizability in probability). The controlled system is locally asymptotically stabilizable in probability at the origin if for all ε, k > 0 there exists δ > 0 such that for every |x| ≤ δ there exists a control α . ∈ A x such that the corresponding trajectory X . verifies This is equivalent to assume that for all k > 0 There is a global version of the previous stability notion: Definition 9 (asymptotic stabilizability in the large). The controlled system is asymptotic stabilizable in the large at the origin if it is Lyapunov stabilizable in probability around the equilibrium and for every This means that for every ε > 0 and for every initial data x we can choose an admissible control in A x which drives the trajectory to the equilibrium with probability greater than 1 − ε.
Next we give the appropriate definition of control Lyapunov functions for the study of the stochastic stabilities defined above. (iii) it is bounded; (iv) it is a viscosity supersolution of the equation We introduce the notion of strict Lyapunov function both in the local and global setting.
where l : O → IR is a positive definite, bounded and uniformly continuous function. (iv) it is a viscosity supersolution of the equation where l : O → IR is a positive definite uniformly continuous function.

Direct Lyapunov theorems
In this section we develop a direct Lyapunov method for the study of stabilizability in probability of controlled diffusions both in the local and global setting. For the uncontrolled case, the extension of the Lyapunov second method to the case of stochastic systems is due to Hasminskii and Kushner independently (see the monographs [21], [25], see also the references therein for earlier related results).
The main tool of the proof of the Lyapunov theorems is the representation formula for viscosity solutions obtained in Section 3.
Theorem 13 (Stabilizability in probability). Assume conditions (4), (5) and the existence of a local Lyapunov function V in the open set O. Then: (i) the system is stabilizable in probability, (ii) if in addition the Lyapunov function is global, then the system is also Lagrange stabilizable in probability.
Proof. We start proving (i). We fix k > 0 such that B k ⊂ O. We fix ε > 0 and define η = ε min |y|≥k V (y). We denote with τ α k (x) the first exit time of the trajectory X α t from the open ball B k centered at the origin with radius k. By the continuity at the origin of V we can find θ > 0 such that if |x| ≤ θ then V (x) ≤ η 2 . The superoptimality principle in Corollary 6 gives, for We choose now an η 2 optimal control α ∈ A x for the previous control problem, we denote by X t the corresponding trajectory, stopped at the exit time from O, and we get for every t ≥ 0 As t → +∞, we obtain the following bound on the probability that the trajectory X t leaves the ball B k P(sup This proves the stabilizability in probability. We pass now to (ii). Repeating the argument above we get that for every k > 0 This implies the Lagrange stabilizability: indeed given R > 0 and ε > 0, we choose k such that In the case the system admits a strict Lyapunov function we prove that there exists a control which not only stabilizes the diffusion in probability but also drives it asymptotically to the equilibrium. We obtain the result using standard martingale inequalities; in the uncontrolled case, a similar proof of asymptotic stability has been given in [14] (see also [30]).
Theorem 14 (Asymptotic stabilizability). Assume conditions (5), (4) and the existence of a local strict Lyapunov function in an open set O. Then (i) the system (CSDE) is locally asymptotically stabilizable in probability; (ii) if the strict Lyapunov function is global, then the system (CSDE) is asymptotically stabilizable in the large.
Proof. We start proving (i). For every k > 0, such that B k ⊂ O, we get, by Corollary 6, that the function V satisfies, for x ∈ B k , the following superoptimality principle where the trajectories are stopped at the exit time from B k . By Theorem 3 there exists an optimal control α ∈ A x for this value problem. We indicate with X . the corresponding trajectory and with τ the exit time from the open ball B k . Repeating the proof of Theorem 13 we get the stabilizability in probability: .
We claim that l(X t (ω)) → 0 as t → +∞ for almost all ω ∈ B(x), from this, using the positive definiteness of the function l, we can deduce that which gives, by the continuity at the origin of the function V , the asymptotic stabilizability in probability.
We indicate with τ (s) the minimum between τ and s and compute, for t ≥ 0 fixed By Theorem 3.4 in [15] (the process | s t σ(X u , α u )dB u | is a positive semimartingale) we get Then, Chebyshev inequality gives, for every t ≥ 0 fixed Since l is continuous, we can fix δ such that |l(x) − l(y)| ≤ ε 2 if |x − y| ≤ δ and |x|, |y| ≤ k: we compute We choose h such that 0 From (22), letting t → +∞, we get where the last inequalities are obtained using the strong Markov property of the process X τK (.) . This gives a contradiction: then, for every ε > 0, P(Ω ε ) = 0. We have proved that l(X t ) → 0 as t → +∞ for almost all ω ∈ B(x), now the positive definiteness of l implies that .
We prove now the statement (ii). If O coincides with the whole space, arguing as above, we get that for every k > 0 and x ∈ B k there exists a strict control α k such that the corresponding trajectory X k verifies Using the properness of the function V , by letting k → +∞, we get that for every which gives, by the positive definiteness of the function l, the asymptotic stabilizability in the large.
Remark (Uniform asymptotic stabilizability in probability). The existence of a Lyapunov function implies a stronger asymptotic stability of the system, which we call uniform asymptotic stabilizability. Moreover we will show in a forthcoming paper that the uniform asymptotic stabilizability can be completely characterized in terms of strict Lyapunov functions.
The system (CSDE) is uniformly asymptotically stabilizable in probability in O if for every x ∈ O there exists α ∈ A x such that for every k > 0 is the expected time spent by the trajectory X in the set O \ B k . The fact that the existence of a Lyapunov function implies the uniform asymptotic stabilizability follows very easily from the representation formula for the function V and the positive definiteness of the function l: The proof of the fact that uniform asymptotic stability implies asymptotic stability (in particular that for every initial data there exists a control driving asymptotically the trajectory to the origin almost surely) is an argument based on continuity properties of trajectories of (CSDE) of the type (23) we proved in Theorem 14.

Attractors
Next we extend the results in section 4 to study the stabilizability of general closed sets M ⊆ IR N . We denote by d(x, M ) the distance between a point x ∈ IR N and the set M .
We recall that a closed set M is viable with respect to a stochastic controlled dynamical system if for every x ∈ M there exists an admissible control such that the corresponding trajectory remains almost surely in M .
Definition 15 (Stabilizability in probability at M ). A closed set M is stabilizable in probability for (CSDE) if for every k > 0 there exists δ > 0 such that, for every x at distance less than δ from M , there exists an admissible control α such that the corresponding trajectory X . verifies Remark. We observe that if M is stabilizable in probability according to the previous definition, then in particular it is viable. In fact for every ε > 0 fixed, the definition gives that, for x ∈ M , inf α E x +∞ 0 e −λt k ε (X t )dt = 0 for any λ > 0 and any smooth function k ε which is nonnegative, bounded and null on the points at distance less than ε from M . By Theorem 3, the infimum is attained, therefore for every ε > 0 there is a control α ∈ A x whose corresponding trajectory stays almost surely at distance less than ε from M : in particular, for every λ > 0, inf α E x +∞ 0 e −λt |X t |dt = 0. Therefore, again by Theorem 3, there exists, for every x ∈ M , a minimizing control whose corresponding trajectory stays in M almost surely for all t ≥ 0.
A geometric characterizations of viability of closed sets with respect to a stochastic differential controlled equation has been given in [6] (see also references therein). According to this characterization, we note that the fact that the set M is stabilizable in probability implies that the diffusion has to degenerate on its boundary: for every x ∈ ∂M there exists α ∈ A such that σ(x, α) · p = 0 for every p generalized normal vector to M at x.
We introduce the notion of controlled attractiveness: it coincides, when the system is uncontrolled, with the standard notion of pathwise forward attractiveness (see [21]). This means that for every ε > 0 there exists α ∈ A x such that the corresponding trajectory approaches asymptotically the set M with probability at least 1 − ε.
The set O is called domain of attraction for M : if it coincides with IR N the set M is a global attractor.
Remark. We consider a function V : IR N → IR which satisfies the conditions in the Definition 12 of strict global Lyapunov function with the only difference that the function l is assumed only nonnegative definite. The proof of Theorem 14 can be repeated in this case: we obtain that for every We introduce the set L := {y | l(y) = 0}. From (26) we get that for every which means that L is a controlled global attractor for the system. For uncontrolled diffusion processes results of this kind can be found in [30] and [14]. The earlier paper of Kushner [27] studies also a stochastic version of the La Salle invariance principle, namely, that the omega limit set of the process is an invariant subset of L, in a suitable sense.
We can generalize the notion of control Lyapunov function in order to study the attractiveness and the stabilizability of a set M .
If l(x) ≥ 0 then V is a control M -Lyapunov function, if l(x) is a M positive definite, Lipschitz continuous bounded function then V is a strict control Lyapunov function.
We can therefore prove for the case of a set M very similar results as for the case of an equilibrium point.
Theorem 18. If the system (CSDE) admits a control M -Lyapunov function V then the system is M stabilizable; if moreover the function V is a strict control M -Lyapunov function then the set M is a controlled attractor for the system with domain of attraction equal to O.
Proof. In order to prove this result, one can repeat the proofs given in Theorems 13 and 14 since the function V satisfies a superoptimality principle and the level sets of V are a local basis of neighborhoods of M .

Examples
In this section we present some very simple examples illustrating the theory.
The first example is about a stochastic perturbations of stabilizable systems. We apply the Lyapunov Theorems to show that an asymptotically controllable deterministic dynamical system continues to be stabilizable or asymptotically stabilizable in probability if we perturb it with a white noise of intensity small enough. The idea to prove it relies on the fact that, if the stochastic perturbation is small enough, then a Lyapunov function for the deterministic systems remains a Lyapunov function also for the stochastic one. Example 1. We consider a deterministic controlled system in IR Ṅ X t = f (X t , α t ) (27) where f (x, a) is a Lipschitz continuous, locally bounded function in x uniformly with respect to a and the control α is a measurable function taking values in a compact space A. We assume that the system is globally asymptotically (open loop) stabilizable at the origin, i.e., asymptotically controllable in the terminology of deterministic systems [35,36]. By the converse Lyapunov Theorem [34,36], there exists a continuous control Lyapunov function for the system, i.e., for some positive definite continuous function L, there exists a proper, positive definite function V satisfying in IR N max in the viscosity sense. Moreover we can choose the function V to be semiconcave away from the origin as proved by Rifford in [31]. This means that for every δ > 0 there exists a semiconcavity constant C δ > 0 such that the function is concave in IR N \B δ . The semiconcavity constant C δ is an upper bound on the second derivatives of the function (to be intended in the sense of distributions). In particular, by the definition of semiconcavity, we get that if |x| > δ and (p, X) ∈ J 2,− V (x) then We study under which conditions the system continues to be asymptotically or Lyapunov stabilizable if we perturb it with a white noise. We consider the perturbed system dX t = f (X t , α t )dt + σ(X t , α t )dB t where (B t ) t is a M -dimensional white noise and the function σ(x, a) is Lipschitz continuous in x uniformly with respect to a and takes values in the space of the N × M dimensional matrices with entries in IR.
By the semiconcavity inequality (29) and by (28) Therefore, if the diffusion σ satisfies a small intensity condition trace a(x, α) ≤ L(x) C δ ∀α ∈ A ∀ |x| > δ, we can conclude that the function V is a control Lyapunov function for the stochastic system and then, according to Theorem 13, the system is both Lyapunov and Lagrange stabilizable in probability.
If moreover for every δ > 0 trace a(x, α) < L(x) C δ ∀α ∈ A ∀ |x| > δ, it is possible to construct a positive definite, Lipschitz continuous function l such that V is a viscosity supersolution of max a∈A −f (x, a) · DV (x) − trace a(x, α)D 2 V (x) ≥ l(x) and then, by Theorem 14, the system is asymptotically stabilizable in the large at the equilibrium. A similar result can be obtained in the case of local asymptotically controllable systems.
In the next example we give conditions on a radial function to be a Lyapunov function for the stability in probability.
Example 2. In this example we consider as candidate Lyapunov function for the general controlled system (CSDE) the function V (x) = |x| γ for some positive γ > 0 and study under which conditions the system is stabilizable.
Therefore If γ ≤ 2, this gives that the V is a Lyapunov function for the system if for every x there exists α ∈ A such that f (x, α) · x + trace a(x, α) ≤ 0. We can observe that, since trace a(x, α) ≥ 0 for every α, the radial component of the drift f must be everywhere nonpositive, for some α ∈ A.
In particular it must be negative to compensate the destabilizing role of the diffusion, whenever trace a(x, α) is nonnull.