{'fraction_non_alphanumeric': 0.038483720442510354, 'fraction_numerical': 0.00812667121061186, 'mean_word_length': 4.5335073977371625, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data-Driven Robust Control Using Reinforcement Learning

This paper proposes a robust control design method using reinforcement-learning for controlling partially-unknown dynamical systems under uncertain conditions. The method extends the optimal reinforcement-learning algorithm with a new learning technique that is based on the robust control theory. By learning from the data, the algorithm proposed actions that guarantees the stability of the closed loop system within the uncertainties estimated from the data. Control policies are calculated by solving a set of linear matrix inequalities. The controller was evaluated using simulations on a blood glucose model for patients with type-1 diabetes. Simulation results show that the proposed methodology is capable of safely regulates the blood glucose within a healthy level under the influence of measurement and process noises. The controller has also significantly reduced the post-meal fluctuation of the blood glucose. A comparison between the proposed algorithm and the existing optimal reinforcement learning algorithm shows the improved robustness of the closed loop system using our method.


Introduction
Control of unknown dynamic systems with uncertainties is a challenge since most of the controllers require an exact mathematical model. Due to the fact that many processes are complicated, nonlinear and varying with time, a control algorithm that does not depend on a mathematical model and is able to adapt to time-varying conditions is required. A popular approach is to develop a universal approximator for predicting the output of unknown systems [1]. Control algorithms can then be designed based on the parameters of the approximators. Many control techniques have been proposed based on this approach using neural networks and fuzzy logic. For example, Goyal et al. [2] proposed a robust sliding mode controller which can be designed from Chebyshev neural networks. Chadli and Guerra [3] introduced a robust static output feedback controller for Takagi Sugeno (TS) fuzzy models. Ngo and Shin [4] proposed a method to model unstructured uncertainties using type-2 fuzzy neural networks and the Takagi Sugeno fuzzy controller based on the model. However, obtaining a good approximator requires a significant amount of data, especially for a complicated model with many inputs and outputs. The data-driven model must also be updated frequently for time-varying systems. Also, many control design techniques assume uncertainties as functions of system parameters. However, in many cases, the causes of uncertainties are not known (i.e. unstructured uncertainties).
With the development of data science and machine learning, reinforcement learning (RL) has emerged as an effective method to control unknown nonlinear systems such as robotics manip-Type-1 diabetes is a disease caused by the lack of insulin secretion. The condition results in uncontrolled increase of blood glucose level if the patients are not provided with insulin doses. High blood glucose level can lead to both acute and chronic complications, and eventually result in failure of various organs. One of the major challenges in controlling the blood glucose is that the biochemical and physiologic kinetics of insulin and glucose is complicated, nonlinear, and only approximately known [10]. Also, the stability of the control system is very essential in this case since unstable control effort will lead to life-threatening condition for the patients.
This paper proposes a data-driven robust control algorithm using reinforcement learning for partially-unknown dynamical systems. The purpose of the algorithm is to ensure the stability of the closed loop system under uncertainty conditions. The proposed methodology will be applied to a blood glucose model for testing its effectiveness in controlling the blood glucose level in patients with type-1 diabetes.
The content of the paper is organized as follows. Section II describes the proposed robust reinforcement-learning algorithm. Section III shows the simulation results of the methodology. The conclusion is given in Section IV.

Robust Control Using Reinforcement Learning
In this paper, we consider a class of dynamical system which can be described by the following linear state-space equation: where x ∈ R n is the vector of n state variables, u ∈ R m is the vector of m control inputs.
A ∈ R n×n is the state matrix and B ∈ R n×m is the input matrix. It is assumed that matrix A is unknown. Our target is to derive a control algorithm u(t) that can regulate the state variables contained in x(t) without knowing matrix A and based on input and output data.
As a RL framework, the proposed robust control algorithm consists of an agent that takes actions and learns the consequences of its actions in an unknown environment. The environment is defined by a state vector x(t) that describes its states at time t. The action at time t is represented by u(t). As a consequence of the action, a cost r(t) is incurred and accumulated. The cost function r(t) is assumed to be known and pre-defined as a function of the current state and action. The objective of the learning process is to minimize the total cost accumulation in the future.
At each decision time point, the agent receives information about the state of the environment and chooses an action. The environment reacts to this action and transitions to a new state, which determines whether the agent receives a positive or negative reinforcement. Current RL techniques proposed optimal actions by minimizing the predicted cost accumulation. However, uncertainties due to noises in the data or inaccurate estimation of the cost accumulation can lead to sub-optimal actions and even unstable responses. Our target is to provide the agent with a robust and safe action that can guarantee the reduction of the future cost accumulation in the presence of uncertainties. The action calculated by the proposed algorithm may not be the optimal action that reduce the cost in the fastest way, but it can always guarantee the stability of the system, which is imperative in many critical applications.

Estimation of the Value Function by the Critics
In the RL context, the accumulation of cost over time, when starting in the state x(t) and following policy π, is defined as the value function of policy π, i.e.: where γ = 1 is the discount factor. The cost r(t) is assumed to be a quadratic function of the states: where the positive definite matrix Q ∈ R n×n is symmetric and positive semidefinite (since the cost is assumed to be non-negative) contains the weighting factors of the variables that are minimized.
In order to facilitate the formulation of the stability condition in the form of linear matrix inequalities (LMI), the value function V (x(t)) is approximated by a quadratic function of the states: where the kernel matrix P ∈ R n×n is symmetric and positive semidefinite (since matrix Q in the cost function is symmetric and positive semidefinite).
By using the Kronecker operation, the approximated value function can be expressed as a linear combination of the basis function φ(x(t)) = (x(t) ⊗ x(t)): where w is the parameter vector, φ(x(t)) is the vector of basis functions and ⊗ is the Kronecker product. The transformation between w and P can be done as follows: w = vec(P ) = [P 11 , P 21 , ..., P n1 , P 12 , ..., P 1n , P nn ] T where P i,j is the element of matrix P in the i th row and j th column. With T as the interval time for data sampling, the integral reinforcement learning (IRL) Bellman equation can be used to update the value function [8]: By using the quadratic cost function (Eq. 3) and the approximated value function (Eq. 5), the IRL Bellman equation can be written as follows: At each iteration, n samples along the state-trajectory are collected ( The mean value of w can be obtained by using least square technique: where with i = 1, 2, ..., N . The confidence interval for the coefficient w (j) is given by: where 1−θ is the confidence level, q 1− θ 2 is the quantile function of standard normal distribution, τ j is the jth element on the diagonal of (XX T ) −1 andσ 2 =ˆ Tˆ n−p , with = Y −ŵX. From that, the uncertainty ∆w is defined as the deviation interval around the nominal value : MatricesP and ∆P can be obtained by placing elements ofŵ and ∆w into columns.

Policy Improvement by the Actor
Linear feedback controller has been widely used as a stabilization tool for nonlinear systems where its dynamic behavior is considered approximately linear around the operating condition [11][12][13]. Hence, in this paper, we use linear functions of the states with gain K i as the control policy at iteration i: and the level of uncertainty is constant during the controlling process, the task of the actor is to robustly improve the current policy such that the value function is guaranteed to be reduced during the next policy implementation. If the following differential inequality is satisfied: with some positive constant α then by using the comparison lemma (Lemma 3.4 in [14]), the derivative of functionV i (x(t)) can be bounded bẏ Therefore, maximizing the rate α will ensure a maximum exponential decrease in the value oḟ V i (x(t)).
The following part shows the main results of the paper, which describe how the policy gain can improved during the learning process. Derivations of the results are provided in the stability analysis (Subsection 2.3). In order to relax the stability condition and maximize the chance to obtain feasible solutions, we divide the problem into two cases depending on whether the policy update is made frequently or not. In the general policy-update case, it is assumed that the sign of all the state variables can be changed between each policy update interval. In the frequent policy-update case, it is assumed that the sign of all state variables cannot be changed between each policy update interval.

General Policy Update
The improved policy K i+1 can be obtained by solving the following linear program: Minimize α subjected to the following LMIs and where the notion S ≤ 0 is a generalized inequality meaning S is a negative semidefinite matrix, β is the worst case norm of ∆P i , which can be estimated using the µ analysis [15], and Inequality (20) provides the stable condition and inequality (21) provides the upper bound for the norm of the updated gain K i+1 . The derivation of (20) is provided in Subsection 2.3.

Frequent Policy Update
Definition 1. Assume A is a square matrix with dimension n × n and x is a vector with dimension n × 1. The maximize operation on matrix A and vector x is defined as follows: where The improved policy K i+1 can be obtained by minimizing α subject to where: and . Similar to the general policy update, inequality (27) provides the stable condition and inequality (28) provides the upper bound for the updated gain K i+1 . The derivation of (27) is provided in Subsection 2.3.

Stability Analysis
With the control policy as described in Eq. 17, the equation for the closed loop system can be derived as follows: Lemma 1. Assume that the closed loop system described by Eq. (31) is stable, solving for P in Eq. (8) is equivalent to finding the solution of the underlying Lyapunov equation [8]: Proof. We start with Eq. 32 and try to prove that matrix P is also the solution of Eq. (8).
where P is the solution of Eq. (32), we have: Since the closed-loop system is stable, the Lyapunov equation (32) has a unique solution, P i > 0. From (33), this solution will satisfy: which is equivalent to Therefore, P is also the solution of Eq. (8).

Lemma 2.
Given matrices E and F with appropriate dimensions, the following linear matrix inequality (LMI) can be obtained: Proof. From the properties of matrix norm, we have: which is equivalent to: If: • the current control policy u(t) = π i (x(t)) = −K i x(t) is stabilizing, • the LMI given in (20) is satisfied with some positive constants γ 1 and γ 2 , then the closed loop system with the control policy u(t) = −K i+1 x(t) is quadratic stable with convergence rate α.
Proof. Since the current control policy is stable, the estimated parameter matrix P i is positive definite. Hence, V i (x(t)) = x T t P i x t > 0. Here, we will use V i (x(t)) as the Lyapunov function for the updated control policy u(t) = π i+1 (x(t)) = −K i+1 x(t). For notation convenience, the state vector x(t) and input vector u(t) are denoted as x t and u t , respectively. We have: By using Lemma 2, the following inequalities can be obtained: Hence,V i (x(t)) + αV i (x(t)) can be bounded by: Using the Lyapunov theory, the system will be quadratic stable with the convergent rate α iḟ . This condition will satisfy if: The above condition can be written in the matrix form as shown in the theorem.
Lemma 3. Given A as a square matrix with dimension n × n and x as a vector with dimension n × 1, the following LMI can be obtained: where C = maximize(A, x) as in Definition 1.
Proof. We have: If: • the current control policy u(t) = π i (x(t)) = −K i x(t) is stabilizing, • the LMI given in (27) is satisfied with some positive constant γ 2 , then the closed loop system with the control policy u(t) = −K i+1 x(t) is quadratic stable with convergence rate α.
Proof. Since the current control policy is stable, the estimated parameter matrix P i is positive definite. Hence, V i (x(t)) = x T t P i x t > 0. Here, V i (x(t)) is used as the Lyapunov function for the updated control policy u(t) = π i+1 (x(t)) = −K i+1 x(t). For notation convenience, the state vector x(t) and input vector u(t) are denoted as x t and u t , respectively. We have: We have the following inequalities: and where and ∆P i,max = maximize(∆P i , x). The maximize operator is defined in Definition 1.
Hence,V i (x(t)) + αV i (x(t)) can be bounded by: 48) Using the Lyapunov theory, the system will be quadratic stable with the convergent rate α iḟ V i (x(t)) ≤ −αV i (x(t)). This condition will satisfy if: The above condition can be written in the matrix form as shown in the theorem.
By using theorem 1 and 2, it can be seen that with the proposed improved policy, the closed loop system will be asymptotically stable.

Robust Reinforcement Learning Algorithm
The robust reinforcement learning algorithm for controlling partially unknown dynamically systems includes the following steps: • Select an initial insulin policy u(t) = K 0 x(t).
Estimation of the value function (step i = 1, 2, ...) • Apply the control action u(t) based on the current policy u(t) = −K i x(t).
• At time t + T , collect and compute the data set (X, Y ), which are defined in Eq. 11 and Eq. 13.
• Update vector w by using the batch least square method (Eq. (15)).

Control policy update
• Transform vector w into the kernel matrix P using the Kronecker transformation.
• Update the insulin policy by solving the LMI in Theorem 2.

Simulation Setup
A simulation study of the proposed robust RL controller was conducted on a glucose kinetics model, which can be described by [16][17][18][19]: In this model, variable descriptions and parameter values can be found in Table 1 and Table 2.
Variable w(t) is the process noise. The measured blood glucose value is affected by a random noise v(t): The inputs of the model are the amount of CHO intake D and the insulin concentration i. The value of i(t) − i b (t) must be non-negative:  Glucose in compartment 2 mmol g(t) Plasma glucose concentration mmol/l χ(t) Interstitial insulin activity min −1 i(t) Plasma insulin concentration µIU/ml

Results
In order to evaluate the performance of the robust RL controller, we implemented the controller on the glucose kinetics model as described in the previous section under a daily scenario of patients with type-1 diabetes. In order to make the scenario realistic, three different levels of uncertainties were used in the model. Uncertainties include process noise (w(t)) and measurement noise (v(t)). It is assumed that the noises are Gaussian distributions with standard deviations for each case as shown in Table 3.

Without Meal Intake
This part describes the simulation results during the fasting period (without meal intake). The purpose of the simulation is to compare the performances of the robust RL algorithm with the conventional optimal RL algorithm in the nominal condition (uncertainty case 1). The initial blood glucose for both scenario was set at 290 mg/dL and the target blood glucose is 90 mg/dL. The initial policy at the beginning of the simulation was chosen as follows: Fig. 1 shows the comparison in blood glucose level between the robust RL and the optimal RL algorithm in the nominal condition. From the results, it can be seen that the robust RL  Figure 1: Comparison of blood glucose responses in nominal case without meal intake Figure 2: Comparison of insulin concentration in nominal case without meal intake successfully reduces the blood glucose level while the optimal RL becomes unstable when the blood glucose approaches the desired value. The instablity of the optimal RL in this case can be explained by the nonlinearity of the system (due to the coupling term χ(t)g(t) in Eq. 51), the saturation of the insulin concentration (Eq. 54), and the lack of perturbed data when the blood glucose approaches the steady state value. The insulin concentration during the simulation can be found in Fig. 2. In this figure, the dotted blue line indicates the unstable insulin profile.

With Meal Intake
In this part, the performance of the robust RL controller was tested where the system is subjected to meal intakes with the carbohydrate profile as shown in Fig. 6.
During the simulation period with meal intakes, blood glucose responses throughout the day of the robust RL control systems under four uncertain cases are shown in Fig. 7. The insulin concentration during the process can also be found in Fig. 8. The results show that the controller provides the most aggressive action under case 1 (no uncertainty) and the least aggressive action under case 4 (with highest level of measurement and process noises). This leads to the largest and smallest reduction of postprandial blood glucose in case 1 and case 4, respectively. Most importantly, the robust RL algorithm kept the system in stable condition and there is no hypoglycemia event during the simulation for all four cases under different level of uncertainties.

Conclusion
The paper proposes a robust reinforcement learning algorithm for dynamical systems with uncertainties. The uncertainties can be approximated by the critic and represented in the value function. LMI techniques were used to improve the controller gain. The algorithm was simulated on a blood glucose model for patients with type-1 diabetes. The objective of the simulation is to control and maintain a healthy blood glucose level. The comparison between the robust RL algorithm and the optimal RL algorithm shows a significant improvement in the robustness of the proposed algorithm. Simulation results show that the algorithm successfully regulated the