{'fraction_non_alphanumeric': 0.0388227612877304, 'fraction_numerical': 0.006529054291597609, 'mean_word_length': 4.5311111111111115, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
State Constrained Stochastic Optimal Control Using LSTMs

In this paper, we propose a new methodology for state constrained stochastic optimal control (SOC) problems. The solution is based on past work in solving SOC problems using forward-backward stochastic differential equations (FBSDE). Our approach in solving the FBSDE utilizes a deep neural network (DNN), specifically Long Short-Term Memory (LSTM) networks. LSTMs are chosen to solve the FBSDE to address the curse of dimensionality, non-linearities, and long time horizons. In addition, the state constraints are incorporated using a hard penalty function, resulting in a controller that respects the constraint boundaries. Numerical instability that would be introduced by the penalty function is dealt with through an adaptive update scheme. The control design methodology is applicable to a large class of control problems. The performance and scalability of our proposed algorithm are demonstrated by numerical simulations.


I. INTRODUCTION
Optimal control has wide applications in robotic control [1], navigation [2], to name a few. With the availability of higher computational power and powerful optimization software, such as SNOPT [3], optimization algorithms have been applied to increasingly complex control problems. Optimal control casts the control problem in terms of cost functions that are addressed through numerical optimization techniques. In reality, systems also contain many uncontrolled inputs, such as measurement noise and external forces with unknown distributions, which if not taken into consideration in design, will lead to performance degradation. To address these uncontrolled inputs, a set of optimal control problems, namely stochastic optimal control, have been considered, which takes disturbances or measurement noise into consideration more explicitly.
The solution to a SOC problem involves solving a nonlinear partial differential equation (PDE), known as the Hamiliton-Jacobi-Bellman (HJB) equation. One approach is to approximate the state dynamics by linear [4] or quadratic [5] systems. However, this requires fine-grained time discretization or specialized linearization techniques. When the system is of high dimension, solving the HJB equation becomes difficult due to the curse of dimensionality. An alternative to linearization is sampling-based methods, such as Markov-Chain Monte Carlo (MCMC) approximation [6] to the HJB equation and forward-backward stochastic differential equations (FBSDE) [7]. However, numerical This work is supported in part by the NSF under grant DMS-1907518. 1  methods for MCMC-based methods are difficult to scale due to their reliance on predefined grids for value function backward propagation [6]. They also suffer from compounding errors in least square approximations for FBSDE-based methods [7]. Work has also been done in state constrained SOC problems, [8] translated the problem into a stateconstrained target problem and used its backward reachable sets to describe the value function. However, the efficacy of [8] was not demonstrated on a physical system.
Deep learning has been successfully applied to several application domains in recent years. One application of deep learning is to serve as an alternative to numerical methods in solving high-dimensional and nonlinear PDEs. In [9], DNNs are used to solve PDEs, such as Black-Scholes, HJB, and Allen-Cahn equations. However, in [9], only problems with available analytic solutions and uncontrolled state dynamics are tested. Following this line of work, DNNs have been utilized to solve FBSDEs for SOC problems. By using DNNs, we can learn the initial condition and then propagate forward the solution to the backward stochastic differential equation (BSDE). Additionally, solving the FBSDE using neural network (NN) based methods resolves the aforementioned compounding-error issue. The basic formulation of using LSTMs to solve FBSDEs was shown in [10], along with adding control constraints. However, the paper did not consider state constraints, which occur in most physical control problems. Another branch of deep learning, namely deep reinforcement learning (DRL), has also been deployed to solved optimal control problems. One significant difference between deep FBSDE based methods and DRL is the control in deep FBSDE based methods are given as functions of the value function whereas in DRL, the value function is either used as a baseline to reduce variance [11] or a state-action value function is used [12] to obtain control actions.
In this paper, we propose a SOC setting for nonlinear systems that incorporates state constraints. In Section II, the state constrained SOC formulation is given. In Section III, the deep FBSDE algorithm is presented along with modifications to include control saturation. In Section IV, an algorithmic solution is provided for adding state constraints. This results in a log-barrier-like method. In Section V, the NN architecture and overall algorithm are presented. In Section V, an adaptive update scheme to the state constraint penalty function is shown, which enhances stability during NN training. In Section VI, we show the efficacy of our approach on a cart-pole system under two different state constraints, namely cart movement and system energy. Finally, the paper is concluded and potential future research directions are presented.

II. PROBLEM FORMULATION
In this section, we outline the optimal control problem under state constraints. A system with dynamics that involves stochastic processes can be described using a stochastic differential equation (SDE) as follows (1) with initial condition x(0) = x 0 and w(t) ∈ R ν being a standard Brownian motion. The states are denoted by x ∈ R n , time by 0 < t < T < ∞, and the control input by u ∈ R m . In (1), f : R n ×[0, T ] → R n represents the drift, G : R n × [0, T ] → R n×m represents the control influence, and Σ : R n × [0, T ] → R n×ν represents the diffusion (influence of the Brownian motion on the state evolution). The state constrained SOC problem is to find a controller u(t) that minimizes an objective function J u (x, t) ∈ R + under a set of state constraints. The objective function is defined as where g : R n → R + is the terminal state cost, q : R n → R + is the instantaneous state cost, and the control cost matrix is R ∈ R m×m , which is positive definite. We consider state constraints in the following form where c s (x) ∈ R r is a vector of functions of the state, and c min ∈ R r and c max ∈ R r represent the lower and upper bounds of c s (x) component-wise, respectively. An example of this kind of constraint is a box constraint on states, i.e., wherein c s (x) = x. Additionally, control saturation (with U max ∈ R m ) is introduced, which has the form of where u i and U i,max are the i th elements of u and U max , respectively. The value function V (x, t) ∈ R + is defined as where the inf is computed over all control signals u(.) over the time interval (t, T ] satisfying the constraint (4).

III. DEEP FBSDE FORMULATION
To solve the state constrained SOC problem, a FBSDE formulation is used. In this section, we first consider the unconstrained SOC problem. Later in this section, we introduce the deep FBSDE formulation that is modified to handle control saturation. Modifications required to handle state constrained SOC problems will be introduced in Section IV. Starting from (2), we can use Dynkin's formula [13] along with Bellman's principle [14] to arrive at the HJB equation where V t is the partial derivative of V with respect to t. The generator function LV (x, t) is defined as where V x and V xx denote the first and second order partial derivatives of V w.r.t. x, respectively. The Hamiltonian is (8) Using first-order conditions to solve for the optimal control in the Hamiltonian yields This form of G captures the characteristic that the range of G belongs to the range of Σ, which excludes the case of a channel containing control input without noise. This representation aligns with the fact that no noiseless control signal exists in reality. Additional to this basic formulation, the control inputs are saturated as shown in (4). The saturation is introduced using the sig function as in [10], which is defined as The optimal control action is then calculated as with the control saturated between [−U max , U max ]. In (11), " * " represents element-wise multiplication. For the control saturated system, a new value function is introduced with constant weights c i , dummy variable for integration v. For this control saturated system, the Hamiltonian will become Following the formulation in [10], we can solve for (11) using a FBSDE, which has the form of The deep FBSDE method [10] introduces a NN with parameters θ to estimate V x (x(t n ), t n ) at each time step, which is denoted as V x (x(t n ), t n ; θ). Using the V x (x(t n ), t n ) estimation and following (11), we can obtain the control action u * n . The initial value function estimation V (x(t 0 ), t 0 ) and its partial derivative V x (x(t 0 ), t 0 ) are also learned via trainable weights φ; with some abuse of notation, we denote these learned models as V (φ) and V x (φ). This enables the forward propagation of the backward part in the FBSDE. The original continuous finite time horizon problem is discretized into N time steps (T = N ∆t). This gives us ∆y(t n ) and ∆x(t n ). Using a numerical integration scheme as in [9], we can find the value function estimation at the last time step V (x(t N ), t N ) along with the terminal state x(T ). The loss function of the DNN is chosen such

IV. STATE CONSTRAINT FORMULATION
In this section, we present our formulation to incorporate state constraints on top of the control saturated FBSDE formulation. At each time instant, a NN is used to estimate V x . Then, V x is used to calculate the optimal control that minimizes the cost-to-go. Additionally, the optimal controller of the state constrained SOC problem needs to satisfy the state constraints. We can modify the cost function, such that the cost-to-go is minimized only when the state constraints are respected. This can be achieved by adding a "soft" constraint in the form of a penalty, which increases the cost when the state is outside the constraint boundary. This approach is preferred over using a sigmoid-like function to force the state dynamics into a predefined range. Using the sig function, the dynamics would be altered using a "fake" saturation which is not present in the real dynamicshence, training on such an altered dynamic model would not yield a controller that functions properly under the real dynamics. When adding the penalty, states inside the constraint boundary should not be penalized, but for states outside the constraint boundary, a large penalty should be given. Therefore, the penalty function should be close to zero inside the constraint boundary and a large number outside. The function satisfies the aforementioned requirements. L ∈ R + is a scalar determining the maximum value of the penalty, k ∈ R + determines the steepness of the boundary (larger k leads to steeper boundaries), as shown in Figure 1, and µ = 0.5(c max + c min ) ∈ R r represents the mid-point of the constraint region. The proposed penalty function consists if σ c < β or l mod η = 0 then 8: if δ < 0 then End Algorithm of two parts: the first part is the two logistics functions, which gives the "U"-like shape; the second part moves the minimum value of the penalty function to zero. Using the penalty function p(x), the new state cost at each time instant becomes Note, when choosing parameters for p(x), ideally we would pick larger k and L values to ensure a steeper boundary. After applying a penalty function, the Hamiltonian becomes where u * is defined in (11). Thus, state constraints can be applied by simply swapping the Hamiltonian defined in (14) to the form in (18).

V. STATE CONSTRAINED DEEP FBSDE CONTROLLER
In this section, we present the state constrained deep FBSDE algorithm, an overview of the corresponding NN architecture, and an adaptive update scheme to the state constraints that greatly enhances training stability.

A. Algorithmic Design
The time horizon T is discretized into N time steps. At each time step n, we use a NN parameterized by θ. The input to the NN at each time step is the current state x m n (and the hidden state from the previous time step H m n−1 when using LSTMs), where m represents the m-th sample in the mini-batch. The output of the NN at each time step is V m x (x(t n ), t n )) , which is then used to calculate the control action using (9). Following (15), we can obtain the next value function estimation and the next state using Euler's integration. After the last time step, the loss is computed as where λ determines the weight of the regularization term. The initial value function estimate V (x(t 0 ), t 0 ) and its partial derivative V x (x(t 0 ), t 0 ) are parameterized by φ V (if using LSTMs, initial hidden states H 0 are parameterized by φ H ). The initial state ξ is fixed. We use a batch size of M during training. The NN is trained for a total L iterations using the Adam optimizer under a varying learning rate. All of the weights are initialized using the Xavier normal initializer.

B. Neural Network Architecture
Both dense-layer based and LSTM based NN architectures have been used in [10]. The benefits of using a LSTM based architecture are two-fold. Firstly, the weights are shared among time steps, compared to a dense-layer based architecture resulting in reduced number of weights. Secondly, for dense-layer based methods, weights at each time step are step specific, making it difficult to scale to long horizons. For LSTM based architectures, we can train for a shorter time horizon than during testing, given the state distribution for longer time horizons are the same. The NN architecture illustrated in Figure 2 is inspired by [10]; thus, only a brief introduction is provided. A LSTM-based architecture is used to tackle the vanishing gradient problem in long time horizons. Additional to the basic LSTM model, we have a forward part which represents the flow of state dynamics and a backward part representing the flow of the value function.

C. Adaptive Update Scheme
In the initial stage of training, parts of many trajectories lie outside of the constraint boundary. If a steep boundary is in use, it will result in a very large gradient and cause numerical instabilities in training even if the gradient is clipped. If the steepness of the boundary is fixed at a small value, the penalty of violating the constraint is not significant enough for the learning algorithm to achieve the required state constraint. Therefore, an adaptive update scheme is needed to gradually increase the boundary steepness k in order to avoid exploding gradients while shaping the controller behavior.
For a given k, we train the DNN until the performance of the controller cannot be further improved, then we update k. To determine the performance of the controller, we can use the square root of state cost variance σ c ∈ R over a given amount of iterations. The state cost is defined in (17); its square root variance decreases as the learning algorithm converges to a solution. Define the threshold of determining convergence as β ∈ R + . Once σ c becomes smaller than β, we update k. We check for the condition σ c < β every η iterations; if the condition is not satisfied after η iterations, k will also be updated (may be stuck in a local minimum).
During the training process, the variance decreases. Thus, β should also be decreased after each update. To make the decrease in β smoother, we let the "acceleration," ∆ ∈ R + , become negative. Similarly, the acceleration of the increase of k is made negative, which is denoted by ∆ δ ∈ R + , leading to more fine-grained changes in k at later stages of training. Algorithm 1 shows the update scheme for k and the overall algorithm is given in Algorithm 2.

VI. EXPERIMENTS
In this section, we show the efficacy of our control algorithm on the cart-pole for a swing-up task under two different state constraints: cart movement and system energy. For both systems, the trained model is evaluated over 256 trials. All experiments are carried out using TensorFlow.

A. Cart-pole Swing-Up Task I
The task for the cart-pole system is to swing the pole up from downward position and stabilize it at the upright position. The cart-pole dynamics are given as where X i is the difference between the current state and the target state. Q is the cost weight matrix for the state, and S i is defined in (13). p(·) is the penalty function that incorporates the state constraint. The state dynamics under the control generated by the trained model are shown in Figure 3. For the constrained stochastic optimal controller, the state lies within the constraint boundary for all time. Figure 4 shows that without the presence of a state constraint, the cart position and velocity do violate constraints significantly. After applying the state constraint, the effect can be    Figure 3. The state target and constraint boundary also follows the color scheme in Figure 3. The leftmost figure shows that under no state constraints the cart velocity violates the constraints. After increasing k, the velocity trajectory gradually retreats inside of the boundaries and eventually, it entirely satisfies the constraints.
immediately seen on the state trajectories. With k = 1.5, the portion of the trajectory that lies outside of the constraint boundaries is greatly reduced. After gradually increasing k to 6.0 following Algorithm 1, the entire trajectory of the cart velocity lies within the constraint boundaries. Also, we tested directly training on k = 6.0 without such gradual increase, the algorithm becomes numerically unstable after one forward pass due to large gradients. This demonstrates that our method provides a stable training scheme.

B. Cart-pole Swing-Up Task II
Using the same model as before, a state constraint is applied to limit the total energy of the system (sum of potential and kinetic energy), which is a nonlinear function Algorithm 2 State Constrained Deep FBSDE Controller 1: Given: 2: x 0 = ξ, f (t, x), G(t, x), Σ(t, x): Initial state and state dynamics; 3: g(x), ∇ x g(x), q(x), R: Cost function parameters; 4: N : Task horizon, K: Number of iterations, M : Batch size, ∆t: Time interval, λ: Weight decay parameter; 5: ∆, η, η , ∆ δ : state constraint parameters (refer to Algorithm 1); 6: Initialization: 7: θ: Weights and biases of dense and LSTM layers; 8: φ: Weight of the initial value function and hidden unit; 9: k, δ, β, γ (refer to Algorithm 1); 10: is constrained at ±5J. The cost is similar to (22), with the only difference being changing p(·) to incorporate the system energy constraint. The total energy under the unconstrained and constrained controllers are shown in Figure 5.

VII. CONCLUSION
In this paper, SOC problems with state constraints are formulated as a FBSDE and solved utilizing an LSTMbased DNN to alleviate the curse of dimensionality and Fig. 5. Energy Constrained Cart-pole Controller: the color scheme follows the one from Figure 3. It can be shown that under the constrained controller the total energy is able to be maintained under 5J which is not achieved under the unconstrained controller. numerical integration issues. An adaptive update scheme is used to apply state constraints, which greatly enhanced the stability during training of the LSTM network. The efficacy of our approach is demonstrated on a cart-pole system under two different state constraints: 1) cart position and velocity, and 2) system energy. Potential future directions include application to systems with partial state measurements and robust control under various uncertainties.