{'fraction_non_alphanumeric': 0.034345368625876393, 'fraction_numerical': 0.007536213979059209, 'mean_word_length': 4.421634293369055, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning Min-norm Stabilizing Control Laws for Systems with Unknown Dynamics

This paper introduces a framework for learning a minimum-norm stabilizing controller for a system with unknown dynamics using model-free policy optimization methods. The approach begins by first designing a Control Lyapunov Function (CLF) for a (possibly inaccurate) dynamics model for the system, along with a function which specifies a minimum acceptable rate of energy dissipation for the CLF at different points in the state-space. Treating the energy dissipation condition as a constraint on the desired closed-loop behavior of the real-world system, we formulate an optimization problem over the parameters of a learned controller for the system. The optimization problem can be solved using model-free policy optimization algorithms and data collected from the real-world system. One term in the optimization encourages choices of parameters which minimize control effort, while another term penalizes violations of the safety constraint. If there exists at least one choice of learned parameters which satisfy the CLF constraint then all globally optimal solutions for the optimization also satisfy the constraint if the penalty term is scaled to be large enough. Furthermore, we derive conditions on the structure of the learned controller which ensure that the optimization is strongly convex, meaning the globally optimal solution can be found reliably. We validate the approach in simulation, first for a double pendulum, and then generalize to learn stable walking controllers for underactuated bipedal robots using the Hybrid Zero Dynamics framework. By encoding a large amount of structure into the learning problem, we are able to learn stabilizing controllers for both systems with only minutes or even seconds of training data.


I. INTRODUCTION
Recently, the literature has displayed a renewed interest in data-driven methods for controller design [1]- [5]. Much of this excitement has been driven by recent advances in the model-free reinforcement learning literature [6]- [8]. Despite their generality, model-free policy optimization methods are known to suffer from poor sample complexity, as they generally are unable to take advantage of known structure in the control system. This paper bridges the gap between model-based and model-free design paradigms by embedding Lyapunov-based design techniques into a model-free reinforcement learning problem. By encoding basic information about the structure of the system into the learning problem through a CLF, our approach is able to learn optimal stabilizing controllers for highly uncertain systems with as little as seconds or few minutes of data.
Specifically, the paper proposes a framework for learning a min-norm stabilizing control law for an unknown system using model-free policy optimization techniques. Our approach begins by first designing a CLF for a nominal dynamics model of the system alongside a function which specifies the desired rate of convergence for the closedloop system. To impose this desired behavior on the real world control system, we then formulate a continuoustime optimization problem over the parameters of a learned controller which treats the energy dissipation condition as a constraint. The cost function for the optimization encourages choices of parameters which minimize control effort, but uses a penalty term to ensure that the dissipation constraint is satisfied, if possible. The terms in the optimization depend on the dynamics of the unknown system, but discrete-time approximations to the problem can be solved using policyoptimization algorithms and data collected from the plant.
On the theoretical side, we demonstrate that if there is at least one choice of parameters which satisfy the dissipation constraint, then all globally optimal solutions to the optimization will satisfy the constraint when the penalty term is large enough. However, in general, the optimization problem is non-convex, meaning we cannot reliably find its globally optimal solutions. To address this issue, we demonstrate that if the learned controller is constructed using a linear combination of independent basis functions, then the optimization problem becomes strongly convex, meaning its unique globally optimal solution can be found reliably. In this special case, we are able to reliably find the set of learned parameters which minimize the exerted control effort while satisfying the desired dissipation constraint.
To demonstrate the utility of the proposed framework, we apply the method in simulation to a double pendulum and a high-dimensional model of a bipedal robot. For the double pendulum example, the learned controller is comprised of a linear combination of radial basis functions so that the convexity result discussed above applies, and we demonstrate empirically that the learned controller is able to closely match the true min-norm controller performance. The walking example demonstrates how to extend our results in the body of the paper to encompass the Hybrid Zero Dynamics framework as in [9]. For this high-dimensional system, a feed-forward neural network is used for the learned controller. While we cannot guarantee that the optimal set of parameters is found, the learned controller still produces a stable walking motion in the face of high model uncertainty.

A. Related Work
CLF-based controllers [10], [11] have been proved to be effective for a wide variety of complex robotic tasks, such as bipedal walking [12], [9], manipulation [13] and multiagent coordination [14]. In [12] and [13] quadratic programs (CLF-QP), which integrate the CLF condition as a constraint, are used to get optimal min-norm stabilizing controllers. The CLF-QP is solved online and additional constraints, such as input saturation, can be added.
However, the dynamics of many real-world systems have nonlinearities that might be difficult to model correctly and or physical parameters which are difficult to identify. Input-tostate stability has been used to tackle this problem in [15], [16]. Also, adaptive [17] and robust [18], [19] versions of CLF-based controllers have been developed in recent years. However, these approaches sometimes fail to account for the correct amount of uncertainty due to the typical assumptions they make on the uncertainties' structures and bounds.
Our work most closely aligns with recent research that use data-driven approaches to tackle the issue of model uncertainty in nonlinear controllers. Our works builds on [20], where reinforcement learning is used to account for uncertainty when performing feedback linearization of nonlinear systems. In [2], an episodic learning algorithm is introduced which estimates the relationship between the control input and the time-derivative of the CLF. This information is then incorporated into a CLF-QP which calculates an approximation to the min-norm via online optimization. In contrast, our approach directly learns the min-norm stabilizing controller for the system by incorporating the timederivative of the Lyapunov function into the penalty term of the policy-optimization problem.
There have also been extensive efforts to learn optimal stabilizing control laws for systems without the use of a nominal dynamics model [21], [22]. These methods pose an infinite horizon optimal control problem whose solution is known to yield a stabilizing controller for the plant. The problem is solved online, typically using off-policy reinforcement learning methods. During the learning process, a value function for the optimal control problem is constructed, which can be thought of as a CLF for the learned controller. While these methods are quite general, the advantage of our approach is that it enables the system designer to more directly design the desired closed-loop behavior of the system.

B. Organization
The rest of the paper is organized as follows. Section II revisits Control Lyapunov Functions. Section III presents the proposed learning problem, develops our theoretical guarantees, and then demonstrates how the discrete-time approximations to the problem can be solved using reinforcment learning and an approach similar to [20]. In Section IV the proposed method is used to stabilize a double pendulum and the walking gait of an underactuated nonlinear bipedal robot. Finally, Section V provides concluding remarks.

II. CONTROL LYAPUNOV FUNCTIONS
Throughout the paper we will consider affine control systems of the formẋ where x ∈ R n is the state and u ∈ R m the input. The mappings f : R n → R n and g : R n → R n×m are both assumed to be Lipschtiz continuous and we assume that f (0) = 0. We say that continuously differentiable V : R n → R is a Control Lyapunov Function (CLF) for (1) if it is radially unbounded and for each x ∈ R n \ {0} we have where σ : R n → R is locally Lipschitz continuous, positive definite and specifies a minimum acceptable rate of energy dissipation for the system. We recall that a function V is radially unbounded if V (x) → ∞ as x → ∞. It is wellknown that if the above conditions are satisfied then the system is asymptotically controllable in the sense that the state can be driven to the origin asymptotically for every initial condition. For many physical systems it is desirable to find a locally Lipschitz continuous feedback rule u : and the closed loop system is asymptotically stable. It should be noted that not all systems which satisfy (2) admit such a controller [23], but a number of important systems such as the ones considered in this document are continuously stabilizable. One popular choice of control law which satisfies the dissipation constraint (3) is the min-norm control law u * : R n → R m which is defined point-wise by: At every point, this controller selects the smallest input which satisfies the CLF constraint. If V is a CLF for the system, a sufficient condition for u * to be locally Lipschitz continuous is that f , g and the gradient of V are each locally Lipschitz continuous [24]. Moreover, letting the min-norm has the following closed-form representation: However, one advantage of formulating the min-norm controller as as a point-wise optimization as in (4) is that the optimization can easily incorporate bounds on the allowable control efforts for the system by restricting u ∈ U ⊂ R m in the optimization. This is important in many applications where the actuators of the system have physical limitations.
In the near future we hope to extend our the approach presented below to allow for constraints on the inputs.

III. LEARNING MIN-NORM STABILIZING CONTROLLERS A. Learning a Min-norm Stabilizing Controller for a System with Unknown Dynamics
Despite the wide-spread utility of the CLF-based controllers introduced in the previous section, the primary drawback of these methods is that they require an accurate dynamics model to implement. Our present objective is to learn a min-norm stabilizing controller for the planṫ with unknown dynamics, while ensuring that the learned controller adheres to the dissipation constraint imposed by some candidate CLF V : R n → R ≥0 and associated decay rate σ : R n → R ≥0 . In practice, our choice for these functions will likely be informed by a nominal dynamics which incorporates any information we have about the plant, but may be inaccurate due to nonlinearities which are difficult to model or dynamics parameters which are challenging to identify. The inputs and states for the plant and model are both assumed to have the same dimension as in (1), with domains and codomains of the vector fields f p , g p , f m and g m defined appropriately. We will focus on learning the min-norm controller for the plant on a compact subset of the state-space. Specifically, we will focus on learning the min-norm stabilizing controller for the system on the set where c > 0 is a design parameter. We will make the following technical assumptions throughout the paper unless otherwise specified: Assumption 1: The components f p , g p , σ and ∇V are each Locally Lipschitz continuous.

Assumption 2: There exists a locally Lipschitz continuous control lawũ
Assumption 2 ensures that V is a true CLF for the system with associated dissipation rate σ. However, since we do not know the dynamics of the plant (8) we cannot directly compute the associated min-norm controller. However, under Assumption 1 and recalling our discussion in Section II, we know that there is a well-defined control law u * p : W c → R ≥0 which asymptotically stabilizes the plant on W c and is given point-wise by We will denote our learned approximation for u * p bŷ u : R n × Θ → R m . For each choice of parameter θ ∈ Θ ⊂ R K the control lawû(·, θ) : R n → R m defines the learned control law supplied to the plant. It is assumed thatû(·, θ) is locally Lipschitz continuous for each θ ∈ Θ and thatû(x, ·) is continuously differentiable for each x ∈ R n . Common function approximators such as feedforward neural networks, radial basis functions or bases of polynomials can be used to construct the learned controller.
Remark 1: In general, the learned controller can incorporate information from the nominal dynamics model by giving it the structureû where u m is a nominal model-based controller and δu : R n × R K → R m characterizes the "gap" between the min-norm controller for the model (9) and plant (8). While our approach can naturally incorporate information from a nominal dynamics model, there is no requirement to do so, as the optimization problem we formulate below can be solved in a completely model-free fashion. Next, in order to find parameters for the learned controller which satisfy the dissipation constraint (11), we will solve optimizations over the parameters of the learned controller of the form where for each λ ∈ R ≥0 we define the loss function where X is the uniform probability distribution over W c , the mapping ∆ : R n × Θ → R is defined by and finally H : R → R ≥0 is defined for each y ∈ R by The first term in the loss L λ encourages small control efforts while the second term penalizes violations of the CLF dissipation constraint, with λ ∈ R ≥0 used to control the magnitude of the penalty. The uniformity of the distribution X ensures that all points in W c are considered when optimizing over the parameters ofû. While we do not know ∆(x, θ) a priori, we can measure this quantity by applying the controlû(x, θ) to the plant at the point x and measuring the resulting time derivative of V . Then, equation (15) can be used to compute the desired quantity. Thus, any stochastic optimization algorithm can by used to solve P λ by running experiments to evaluate the terms in L λ . We will discuss this in further detail when we present practical approaches for solving P λ below.

B. Theoretical Results
We now study how the solution set of P λ changes as the penalty parameter λ is increased and derive conditions under which the problem is convex, meaning that it can be solved reliably to global optimality using iterative gradient-based optimization algorithms. To simplify the statement of our results, for each λ ∈ R ≥0 we define to capture the set of global minimizers for P λ . We also define to be the set of parameters for which the corresponding learned controller satisfies the desired CLF dissipation constraint at every point in W c . Next, we present our theoretical results in Lemma 1 and Theorems 1 and 2, whose proofs can be found in the Appendix at the end of the document. First, we compare the sets Ξ and S λ as the penalty term λ is increased: Lemma 1: Assume that Ξ is non-empty so that there exists at least one choice of learned parameters which satisfy the desired CLF constraint. Then there existsλ ∈ R ≥0 such that for each λ >λ all global optimizers of P λ also satisfy the dissipation constraint, namely, S λ ⊂ Ξ.
In other words, if the penalty parameter λ ∈ R ≥0 is chosen to be large enough then P λ recovers the set of learned parameters which stabilize the plant and satisfy the CLF constraint. Note that if θ * ∈ Ξ is one such choice of parameters then it must be the case that E x∼X λH(∆(x, θ * )) = 0. Thus, when Ξ is non-empty and λ is chosen to be large enough the minimizers of P λ are selected by the set of parameters which minimize the term E x∼X u(x, θ) 2 2 , which is the average control effort exerted over the state-space by the corresponding learned controller. By definition, the minnorm stabilizing controller u * p minimizes the control effort needed to satisfy the CLF dissipation constraint at every point in the state-space. Thus, if λ is large enough and u * p is in the space of learned controllers spanned byû it must be recovered by the optimization: Theorem 1: Assume that there existsθ ∈ Θ such that u(x,θ) = u * p (x) for each x ∈ W c . Then there exists λ ∈ R ≥0 such that for each λ >λ and θ * ∈ S λ we havê u(x, θ) = u * p (x) for each x ∈ W c . However, the family of optimization problems we have formulated over the parameters of the learned controller will generally be non-convex, meaning that we cannot efficiently find their globally optimal solutions. Thus, we seek conditions under which P λ becomes convex so that we can reliably find its global minimizers using the iterative methods. Towards this end we will now assume that our learned controller is of the form where {u k } is a set of locally Lipschitz continuous mappings from R n to R m and θ k is the k-th entry of the learned parameter 1 . Linearity assumptions of this sort are common in convergence proofs found in both the adaptive control and reinforcement learning literature. In the statement of the following result we informally view each basis element u k as a subset of C(W c , R m ), the linear space of continuous functions from W c to R m . Lemma 2: Assume thatû is of the form (19) and that the set {u k } K k=1 is linearly independent on C(W c , R m ). Then for each λ ∈ R ≥0 the loss L λ is strongly convex.
This leads to the main theoretical result of this paper, which follows from an immediate application of Theorem 1 and Lemma 2.
Theorem 2: Assume that u is of the form (19) and that the set {u k } K k=1 is linearly independent on C(W c , R m ). Further assume that Θ ⊂ R K is convex and that there exists θ * ∈ Θ such thatû(x, θ * ) = u * p (x) for each x ∈ W c . Then there existsλ ∈ R λ ≥ 0 such that for each λ >λ the problem P λ is a strongly convex optimization problem with θ * its unique global minimizer.
There are many well-known bases such as radial basis functions or polynomials which can approximate continuous functions on compact spaces to an arbitrary degree of accuracy by including enough elements in the basis. Thus, by constructing {u k } K k=1 using such a basis and choosing K to be large enough, we can theoretically recover u * p to a pre-defined degree of accuracy by solving P λ with λ ∈ R ≥0 sufficiently large. However, in practice the number of elements required in such an expansion can quickly become prohibitively large as the dimension of the state grows. Thus, for high dimensional systems, such as the bipedal robots we consider below, practical implementations may require the use of more compactly represented function approximation schemes, such as multi-layer feed-forward neural networks, which can also approximate continuous functions to a desired degree of accuracy (Universal Approximation Theorem [25], [26]), but do not possess the structure (19) needed for our theoretical proofs.

C. Solving Discrete-time Approximations with Reinforcement Learning
In this section we discuss how to solve approximations to the problem P λ using policy optimization techniques from reinforcement learning. The approach uses a finite difference approximation of the derivative of the candidate Lyapunov function to approximate the continuous-time loss function. Our description of this process will be brief, since the approach is similar to the one described in [20].
For the reinforcement learning problem we will assume that the control supplied to the plant can only be updated at a fixed minimum sampling period ∆t > 0. We will let t k = k × ∆t for each k ∈ N denote the set of sampling intervals. When the controlû(x, θ) ∈ R m is applied over the interval [t k , t k+1 ] a Taylor expansion can be used to show that (20) Thus for small ∆t the point-wise loss is well-approximated byl We use this approximate point-wise loss to define the following reinforcement learning problem, which serves as an approximation to P λ : Here, the curve x : R → R n is the trajectory of the plant starting from initial condition x(0) = x 0 , and N ∈ N is the number of time steps in each rollout. Probing noise can be added to the input to encourage exploration, e.g, by instead setting u k =û(x k , θ) + w k where w k ∼ N (0, σ 2 I) is zero mean random noise. This is a standard form for reinforcement learning problems, and (22) can be solved using standard on-policy and off-policy reinforcement learning algorithms [27]- [30]. Note that in the special case that N = 1 the cost incurred when solvingP λ approaches the cost incurred when solving P λ as ∆t → 0. In future work we plan to more formally study the relationship between these two problems.

A. Double Pendulum
We first use our approach to learn a controller which stabilizes the double pendulum depicted in Figure 2 to the upright position, using the input-output linearizationbased CLF design approach introduced in [9] to design the candidate CLF for the learning problem. The system has two generalized coordinates q = (q 1 , q 2 ) which represent the angles that each of the arms make with the vertical. The system is actuated by motors at each joint. The dynamics of the system are Lagrangian and thus of the form where M (q) is the mass matrix, H(q,q) collects the gravity and Coriolis terms and τ ∈ R 2 are the joint torques supplied by the motors. This can be put into a state space representation of the forṁ with state x = (q 1 , q 2 ,q 1 ,q 2 ) T and input u = τ .
As depicted in Figure 2a) the system is parameterized by the masses of the two arms m 1 , m 2 as well as their lengths, l 1 , l 2 ∈ R. For the purposes of simulation, we set m 1 = m 2 = l 1 = l 2 = 1. To set up the learning problem, we assume that we are given an inaccurate dynamics model with inaccurate estimatesm 1 ,m 2 ,l 1 ,l 2 . Specifically, we setm 1 = m 2 =l 1 =l 2 = 1 2 so that each of the parameter estimates are half of their true value.
Using the input-output linearization design technique from [9], we design a CLF for the system of the form V : where I is the 2×2 identity matrix and by setting the desired dissipation rate to be σ(x) = x T x. This can be shown to be a valid CLF for both the inaccurate dynamics model and the true plant. We focus on learning the min-norm controller for the plant on the set W c = {V (x) ≤ c} with the design parameter c = 2 and construct our learned controller by settingû where u m is the min-norm CLF controller computed using the inaccurate dynamic parameters and the learned augmentation δu is comprised of a linear combination of 500 radial basis functions so as to match the assumptions of Lemma 2. We trained the learned component using a policy-gradient algorithm with action conditioned baselines [27]. Each training epoch consisted of 50 1-step roll-outs and a total of 500 epoch were used. The time step for the simulator was 0.05 seconds. The performance of the ultimate learned controller is depicted in Figure 2, where we see that the learned controller closely matches the behavior of the true min-norm controller for the system. To further evaluate the performance of the learned controller, we randomly selected 1000 states {x i } 1000 i=1 in W c and calculated the ratio where u * p is the true min-norm controller for the system and θ * is the parameter selected by the training process. We calculated R = 0.044, indicating that the learned controller was able to closely match the performance of the true minnorm controller for the system. As depicted in Figure 1, the learning converges in about 200 iterations, which corresponds to about eight minutes of data. Our implementation of the learning algorithm for this problem was hand-coded, and we believe the sample efficiency for this problem could match that of the walking example below by improving the implementation.

B. Bipedal Walking
Next, we discuss how to apply our method to the Hybrid Zero Dynamics (HZD) framework using the CLF-based design approach proposed in [9] in order to learn an efficient, stable walking controller for a bipedal robot. We model the robot as a hybrid system with impulse effects as in [9], Σ : where η ∈ X ⊂ R na represents the controlled (actuated) states, z ∈ Z ⊂ R nu represents the uncontrolled states and u ∈ U ⊆ R m represents the control inputs. The model assumes alternating phases of single support, where one foot is off the ground (swing foot) and the other (stance foot) is assumed to remain at a fixed point without slipping. The impact between the swing foot and the ground is modelled as a rigid impact and occurs when (η, z) ∈ S, where S is a smooth switching manifold. Here, η + ∈ X and z + ∈ Z represent the post-impact states while η − ∈ X and z − ∈ Z denote the pre-impact states.
Following the framework in [9], an input-output linearization based CLF is designed for the actuated coordinates. Namely, we design a Lyapunov function V : R nu → R and dissipation rate σ : R nu+na → R such that the following condition holds for each (η, z) ∈ X × Z: As shown in [9], by appropriate choice for V and σ, we can obtain exponential convergence of the actuated states to the origin and, even more, we can guarantee a rapid enough convergence rate that achieves stability of the hybrid system. In that case V is called a Rapidly Exponentially Stabilizing Control Lyapunov Function (RES-CLF) and we note that by driving η → 0 a stable walking motion is produced. We refer the readers to [9] for more details on this procedure. Thus, the control objective is to drive only the actuated states to zero. During the design process, the coordinates (η, z) are chosen such that the manifold {(η, z) ∈ X × Z : η = 0} contains an exponentially stable periodic orbit for the hybrid system.
To accommodate this new objective, our goal is to learn a control law u : X × Z × Θ → R m such that ∇V (η)[f p (η, z) + g p (η, z)u(η, z, θ)] + σ(η, z) for each (η, z) ∈ X ×Z for our choice of learned parameters θ ∈ Θ. Here, f p and g p are the terms in true dynamics of the plant, which may differ from the nominal dynamics in (28). To modify our approach to this new setting, for each λ ∈ R ≥0 we now define the losŝ L λ (θ) = E (η,z)∼X u(η, z, θ) 2 2 + λH(∆(η, z, θ)), (31) where X is now the uniform distribution over X × Z. Despite the fact that the CLF is defined only over the lower dimensional state η, the theoretical results from section III-B naturally extend to this case. Moreover, the techniques from section III-C can be used to find local minimizers ofL λ .
In particular, the proposed method is validated on a model for RABBIT [31], an under-actuated five-link planar bipedal robot with seven degrees of freedom. Model uncertainty is introduced by scaling the mass of each of RABBIT's links by a factor of two i.e. the real plant's masses are twice the nominal model's masses. Our learned controlled takes the formû where u m is the min-norm CLF controller obtained using the nominal model dynamics. The term δu(η, z, θ) ∈ R 4 takes the form of a Multi-Layer Perceptron (MLP) neural network with 2 hidden layers of width 64 each, tanh activation functions and layer normalization. We use the Soft Actor Critic algorithm [32], an off-policy method, for training the learned policy δu(η, z, θ).
The training is done on episodes consisting of one walking step each. The simulations are conducted on the open-source physics simulator PyBullet [33] using a discrete time step of one millisecond. As it can be seen in Fig. 3, the training converges in about 20,000 time steps, which corresponds to roughly 50 episodes, taking only about 10 minutes of computation using the six cores of an Intel(R) Core(TM) i7-8705G CPU (3.10GHz), without using a GPU. Figure 4 shows a comparison between the proposed learned controller, the nominal controller u m and the ground truth u * p , which is the CLF-based controller of the plant computed using the true (unknown) dynamics. This figure shows that while the nominal controller fails after ten walking steps making the robot fall, the learned controller achieves stable walking for an indefinite number of steps and gives good tracking error performance. It is also important to notice that the learned controller achieves this while using similar magnitudes of control inputs as the nominal and the true CLF-based controllers.
From Figure 4, we note that the neural network policy δu(η, z, θ) has converged to a local minimum, which gives a performance that is evidently higher than that of the nominal controller, while using inputs of smaller magnitude than those of the actual CLF-based controller of the plant. However, the tracking error performance is not as good as with the actual CLF-based controller of the plant, as expected. It is remarkable that, by the means of using an off-policy reinforcement learning algorithm and incorporating knowledge from a model -which is nonetheless very Trajectories corresponding to different initial conditions for the learned controller and true min-norm controller for the system. Each color represents trajectories starting from a specific initial condition. Solid lines denote the trajectories generated by the true min-norm controller for the system while the dashed lines correspond to the trajectories generated by the learned controller. Observe that the learned controller closely matches the desired closed-loop behavior. Note that the velocities of the trajectories are not depicted, which is why several of the plotted curves intersect.  data (about 50 steps of walking) the proposed learning framework has been able to obtain a controller that achieves stable walking. Additionally, we note that the walking speeds for the learned controller and the true minnorm CLF controller for the plant are different. Underacted robots such as RABBIT may contain multiple periodic orbits on the surface {(η, z) ∈ X × Z : η = 0. Thus, while both controllers successfully drive the system to this set, the periodic orbits the two controllers converge to are different.

V. CONCLUSION
In this paper, a framework to learn min-norm stabilizing control laws for unknown dynamical systems was proposed.
By including basic information of the structure of the system into the learning process through Control Lyapunov Functions, our method was able to learn optimal stabilizing controllers for highly uncertain systems with as little as few minutes of data. Learning convergence theoretical guarantees were given for the case of using a linear combination of independent basis functions to construct the learned controller. Finally, the presented approach was proved to be effective and sample efficient not only for simple low-dimensional systems, such as a double pendulum, but also for higherdimensional nonlinear underactuated hybrid systems, such as a bipedal walking robot.

APPENDIX
This Appendix contains proofs for several assertions made in the body of the document.

C. Proof of Lemma 2
To prove the claim, we will first consider the two maps θ → E x∼X u(x, θ) 2 2 and θ → E x∼X λH(∆(x, θ)) separately. In particular, we will show that the first term is strongly convex in θ while the second term is simply convex. The result of the theorem then follows from the fact that the addition of a strongly convex function and a convex function yields a strongly convex function.
First, we rewrite u(x, θ) 2 2 as θ T W (x) T W (x)θ where W (x) = [u 1 (x), u 2 (x), . . . , u K (x)] T collects the basis of control functions. Note that the positive semi-definite matrix W = E x∼X W (x) T W (x) is the Grammian for {u k } K k=1 on C(W c , R m ), and thus will be full-rank and positive definite iff {u k } K k is linearly independent on this space. Collecting these observations, we see that E x∼X u(x, θ) 2 2 = θ TW θ is a strongly convex quadratic function of the parameters.