{'fraction_non_alphanumeric': 0.03790776636437873, 'fraction_numerical': 0.011733356255641036, 'mean_word_length': 4.620289855072464, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Nonholonomic Yaw Control of an Underactuated Flying Robot with Model-based Reinforcement Learning

Nonholonomic control is a candidate to control nonlinear systems with path-dependant states. We investigate an underactuated flying micro-aerial-vehicle, the ionocraft, that requires nonholonomic control in the yaw-direction for complete attitude control. Deploying an analytical control law involves substantial engineering design and is sensitive to inaccuracy in the system model. With specific assumptions on assembly and system dynamics, we derive a Lie bracket for yaw control of the ionocraft. As a comparison to the significant engineering effort required for an analytic control law, we implement a data-driven model-based reinforcement learning yaw controller in a simulated flight task. We demonstrate that a simple model-based reinforcement learning framework can match the derived Lie bracket control (in yaw rate and chosen actions) in a few minutes of flight data, without a pre-defined dynamics function. This paper shows that learning-based approaches are useful as a tool for synthesis of nonlinear control laws previously only addressable through expert-based design.


I. INTRODUCTION
Generating control laws for novel robots is subject to high cost-per-test and imperfect dynamics models, and these deleterious effects are amplified when deriving complex, nonlinear controllers. For example, nonholonomic path planning has been applied to multiple underactuated systems, but requires precise knowledge of the controlled dynamical system and assumptions on resulting motion. In the context of novel robotics, which are often hand assembled, a perfect understanding of the dynamics is improbable. We investigate if the performance of a novel nonholonomic control law can be matched with data-driven, reinforcement learning techniques that do not require any assumptions on dynamics. Model-based reinforcement learning (MBRL) has been successfully applied to a variety of robotics tasks [1], [2], [3], [4]. MBRL techniques iteratively plan a sequence of actions on an data-optimized dynamics model. In this paper we ask the question: can data driven model-based planning effectively plan multiple steps of nonholonomic control?
Nonholonomic control is most known for the example of the parallel parking problem in automobiles -defined by constraints limiting the path a system can take [5], [6]. Multiple applications of nonholonomic control in robotics demonstrate the potential to control these systems with closed loop, nonlinear control laws [7], such as the Lie bracket sequence [8]. Lie bracket control executes a specific, repeated control law in two planes of motion to achieve motion in an only-indirectly addressable control manifold. Pi tc h -Pit ch (b) An illustration of Lie bracket control to produce a change in yaw. The Lie bracket is the manifold of motion between the starting and final position above, yielding yaw = 2 [f, g](x0) at t = 4 . Our simulations show that strict Lie bracket control can achieve angular rates of up to 13.2 degrees/s. This performance is exceeded with a model-based reinforcement learning method. Fig. 1: Above: the novel robot we analyze, the ionocraft [9]. Below: the proposed nonholonomic yaw controller.

Lie-Bracket Control
In this paper, we contextualize our exploration of nonholonomic control and model-based planning with the ionocraft platform, a centimeter-scale flying robot with dynamics similar to a quadrotor [9], [10] i.e. there are four individually actuatable thrusters arranged in a quadrant. An ionocraft is an aerial vehicle which generates thrust using ionic thrusters, characterized in [11]. Quadrotors have control authority over the yaw axis via differential rotor rates that create torque about the z-axis, but the four-thruster ionocraft studied lacks this torque mechanism [12]. A Lie bracket manifold exists in the yaw direction for the studied ionocraft via repeated pitch and roll motions. The derived yaw mechanism for control via a Lie bracket approach is summarized in Fig. 1b.
The contribution in this paper is to show potential of datadriven, model-based RL to improve on peak performance and robustness of analytical nonholonomic control. Specifically, in this manuscript, we 1) derive a new nonholonomic, Lie bracket control law for the ionocraft's yaw axis and 2) demonstrate that model-based reinforcement learning improves on the analytical control policy in terms of yaw rate and crash frequency.

A. Novel Aerial Vehicles
Development of novel actuators and small-scale electronics has progressed to multiple novel micro-aerial vehicles (MAVs) [13], [14]. Variants on flying robots involve robots with spinning propellors, such as a tri-rotor flying robot [15] and others with one spinning motor [16], [17]. Other robots have shown the potential for taking advantage of the flapping wing mechanism found in nature [18], [19]. Robots using ion-thrusters have only recently taken off, and are yet to be controlled in experiment [9], [10]. In contrast with automobiles or walking robots, flying robots have simpler (absence of contact forces), high-speed dynamics, but are not passively stable in the sense that they are prone to crashing, resulting in a challenging control problem.

B. Nonholonomic Robotic Control
Nonholonomic control is a sub-area of nonlinear control theory studying systems with path-dependant states [5], [6]. One method of nonholonomic control uses the Lie bracket, which involves repeated, mirrored actuation on two input manifolds [8], as detailed in Sec. III-A. Nonholonomic control is frequently applied to wheeled robots where the plane of motion must follow the direction of the wheels, leading to the parallel parking problem [20]. An extension of the 4-wheeled parallel parking problem is the controllability of two-wheeled balancing robots [21]. A novel trident snake robot uses nonholonomic control to maneuver with three arms attached to non-slide wheels [22].
There are limited studies of nonholonomic control applied to flying robots (i.e. without surface-contact constraints). Nonholonomic path planning is used for simulated flying vehicles [23], but it does not consider details of low-level dynamics needed for attitude control. A space-robot with a multi-joint arm attached to a main body uses nonholonomic control with a Lyapunav function for path-planning [24] and nonholonomic redundancy to avoid joint limits [25]. Another contactless-medium of motion is swimming, and nonholonomic control is used to imitate carangiform swimming of a planar, rigid fish-robot [26] and to control a AUV [27]. Our work extends the application of nonholonomic, Lie bracket control to that of low-level control of a flying rigid-body.

C. Model-based Reinforcement Learning
Model-based reinforcement learning (MBRL) has emerged as a functional candidate for robotic control in a data-efficient manner [1], [3], [2], [28] -this paper extends the functionality of MBRL to nonholonomic planning for flying robots. Visual MBRL (i.e. control from pixels) has been shown to work in a simple, simulated nonholomic car task with Linear Quadratic Regulator (LQR) control and variational inference [29]. Additionally, MBRL has been shown to learn to control a micro-quadrotor with no assumption on dynamics [30] and with suspended payloads modifying dynamics [31]. MBRL for attitude control of quadrotors is encouraging for datadriven control of a novel MAV, and this work extends the results to show the capability of MBRL algorithms to plan complex multi-step action sequences for flying robots.

III. BACKGROUND A. Lie Bracket Control
For a nonlinear, dynamical system with two inputs, u = u 1 u 2 , a Lie bracket control law is defined by a repeated sequence of motions for a small time, , to generate motion in a new manifold of motion, f, g , called the Lie bracket vector field. Consider the following system, The Lie bracket control law is motivated by a series of Taylor series expansions along the vector fields of each input f, g, so that a third manifold of motion emerges, f, g . Specifically, consider the input sequence below, After t = 4 , the state can be expressed as (shown in [32]) For a visualization of the Lie bracket motion in a phase plane caused by two inputs, see Fig. 1b.

B. Model-based Reinforcement Learning
Model-based reinforcement learning (MBRL) follows the framework of an agent interacting in an environment, learning a model of said environment, and then leveraging the model for control [33]. Specifically, the agent acts in a Markov Decision Process (MDP) governed by a transition function x t+1 = f (x t , u t ) and returns a reward at each step , the agent learns a model, x t+1 = f θ (x t , u t ) to minimize the negative log-likelihood of the transitions. We employ samplebased model-predictive control (MPC) using the learned dynamics model, which optimizes the expected reward over a finite, recursively predicted horizon, τ , from a set of actions sampled from a uniform distribution U(a), as IV. IONOCRAFT MODEL In this paper, we model and study an ionocraft [9], [10]; a flying robot with four individually-addressable electrohydrodynamic thrusters, as shown in Fig. 1a. The robot is approximately 4 cm 2 and masses 30 mg not including the sensor payload. Electrohydrodynamic thrust, summarized in [11], is produced by the momentum-transferring collisions between ions drifting in an applied electric field and the particles in a neutral fluid. The forces, F k (i k , β k ), are models for the electrohydrodynamic force from each thruster [9], as Here, i is the ion current, d = 500µm is the air-gap and µ = 2 · 10 −4 m 2 /V is the ion mobility of N + 2 in dry air. Fitting a β k , which represents a deviation in force from that expected by an idealized one-dimensional derivation, to each thruster allows more flexibility in model fitting and simulation. Past work shows β ∈ [0.5, 0.7] for the region of currents i ∈ [0, .5]mA corresponding to voltages v ∈ [1650, 2100]V [9]. The ionocraft dynamics model follows a 12 state rigidbody dynamics, which are simulated at a dynamics frequency of 1 kHz, a control frequency of 100 Hz to match the experimental setup used in [34], and Gaussian state noise in each dimension, σ ∼ N (0, 0.01): The simulator emulates low-level control by actuating the ionocraft with voltages, which map to currents and forces via Eq. (7). Then, the rigid-body model incorporates torques and forces as The important observation for yaw control is that the zaxis torque is totally decoupled from the individual thrusters. Neglecting the damping forces in low velocity maneuvers such as hovering, a corresponding subset of the Jacobian linearization of the ionocraft approximated around equilibrium, A direct way to see a loss of yaw-actuation is with a linearized controllability matrix,W c ( A, B), A robot can actuate all directions if this matrix is full rank. The matrixW c is of rank n − 1, with a corresponding uncontrollable mode ofω z , but this also propagates toψ.
This loss of controllability formally shows the need for a nonlinear yaw control law.

V. NONHOLONOMIC YAW CONTROL
In this section we derive the analytical Lie bracket controller and compare it to a data-driven MBRL approach for yaw control. With both methods, the goal of the approach is to maximize the yaw, ψ, without causing substantial flight disturbance (e.g. reaching the stop condition in the simulator of the magnitude of pitch, θ, or roll, φ, above 45 degrees).

A. Analytical Control: Lie Bracket
Here we map the Lie bracket formulation outlined in Sec. III-A to yaw control for the ionocraft. First, consider the Euler rates defined in Eq. (11), with the trigonometric functions abbreviated as c := cos, s := sin, and t := tan: To study as nonlinear control, expand the differential equations for Euler rates with angular rates as inputs. In order to assume the angular rates are inputs, we must show that they are directly and independently actuated. To do so, consider the angular rate differential equations shown below, assuming the inertial matrix is diagonal: I B = diag(I xx , I yy , I zz ) .
The following assumptions are needed to prove the existence of the Lie bracket yaw controller: 1) For a symmetric body in the XY -plane, I xx = I yy , so the cross terms ofω z are equal, ω y ω x I xx = ω x ω y I yy .
2) The robot starts from rest, ω z (t 0 ) = 0, so the cross terms inω x ,ω y are zero initially. 3) With the lack of yaw coupling in thrusters, the robot applies 0 torque about the z axis, τ z = 0. Combining assertions 2) and 3) enforces ω z = 0∀t. With 1), this means thatω x ,ω y are fully determined by τ x , τ y , so we can model the flight dynamics as a nonlinear system with two manifolds of action, as in Eq. (1). For the flying robot, define the state space with the following state variables and vector fields.
Making the assumptions above guarantees that h(x) does not affect the differential equation, and there is no direct yaw actuation. Consider two Lie brackets f, g and g, f , as defined in Eq. (4).
Computing the Jacobian of our dynamics f and g yields, Finally, the Lie bracket is computed as: The final assumption is that the control law will be applied near hover, θ, φ ≈ 0. This approximation shows controllability in the yaw-direction with low pitch or roll drift captured in the higher order terms, O( 3 ) in Eq. (4) -completing attitude control for the ionocraft.
A visualization of the resulting sequence is shown in Fig. 2. Intuitively, the robot actuates yaw by repeated, mirrored pitch-roll oscillations, but the assumptions on symmetry are highly sensitive to the reality of imperfect assembly outlined in Table II. Now, we define the actions used by the Lie bracket sequence to create the specific control law for the ionocraft.
These actions are designed given flight mass of 50mg to assume a payload such as an IMU and a thrust coefficient, β = 0.6 (equilibrium is the action with all motors set to output of F i = mg 4 ). A hard-coded sequence of these actions at 100 Hz ( = 0.01) achieves a baseline yaw rate of 1 degrees/s. This yaw rate can be increased by a corresponding increase in the time that each action is held, . Table I shows that the raw rate can increase with the Lie bracket controller up to 13.2 degrees/s, but faster yaw actuation causes faster instability in pitch and roll. To slow divergence and get peak yaw-rate with the Lie bracket, we remove all state noise, σ = 0, and zero the initial state, x(t 0 ) = 0.
Translating control methods to robotic hardware strongly correlates with the practical value of a method for control synthesis. Lie bracket control is based on careful assumptions of symmetry in movement and near-equilibrium flight, which can easily break down due to environmental disturbances or nonideality in robot assembly. Specifically, the ionocraft needs an on-board MPU-9250 inertial measurement device  (IMU) placed in the center for state-space measurement. This assembly is done by hand and subject to errors, which can have substantial alteration on the inertial properties of the robot -summarized in Table II. The hand-assembly also conveys an increased cost-per-test, so the ideal method for generating a controller would be able to generalize across inaccurate dynamics and be sample efficient.

B. Data-driven Control: MBRL
The main shortcoming of analytical methods is the strong assumptions on system dynamics, so we explore model-based reinforcement learning. While free from assumptions on system dynamics, the MBRL framework (following closely to [30]) has a new cost of environmental interactions that we quantify and consider. The predictive dynamics model is a simple feed-forward neural network with two hidden layers. In this paper, the hyperparameters (such as the hidden width, learning rate, batch size) required little tuning -they were sourced directly from [30]. A parameter that can strongly affect performance is the prediction horizon of the MPC, and in this paper, we use a horizon of τ = 5. In this section we detail the design decisions that must be made to achieve nonholonomic behavior with a model-based controller, which primarily entails generating a suitable reward function.
A simple formulation for achieving yaw in a reinforcement learning framework would be to encourage maximizing the magnitude of yaw, such as r(x, u) = ψ, but this reward function does not penalize a loss of attitude stability. In practice, this reward function quickly diverges and hits the simulator pitch, roll stop condition of |θ| > 45deg. or |φ| > 45deg.. To mitigate this, the objective becomes a dual optimization with the reward function balancing the reward of yaw versus the cost of pitch and roll. Simply appending a scaled squared cost on pitch and roll (e.g. r(x, u) = ψ 2 − λ(θ 2 + φ 2 )) does not achieve stable yaw control because as yaw increases globally, the controller accepts a corresponding increase in pitch and roll to increase reward -eventually the robot reaches high angles and ends the simulation early in 30% of trials, even when trained on minutes of flight data.
To alleviate this, sliding mode control [35] can ensure attitude stability while maximizing yaw rate when in a stable region of the state space, as This introduces a tune-able hyperparameter, the attitude window, η, weighting the balance between attitude stability and yaw rate that was set to 10 degrees during our experiments. With this reward function, agents never reach the simulation stop condition when trained on at least 20 s of flight data. In order to test the robustness of the MBRL approach to manufacturing and environmental disturbances that substantially degrade the performance of the Lie bracket control law, we randomized the inertial properties to ±15% the nominal value and sample initial Euler angles φ, θ ∈ [−22.5, 22.5]degrees. The performance with this randomization matches closely to that with initial state set to zero, x 0 = 0, and symmetric dynamics, showing the ability for the MBRL approach to adapt to variable environments, shown in Fig. 4. Each trial represents a maximum of 1000 training steps, so the full training set after 10 trials represents just 100 s of training data. The learning agents learn (with and without assembly and environmental variation) mean behavior just below the Lie bracket baseline (in variation free environment), and the MBRL agents have many trials with a substantially greater yaw rate. A simulated flight, where a MBRL agent uses pitch and roll oscillations similar to a Lie bracket control law is shown in Fig. 3.

C. Learning to Mimic the Lie bracket
An important point when considering the learned behavior is if it mimics the Lie bracket control, or does the MBRL algorithm develop unintended, simulation-exploiting maneuvers. To test this, we train the MBRL algorithm with a constrained action space of the actions used in Lie bracket control and equilibrium: , with 65 th , 95 th percentiles shaded, per trial when learning yaw control for the simulated ionocraft with a 15% variation on inertial properties and a wide range of initial states. The agent with environmental variation closely matches the one with symmetric dynamics and stable initial conditions, highlighting the ability for MBRL to cope with realistic hardware variation. One trial is equal to 1000 environment steps. The agent learns to stably actuate yaw, with many trials with a yaw rate over the peak derived Lie bracket at 13.2 degrees/s. The standard Lie bracket formulation actuates each action for one time step to mitigate divergence of other states from higher order interactions. In this experiment, the MBRL algorithm closely resembles the action selection of a repeated Lie bracket with some sampling noise -recall that the controller only runs with a planning horizon of τ = 5. We compare the actions selected by the MBRL algorithm and a Lie bracket sequence repeating each action 5 times ( = 0.05) in Fig. 5, showing the ability for a data-driven approach to closely mimic a carefully derived analytical controller.

D. Discussion
We ran experiments on a simulated ionocraft to see which learning approaches can match a complex, multi-step action. A model-based model-predictive controller learned to achieve a maximum yaw rate of up to 31 degrees/s by closely mimicking the Lie bracket control law. It is important to note that the MBRL controller achieves this yaw rate in a full trial, and does not cause diverging attitude stability like that of the high-Lie bracket controllers. With assembly variation and random initial states, the Lie bracket controller reaches the simulator stop condition in over 90% of trials (N = 100) before reaching an episode length of 1000 points (10 s). The MBRL algorithm can maintain stability and yaw actuation in the presence of assembly and environmental uncertainty. Other sources of variation, such as different forces per thruster, as modeled by β k in Eq. (7), are not studied and could further incentivize data-driven techniques for this robot. As another baseline, model-free reinforcement learning agents trained with Soft Actor-critic [36]

VI. CONCLUSION
This paper provided a case study in synthesising a yaw controller for a novel flying robot, the ionocraft. Generating complex, nonlinear controllers on costly robots is a trade-off between engineer time and experimental cost. Lie brackets, and other closed-form nonholonomic controllers, require non-trivial dynamics model understanding, which is a level of expertise often lacking when deploying robotic systems. In this paper we characterized how a proven MBRL approach can closely match a derived controller at the low-level of action selection and in performance. While translating the nonholonomic control law or model-based RL to real experiments poses high risk for different reasons -namely a trade off between expert time and costly environmental interactions -this paper shows that an encouraging new path exists in data-driven control for roboticists that is not dependant on familiarity with a new platform.

A. Lie Bracket Derivation
Begin with any drift-free, control-affine nonlinear system with two inputs.ẋ = f (x)u 1 + g(x)u 2 (31) With two vector fields f, g execute the action sequence in Eq. (2). The goal is to derive a third manifold of motion, as the time-step → 0 When applying a constant single-input to the system shown in Eq. (31), the system takes the form ofẋ = h(x) For use in the Taylor expansion, take the derivative with respect to time of the general form shown in Eq. (32).
The derivative of the vector field h(x) is Jacobian matrix, J ∈ R n×n define by as The Lie-bracket emerges from an interweaving of two vector fields in a Taylor Expansion. Consider the Taylor series in one dimension of the first vector field direction, f , with a time step of .
x(2 ) = x(0) + (f + g) + 2 1 2 Repeat the step of substituting the Taylor expansion of the previous vector field, for a time-step again.
x(4 ) = x(0) + 2 dg dx f − df dx g + · · · (41) Take the limit in a small time step to achieve the Lie-bracket vector field.
The Lie bracket vector field is defined as Which yields the state at time t = 4 :