{'fraction_non_alphanumeric': 0.026635743903296087, 'fraction_numerical': 0.007027900766041184, 'mean_word_length': 5.058973813072174, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Model Predictive Control for Micro Aerial Vehicles: A Survey

This paper presents a review of the design and application of model predictive control strategies for Micro Aerial Vehicles and specifically multirotor configurations such as quadrotors. The diverse set of works in the domain is organized based on the control law being optimized over linear or nonlinear dynamics, the integration of state and input constraints, possible fault-tolerant design, if reinforcement learning methods have been utilized and if the controller refers to free-flight or other tasks such as physical interaction or load transportation. A selected set of comparison results are also presented and serve to provide insight for the selection between linear and nonlinear schemes, the tuning of the prediction horizon, the importance of disturbance observer-based offset-free tracking and the intrinsic robustness of such methods to parameter uncertainty. Furthermore, an overview of recent research trends on the combined application of modern deep reinforcement learning techniques and model predictive control for multirotor vehicles is presented. Finally, this review concludes with explicit discussion regarding selected open-source software packages that deliver off-the-shelf model predictive control functionality applicable to a wide variety of Micro Aerial Vehicle configurations.


I. INTRODUCTION
Micro Aerial Vehicles (MAVs) and especially systems of the multirotor class, such as quadrotors and hexacopters, correspond to a widely adopted type of aerial robot. Such systems are nowadays extensively used for autonomous inspection [1], surveillance [2] and other remote sensing applications, alongside tasks relating to physical interaction [3], delivery [4] and more. Their success is attributed to a variety of factors including their simplicity, low-cost, reliability, and agile dynamics. Naturally, a key component relates to the accuracy and robustness of the controller onboard such systems which alongside the state estimation process are the two most fundamental algorithms necessary to facilitate autonomous navigation.
In response to this fact, a wide variety of control strategies have been proposed for the problem of MAV flight control including both model-free and model-based methods. In the latter, both linear and nonlinear methods have been considered, alongside methods exploiting piecewise system models, techniques tailored to robots undergoing physical interaction, load transportation, and deep neural networks-based reinforcement learning approaches. Among the multiple approaches, model predictive control has seen wide utilization 1 The authors are with the Autonomous Robots Lab, Norwegian University of Science and Technology (NTNU), 7491, Trondheim, Norway huan.nguyen(konstantinos.alexis)@ntnu.no  and has presented outstanding results in terms of trajectory tracking accuracy and robust performance. Figure 1 presents examples of MAVs relying on model predictive control.
Model Predictive Control (MPC) [5][6][7][8][9][10][11] offers a collection of properties of significant importance for MAVs. Being a model-based method, it can exploit knowledge of the dynamics model of the system. Based on the extensive progress in the domain, MPC methods are now feasible both for linear and nonlinear systems, alongside hybrid model formulations. By optimizing over a horizon, MPC can simultaneously optimize towards optimal tracking of the reference trajectory and satisfy input and state constraints, while retaining robust performance. Furthermore, state constraints may not be limited to box constraint formulations but also model 3D obstacles as regions of the navigation space that must be avoided. Additionally, MPC by nature relates to approximate dynamic programming and is very relevant to modern research in reinforcement learning, a fact reflected in a multitude of new works of the community. Moreover, the power of MPC has enabled it to solve complex problems in MAV autonomy such as the recent perception-aware model predictive navigation method in [12].
In this paper we provide a survey with respect to the methods proposed for trajectory tracking control of MAVs of quadrotor, hexarotor and other multirotor configurations. We cover the domains of Linear Model Predictive Control (LMPC) and Nonlinear MPC (NMPC), as well as MPC for aerial manipulation and load transportation, fault-tolerant control, alongside the interconnection between MPC and neural networks-based reinforcement learning approaches. We present selective comparison results which serve to provide design guidelines and further categorize a set of opensource code packages that provide off-the-shelf functionality for deploying MPC onboard micro aerial vehices.
The rest of this paper is organized as follows. A model of the multirotor dynamics is overviewed in Section II. The survey presentation of MPC for MAVs is detailed in  Section III with subsections on linear and nonlinear methods,  strategies for fault-tolerance, load transportation, physical  interaction and works involving deep reinforcement learning.  Finally, Section IV outlines a selected set of open-source  packages, while conclusions are drawn in Section V. II. MODELING OF MICRO AERIAL VEHICLES A set of contributions have provided extensive means to model multirotor MAVs at selective levels of fidelity. As visually depicted in Figure 2, one may account to a different extent for complex aerodynamic parameters, non-diagonal inertia terms and other effects that have been detailed extensively in pioneering studies [13]. This modular approach allows us to simplify, without loss of generality, the subsequent discussion by considering the hexarotor vehicle as a particular instance of a multirotor system, while researchers that build upon this presentation may decide independently of components such as the propeller model. A hexarotor is typically a platform consisting of six identical rotors and propellers symmetrically configured. This propulsion system generates the thrust and torque normal to the plane of the vehicle, as required to facilitate stable control. For the modeling derivations below we choose an inertial reference frame I with unit vectors { I x , I y , I z } and a body fixed frame B with unit vectors { B x , B y , B z }. The origin of B is located at the Center of Mass (CoM) of the hexarotor and is presented in Figure 3. For the rest of this process, let us denote m as the total mass, J ∈ R 3×3 the inertia matrix with respect to B, R IB ∈ SO(3) the rotation matrix representing the vehicle orientation, ω ∈ R 3 the angular velocity expressed in B, p ∈ R 3 the position of the vehicle's CoM in expressed in I, and υ ∈ R 3 the velocity of the CoM expressed in I.
The dominant forces acting on the vehicle are generated from the propellers. Under a set of common and well-proven assumptions, each propeller is considered to generate thrust proportional to the square of the propeller rotation speed and angular moment due to the drag force. For each propeller i, the generated thrust and moment take the form: where n i is the rotor speed of the propeller, k n , k m > 0 are constants, and e z is a unit vector in the z direction.
This level of modeling fidelity for the forces applied on a multirotor is the one most commonly found. However, if we aim to consider dynamic maneuvers, then two additional phenomena come into play. These effects are the blade flapping and induced drag and introduce additional forces in the x-y rotor plane and thus add more damping to the MAV [14]. Combining these effects into one lumped drag coefficient [15], we derive the following aerodynamic force for propeller i: where K drag = diag(k D , k D , 0), k D > 0, and f T,i is the zcomponent of the i-th thrust force. Then the motion dynamics take the form: where F ext represents any external forces acting on the vehicle, and A is the control allocation matrix and N r the number of propellers. The works in [16,17] present the control allocation matrix derivations for the case of symmetric hexarotor and quadrotors respectively. Attitude Subsystem: It is noted that commonly in application, the attitude dynamics of a multirotor platform are controlled with a fast embedded system running a rather simple to calculate feedback loop often only involving fixed-gains. Therefore, MPC is often deployed as a cascale position controller commanding the closed loop attitude dynamics which now should be identified. For that goal, the inner-loop attitude model can be represented as a first-order model due to the efficiency of onboard control and despite its otherwise second-order nature [18]. The closed-loop attitude dynamics to then be identified take the form:

III. MODEL PREDICTIVE CONTROL FOR MAVS
In this section we overview some of the successful methods and strategies of applying model predictive control for MAVs. In particular, linear and nonlinear schemes are presented, methods for physical interaction and load transportation, alongside techniques combining traditional MPC and neural networks-based reinforcement learning.

A. Linear Model Predictive Control
The basic case of application of MPC for quadrotor control relates to linear methods. Furthermore, in the most widely adopted case, Linear Model Predictive Control (LMPC) is deployed to handle the position dynamics of a MAV assuming that an attitude controller is already deployed and an associated closed-loop attitude dynamics model has been identified as described in Eq. (7). Given this model we can proceed to linearize the remaining system dynamics around hover. We define the following state vector and control input: where T ref is the commanded reference thrust, I φ, I θ are the roll and pitch angles expressed in the inertial frame. The following relation with the robot roll and pitch angles holds: Finally, after linearization and discretization the following state-space form holds in which the effect of external forces F ext,k and the disturbance matrix B d are also considered: Provided the above, the LMPC strategy repeatedly solves the following Optimal Control Problem (OCP) assuming that input constraints apply but no state constraints are considered: where Q x 0, R u 0 are the state and input penalty matrices, while P 0 is the terminal state error penalty. Furthermore x ref,k , u ref,k are the target state and target control input respectively at time k. The input constraints take the following form: Provided the derivation of the control law per iteration, the method then applies the first control input u 0 and the whole process is repeated in a receding horizon fashion. Lastly, it is noted that the derived thrust reference vector is nonlinearly scaled to account for the projection of thrust when the system roll and pitch are nonzero: Disturbance Observer: A disturbance observer can be incorporated to the above design for offset-free tracking. This is achieved by augmenting the system model with the disturbances vector. Considering the need to track the system output y k = Cx k and achieve offset-free tracking, a simple observer to estimate such a disturbance takes the form: x k+1 wherex k ,F ext,k , y m,k are the estimated state, external disturbances and measured output at time k, respectively, while L x , L Fext are the associated observer gains. Assuming a stable observer, we can compute the steady-state MPC state x ref,k and control input u ref,k at time k by solving: where r k the output vector reference at time k. Literature Review: The abovementioned derivation corresponds to the most straightforward application of linear MPC for the position control of MAVs. At the same time the research community has explored a much more rich set of methods. Early in the timeline of this research, the authors in [19] proposed the application of such a receding horizon scheme for the attitude control of a quadrotor vehicle and further accounted for state constraints. As the calculation of MPC subject to input and state constraints can be expensive -especially in comparison to the fast attitude dynamics -multiparametric approaches have been investigated for the explicit derivation of the control law [20]. At a similar period, the authors in [21] proposed LMPC methods with integral terms. Aiming to account for the change in the system dynamics when the operating point departs significantly from the hovering point -but still not employing nonlinear methods -the works in [22,23] present a PieceWise Affine (PWA) modeling approach and associated predictive control policy for the full control of a quadrotor MAV. Furthermore, the work in [24] investigated the design of robust MPC methods and presented extensive disturbance rejection capabilities including the ability to handle slung load disturbances. Currently, LMPC methods have presented significant success and have managed to be utilized reasonably extensively at least in multirotors in research labs as also visible in the discussion for open source packages in Section IV. Connecting the domain of linear and nonlinear MPC approaches, the work in [25] offers a flatness-based approach which exploits feedback linearization and provides agile flight capabiltiies across the flight envelope but with the often reduced computational cost of linear methods. Reachability Analysis: When safety-critical applications are considered, guaranteed control performance is necessary. Generally, for a dynamic system, the reachable set R for a time t, inputs u, disturbances w and a set of initial states S is the set of end states of trajectories starting in S after time t [26]. Despite the importance of reachable set analysis for MPC controllers, the literature in MPC application for MAVs mostly lacks such considerations. Few directly or indirectly relevant exceptions have examined the problem either directly from a MPC standpoint or with regards to learning-based methods [27,28], yet it is believed that the domain deserves further attention.

B. Nonlinear Model Predictive Control
Linear control methods are appealing due to their simplicity and often reduced computational needs. Long experience in the community has indicated that when a multirotor MAV is largely operating around hovering/small-angles then LMPC methods provide high performance and robustness. However, nonlinear control has to be utilized if the complete flight envelope of the system is to be exploited.
Towards that goal we derive a baseline formulation for Nonlinear Model Predictive Control. We consider the following state and control vectors: This in turn allows us to formulate the nonlinear OCP: where f is composed by Eqs. (3) (4) (7). The controller is implemented in a receding horizon fashion, where this optimization needs to be solved in real-time. As typically this corresponds to a computationally expensive task, especially for the fast dynamics of MAVs and the often limited onboard computational capabilities, direct methods [18] have gained significant attention due to their reduced processing needs. Multiple shooting techniques in particular have been used to solve Eq. (20) [18] with the system dynamics and constraints being disccretized over a coarse discrete time grid t 0 , ..., t N within the interval [t k , t k+1 ] and for each interval solving a Boundary Value Problem where additionally continuity constraints are imposed. Disturbance Observer: Analogous to the case of LMPC, we can estimate the external disturbances F ext . This is now achieved through an augmented state Extended Kalman Filter (EKF) that includes the external forces. The EKF uses the same model as in control design but further incorporates the heading angle. The external force estimation in turn incorporates modelling errors and supports offset-free tracking. Literature Review: Beyond this baseline formulation of NMPC for MAVs, the research in the community has investigated further problems. The contribution in [29] considers general MAV designs and an enhanced actuator model for improved tracking performance. The work in [16] examines the problem of applying NMPC directly for the inner attitude dynamics of the system. The authors in [30] present a NMPC approach formulated on the Special Euclidean group SE(3), which has a single optimization layer and offers safe trajectory tracking with obstacle avoidance capacity. The work in [31] explicitly considers the role of input constraints in NMPC design for multirotor MAVs. Towards agile performance combined with lightweight computational needs, the work in [32] presented a method for real-time, unconstrained NMPC that combines trajectory optimization and tracking control in a single, unified approach. It uses an iterative optimal control algorithm -namely Sequential Linear Quadratic -in the MPC setting to solve the underlying nonlinear control problem and simultaneously derive the optimal feedforward and feedback terms. The authors demonstrate that the solver can generate trajectories with a duration of multiple seconds within only a few milliseconds. Focusing on the problem of collision-free flight, the contribution in [33] applies NMPC for the problem of obstacle avoidance for a quadrotor aerial vehicle. Similarly, the work in [34] utilizes NMPC to enable the avoidance of complex obstacles including those with non-convex shape. Considering the specific need of carrying external payloads, the work in [35] applies NMPC for slung load oscillation suppression for a quadrotor MAV.

C. Comparison of Linear and Nonlinear MPC
As free-flight control is the main control task for a multirotor MAV, in this section we present a comparison of two baseline linear and nonlinear MPC approaches for the position tracking problem of a hexarotor MAV.
More specifically, the Linear and Nonlinear MPC controllers' performance are compared using the C++ implementations presented in [36], with the simulated model being an AscTec Firefly hexacopter based on the RotorS open-source simulator [37]. The weight matrices Q x and R u are chosen the same for both controllers, while the terminal  .0cm, respectively. The performance of the linear MPC with parameter uncertainty, in this case the mass parameter, is also verified and the results are illustrated in Figure 5. It can be seen that even though the responses in x, y axes are not affected much, there is offset in the z axis response when the mass of the system is incorrect which necessitates to incorporate a disturbance observer in practical use.  It is known that the number of prediction steps in the MPC problem can greatly affect the feasibility and stability of the closed-loop system. Specifically, increasing the prediction horizon leads to larger region of attraction [38]. The responses of the closed loop system with different prediction horizons and input signal described in Figure 5 are illustrated in Figure 6 and the RMSE values are given in Table I. It can be seen that reasonably increasing the number of prediction steps improves the tracking performance. However, solving the MPC problem with larger prediction horizon requires more computation time as described in the box plot in Figure 7. The outlier values denoted by red crosses in Figure 7 correspond to the cases when the control inputs are close to the limits, which require the solvers to take more iterations to find the solutions. Interestingly, the nonlinear MPC solver based on [39] has smaller computation time compared to the linear MPC solver based on [40]. Step responses of Linear and Nonlinear MPC with different prediction horizons (N = 10, 20, 30Tp with prediction step Tp = 0.1s).

D. Fault-Tolerant MPC
Fault-tolerance is an essential property of every control scheme. As MAVs can undertake critical roles, while their airborne nature makes them potential risk factors, assessing the fault-tolerance of their flight control is particularly important. The work in [41] has demonstrated the potential to retain full or partial controllability of a quadrotor MAV degrees-of-freedom even subject to the loss of one, two or even three propellers. Naturally, more control re-allocation options arise with MAVs integrating additional actuators (e.g., a hexacopter). In terms of MPC work, the contributions in [16,42,43] demonstrated -following different designs -the inherent capability of NMPC to retain dynamic stability for a symmetric underactuated hexacopter subject to propeller loss. Furthermore, the contribution in [44] demonstrates the application of NMPC for a hexarotor with three motor failures. The authors in [45] investigate the role of partial loss of control effectiveness in the actuators of a quadrotor and apply MPC with terminal constraints to enable the accurate reference tracking despite the considered faults. A fault detection and diagnosis system is designed to assist MPC in its task. It is considered that the importance of integration of MAVs in safety-critical applications or the national airspace will increase the importance of faulttolerant predictive control design.

E. Deep Reinforcement Learning
MPC, which aims to find a solution of the constrained finite-horizon optimization problem, is closely related to Reinforcement Learning (RL), which learns how to make sequential decisions to maximize a numerical reward signal through trial-and-error search [46]. The interactive nature of RL combined with the approximation ability of neural networks, allow the replacement of each component in the MPC scheme (or part of it) with this powerful representation. The works in [47,48] derive the terminal and transition cost functions from the value function which is learned by rolling out the current policy and collecting reward signals. This reward signal can be a binary or sparse reward which opens the opportunity to remove the need for hand-tuning the cost matrices in MPC [49]. The authors in [50] use a neural network to learn the dynamic function of the system, while the contribution in [51] proposes a deep quantile regression framework for learning bounds on distributions of trajectories, demonstrated to generate an obstacle avoidance path for a full-state quadrotor model subject to action noise. The computation cost for solving the MPC problem can be high with long prediction horizon, rendering it impractical to be applied to many real-time control problems and in such cases, deep RL can be used to compress the MPC policy. The work in [52] uses an expert MPC in guided policy search to control a MAV which not only reduces the computation time compared to that of the expert MPC but also removes the need for an explicit state estimation. The authors in [53] propose a constrained neural network architecture to imitate an explicit MPC law and then a policy gradient method -with the advantage function calculated by utilizing the terminal cost function in a MPC problem -is developed. It is noted that the use of neural networks to represent the optimal policy in critical constrained optimization problems necessitates the need for verification methods to validate the performance of the close loop systems. The work in [54] demonstrates computing the 10-step forward reachable set of a 6D quadrotor model controlled by a neural network using Semidefinite Programming.

F. Load Transportation
Analogous to their manned counterparts, micro aerial vehicles are considered for load transportation tasks [55]. Despite the robustness of MPC and especially of certain design variations of it [24], special control design is necessary for high-performance load transportation using one or more multirotor systems. The work in [56] presents a method for cable-suspended load transportation using a quadorotor vehicle. The authors in [3] present aerial pick-and-place relying on MPC methods. Considering the benefits of tiltrotor systems, the works in [57,58] propose MPC methods for load transportation. As during a slung-load operation, it is not only the aerial robot that can collide with the world but also the load itself, the contribution [59] explicitly derives safe paths for load transportation operations. Considering the potential of multi-robot synergy in load transportation, a possible MPC design is presented in [60] for two vehicles, while a more general problem formulation is detailed in [61].

G. Physical Interaction
MPC methods have also found their way in the context of research work relating to aerial robots physically interacting with their environment. The authors in [62] derive a hybrid systems-based formulation of a quadrotor that either navigates in free-flight or comes in contact with the environment in order to perform inspection tasks. The work first utilizes a linearized model for the position dynamics of the quadrotor in free-flight given the system identification of the closedloop attitude dynamics. This is combined with a linear model of the system in contact with the environment by accounting for the force applied from the physical surfaces. The applicability of hybrid systems relates to the fact that collision-dynamics are particularly fast and thus allow to handle them as nonsmooth effects instead of stiff differential equations [63]. A broader illustration is depicted in Figure 8. Utilizing similar principles, the work in [64] performs forceful work-tasks using MPC and a tilt-rotor MAV. Fig. 8. Physical interaction with micro aerial vehicles affords hybrid systems formulation. In free-flight the manipulator/end-effector-based induced disturbances should also be accounted, while during physical interaction the forces exerted by the environment have to be considered. Investigating a more challenging task, the authors in [3] proposed a MPC framework for a MAV performing aerial pick-and-place tasks. Examining the problem of aerial manipulation, the authors in [65] propose a NMPC to follow desired trajectories with the end-effecctor of a multirotor. The work further examines the potential enabled by the augmented kinematics the manipulator offers during freeflight. Considering the explicit task of opening a door, the contribution in [66] proposes a model predictive control framework, albeit in simulation, for a quadrotor utilizing an onboard arm to open a hinged door. Extending the potential capacity of a MAV to perform work-tasks in its environment, the work in [67] considers the problem of the robot interacting with its environment through an elastic tool.
IV. OPEN-SOURCE MPC PACKAGES FOR MAVS The success of MPC in the problem of trajectory tracking for MAVs is also reflected in the extensive utilization of relevant open-source packages released. The work in [36] is associated with an open-source Robot Operating System (ROS) package available at https://github.com/ethz-asl/mav_control_rw that offers both linear and nonlinear MPC laws. The code in https://github.com/uzh-rpg/rpg_mpc also provides MPC functionality for multirotors and has extensions to perception-aware functionality [12]. Similarly, it is released as a ROS package. The work in [68] is also released as an open-source contribution and provides both multi-robot and single-robot control such as NMPC for quadrotors. It can be found as a ROS package at https://github.com/DentOpt/denmpc. Last, an implementation for ARM CPUs [69] can be found at https://github.com/klaxalk/multirotor-control-board. Contributing a larger overall software library for control, the work in [70] also provides an example for MPC of quadrotors and can be found at https://github.com/ethz-adrl/control-toolbox. These works are indicative and more are available in the community. Simultaneously, the interested researcher can also directly refer to software packages for general MPC design such as CVXGEN [40]  V. CONCLUSIONS A survey on the application and design considerations of model predictive control for micro aerial vehicles was presented. The literature in the domain includes both linear and nonlinear controllers for the robot flight dynamics, methods for physical interaction and load transportation, faulttolerant control schemes and methods combining modern reinforcement learning techniques. As the integration of MAVs in important application domains becomes wider, we anticipate that the study of novel MPC methods -especially considering the uncertainties and risks in the robot navigation process -will tend to be even more important and possibly essential for achieving robust autonomous flight.