{'fraction_non_alphanumeric': 0.04772793146758558, 'fraction_numerical': 0.011142958358829023, 'mean_word_length': 4.339064649243467, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Anticipating the Long-Term Effect of Online Learning in Control

Control schemes that learn using measurement data collected online are increasingly promising for the control of complex and uncertain systems. However, in most approaches of this kind, learning is viewed as a side effect that passively improves control performance, e.g., by updating a model of the system dynamics. Determining how improvements in control performance due to learning can be actively exploited in the control synthesis is still an open research question. In this paper, we present AntLer, a design algorithm for learning-based control laws that anticipates learning, i.e., that takes the impact of future learning in uncertain dynamic settings explicitly into account. AntLer expresses system uncertainty using a non-parametric probabilistic model. Given a cost function that measures control performance, AntLer chooses the control parameters such that the expected cost of the closed-loop system is minimized approximately. We show that AntLer approximates an optimal solution arbitrarily accurately with probability one. Furthermore, we apply AntLer to a nonlinear system, which yields better results compared to the case where learning is not anticipated.


I. INTRODUCTION
Control design often requires an accurate model of the system dynamics. However, obtaining a mathematical model is often prohibitive due to system intricacy or lack of expertise. Moreover, erroneously assuming that a model is correct can lead to poor control performance. These issues have been increasingly addressed by employing online learning-based strategies, i.e., algorithms that employ system measurements collected online to improve control performance. This is typically achieved either by learning a model of the system, e.g., with Bayesian modeling tools [1]- [7], or by directly learning the optimal control law, e.g., by applying online reinforcement learning [8]. Despite belonging to the broader category of adaptive control, the intricacy of online learningbased control algorithms often does not allow a formal assessment of the resulting control performance, as opposed to many classical adaptive control strategies [9], [10].
Even though online learning-based approaches adapt over time using measurement data, they often include parameters that are data-independent, i.e., parameters that are fixed a priori and do not depend on the collected data. Examples include control gains [1], [6], [11] and safety-relevant parameters [4], [12]. Most of these methods choose the dataindependent parameters such that system safety and stability is guaranteed after an arbitrary model update [4], [6], [12], while others omit guarantees altogether [1]- [3], [7], [8], This [11]. Hence, although learning is an integral part of the control loop, much the same as the control law itself, it only improves the control law in a passive fashion. In other words, the control law is not designed with future learning in mind. This can cause the control to be overly conservative, leading to excessively costly state trajectories.
Efficiently choosing data-independent parameters in a learning-based setting requires accurately assessing how the control law will perform, which is generally achieved by leveraging any prior knowledge about the system. To this end, we introduce a novel algorithm for optimizing dataindependent parameters that quantifies how system uncertainty is expected to be reduced over time due to learning. In other words, the proposed algorithm anticipates the impact that online learning will have on future control performance.
Within the control community, the idea of anticipating and exploiting learning effects in control design has been explored in the form of dual control [13], [14]. So far, dual control has been investigated mostly within the context of structured models with parametric uncertainties, with few exceptions [15], [16]. However, [16] requires the true system to be affine in the control, and both [16] and [15] employ approximations that yield no theoretical guarantees. Hence, developing a general method that provably approximates data-independent parameters arbitrarily accurately remains an open research question.
In this paper, we present AntLer (anticipating learning), a sampling-based algorithm that approximates optimal dataindependent parameters of online learning-based control laws in uncertain settings. Our approach accounts for a broad class of model uncertainties by using a probabilistic Gaussian process model. Given a cost function that quantifies control performance over a finite-time horizon, AntLer is able to express the expected cost for an online learning-based control law. Minimizing the resulting expression with respect to the control law's data-independent parameters corresponds to a stochastic optimal control problem, which AntLer solves approximately using sample average approximation. AntLer is applicable to a wide class of dynamical systems that include an additive uncertainty, as well as process noise. We show that, under reasonable assumptions, AntLer approximates the optimal solution arbitrarily accurately given a large enough number of samples.
The remainder of this paper is organized as follows. Section II describes the general problem setting and the assumptions used in this paper. In Section III the probabilistic approach used to quantify model uncertainty is discussed. Section IV contains our main result. Therein, we introduce the AntLer algorithm and provide a corresponding theoretical analysis. In Section V AntLer is applied to a numerical system. We then provide some concluding remarks in Section VI.
Notation: Let N denote the natural numbers, R the real numbers, and R + the non-negative real numbers. We employ bold lowercase and uppercase letters to denote vectors and matrices, respectively. For µ, σ ∈ R + , a normal distribution with mean µ and variance σ 2 is denoted as N (µ, σ 2 ). For d ∈ N, we denote the space of continuously differentiable functions on R d as C 1 (R d ), and the d-dimensional identity matrix as I d . Moreover, for matrices A, B ∈ R d×d , we use [A, B] to denote the horizontal concatenation of A followed by B. The entry in the i-th row and j-th column of A is denoted by [A] ij . The symbol ∪ denotes the union of two sets. We use E a1,...,a d [ · ] to denote the expected value operator with respect to the probability distribution of the random variables a 1 , . . . , a d ∈ R d .

II. PROBLEM STATEMENT
We consider a discrete-time system of the form where x t ∈ X ⊆ R Nx and u t ∈ U ⊆ R Nu are the system's state vector and control vector at the t-th time step, respectively. The initial state x 0 ∈ X is assumed to be fixed and known. The vector of augmented statesx t := (x t , u t ) ∈X , whereX := X × U , concatenates the state vector x t and the vector of control inputs u t , and is henceforth employed for the sake of simplicity. The system is disturbed by multivariate normally distributed process noise w t ∼ N (0, Σ 2 w ). Here Σ w = diag(σ w1 , . . . , σ wN x ) is a nonnegative diagonal matrix, which we assume to know. The function f ∈ C 1 (X ), corresponds to the prior model of the system dynamics, whereas g ∈ C 1 (X ) is unknown and is assumed to be drawn from a Gaussian process (GP). This is described thoroughly in Section III.
Remark 1: In this paper, we assume that x 0 is fixed and known solely to avoid cumbersome notation. The algorithm proposed in this work extends straightforwardly to the more general case where only the probability distribution of x 0 is known.
Remark 2: This constellation can be assumed for a wide variety of settings. For example, if no prior system knowledge is available, then this is reflected by choos- We assume that a parametric online learning-based control law of the form u : Γ × Θ × X → U is employed to control (1), where Θ denotes the space of data-independent control parameters, and Γ := {{x 0 , . . . ,x t } ∈X t | t ∈ N} is the set of all finite subsets ofX . At every time step, the control law u(·, ·, ·) takes as arguments the system measurement data D t = {x 0 , . . . ,x t−1 } ∈ Γ collected up to time step t, the data-independent control parameters ϑ ∈ Θ, and the current state x t . The collected data D t is employed to update the control law at every time step, e.g., by learning a model of the system. The control parameters ϑ correspond to the data-independent components of the control law, e.g., multiplicative scalars used to scale confidence regions and thereby guarantee operational safety [17], or linear feedback gains [6]. This formulation encompasses most discretetime online learning-based control strategies. We henceforth write u t (ϑ) := u(D t , ϑ, x t ) to denote the online learningbased control law at time step t.
Remark 3: In order to anticipate the effect of online learning, we aim to predict which data set D t will be collected over time and how it will affect the overall control performance. As a baseline, we consider the case where predictions are carried out without anticipating learning, which amounts to predicting the closed-loop behavior under the data-independent counterpart u 0 t (ϑ) := u(D 0 , ϑ, x t ). Here we assume D 0 := {} without loss of generality. In Section V, we compare predictions made with both control laws using a simple example.
Remark 4: The method presented in this paper extends straightforwardly to a setting where the system measurements D t used to update the control law are corrupted by normally distributed observation noise. However, for notational convenience, we focus solely on the case without observation noise.
Our goal is to minimize a finite horizon cost function over the data-independent control parameters ϑ, where c t :X → R + are continuously differentiable functions that express the immediate cost. The probability distribution ofx 1 , . . . ,x T captures both the effect of process noise w t , as well as the model uncertainty g(·). This is discussed in Section III. We denote the minimal value of (2) as C * := min ϑ∈Θ C(ϑ) and the corresponding set of minimizing parameters as Θ * := {ϑ ∈ Θ | C(ϑ) = C * }.
If no assumptions about the online learning-based control law u t (·) are made, then it is generally impossible to reliably predict the closed-loop behavior of (1). Hence, we need to impose some restrictions on the type of control law considered.
Assumption 1 is less restrictive than assuming that Θ is compact, which is often the case in learning-based applications, e.g., in settings where safety-relevant constraints are an issue [18], [19]. Furthermore, Assumption 1 does not impose strong limitations in practice, asΘ may still be very large.
In order to be able to find a minimizer ϑ * ∈ Θ * of (2), we additionally require the control law u(·, ·, ·) to satisfy some regularity conditions. In this paper, we restrict the control law to the broad and practically relevant class of continuously differentiable functions, as described in the following.
Assumption 2: The control law u(D t , ϑ, x t ) is continuously differentiable with respect to its arguments, where continuous differentiability with respect to the data is defined as follows. For every fixed D ∈ Γ, x t ∈ X and ϑ ∈ Θ, the function is continuously differentiable with respect tox for allx ∈X .
Many commonplace control laws are continuously differentiable with respect to the state x t and parameters ϑ ∈ Θ, e.g., linear feedback gains and neural networks. Furthermore, control update rules are often continuously differentiable with respect to the data, e.g., if a model of the system is learned online [5].

III. PROBABILISTIC SYSTEM MODEL
In this section, we provide a brief introduction to GPs, and describe how we use them to capture model uncertainty and predict control performance.

A. Predictions using Gaussian processes
In order to assess how the learning-based control law will perform in an uncertain environment, we require a probabilistic model that expresses model uncertainty given prior system measurements. To this end, we model (1) using a Gaussian process (GP), a probabilistic modeling tool that captures model uncertainty. We opt to employ GPs in this work because they often exhibit good generalization behavior in practice. However, we note that other probabilistic modeling frameworks can be employed, e.g., Bayesian neural networks.
We introduce GPs for the case where the state is a scalar, i.e., N x = 1, and then explain how one-dimensional GPs are extended to the multivariate case. A GP is a collection of dependent random variables, for which any finite subset is jointly normally distributed [20]. It is fully specified by a mean function m :X → R and a positive definite kernel function k :X ×X → R. In this paper, since our prior knowledge is captured by f (·), we set m ≡ 0 without loss of generality [20]. The kernel k(·, ·) is a similarity measure for evaluations of g(·), and encodes function properties such as smoothness and periodicity. Throughout this paper, we assume that the kernel k(·, ·) is continuously differentiable, which reflects the assumption that g(·) is continuously differentiable [20]. Given m(·) and k(·, ·), we denote a GP by GP(m, k). By modeling an unknown function g(·) with a GP, we implicitly assume that any finite set of function evaluations y Dt := (g(x 0 ), . . . , g(x t−1 )) at arbitrary points D t := {x 0 , . . . ,x t−1 } is jointly normally distributed, where the entries of the covariance matrix K Dt are given by Using (3), we are able to condition the GP on any measurements taken prior to the control design. In the following, for the sake of notational simplicity, we assume that no prior measurement data is available, and describe how to recursively draw and condition the GP on samples. However, conditioning the GP on system measurement data is identical to conditioning on samples up to an additive term that represents noise covariance [20].
In order to predict the control performance of u t (·), we aim to draw sample trajectories that satisfy (3). We henceforth distinguish sample evaluations of the GP model, which are drawn using (3), from evaluations of the true system (1) by denoting samples using the superscript s. A sample system trajectory is computed by sequentially sampling from the one-step prediction of the unknown dynamics at time step t and subsequently computing the next sample statẽ where is introduced for simplicity of exposition. The mean and variance of (4) are computed using µ s (7) respectively. Here the vector concatenates previously drawn sample statesx s i ∈ D s t , and consists of kernel evaluations atx s t andx s i ∈ D s t . Remark 5: Here we abuse notation slightly by employing g s (·) to refer to a function sampled from the GP. As can be seen from (6)-(9), g s (·) depends on previously sampled function evaluations. In fact, a sample function evaluation is computed as where ζ s ∈ N (0, 1). In Section IV, we use rigorous notation by referring to sample function evaluations as in (10). Remark 6: It is necessary that the GP samples g s (x s t ) and process noise samples σ s w be drawn separately in order for the vector y D s t to be uniquely defined. This in turn guarantees that the sampled function g s (·) exhibits deterministic behavior at points where samples were previously drawn [20]. We require this to reflect the fact that g(·) is unknown but deterministic. Hence, we draw sample trajectories that satisfy (4) and (5) as where ζ s 1 , ζ s 2 ∼ N (0, 1) are sampled separately. Remark 7: Typically, multiple samples can be drawn from the same GP simultaneously [20]. However, since we are interested in samples that satisfy the system dynamics, we need to draw a sample and compute the resulting state sequentially.
In the case where the state is multidimensional, we model each state transition using a separate GP, i.e., concatenates samples of the i-th component of the GP model for every i = 1, . . . , N x .
Remark 8: Modeling each state transition with a separate GP corresponds to assuming that the state transitions are conditionally independent. Alternatively, a generalization of GPs to multiple dimensions is also applicable [20]. However, the latter approach is significantly more computationally expensive than the former. Moreover, employing separate GPs for each state transition function has been shown to yield good results in practice [21].
Remark 9: For the sake of brevity, we only show here how to model a multidimensional g(·) using a single kernel k(·, ·) for all entries of g(·). However, the methods described herein extend straightforwardly to the case where different kernels are employed for each entry of g(·).
We assume that the model uncertainty due to the unknown function g(·) is faithfully captured by a GP with kernel k(·, ·). Formally, this is stated as follows.
Assumption 3: Let GP(m, k) be a GP with mean m ≡ 0 and known continuously differentiable kernel k(·, ·). Then the entries of the unknown function g(·) are samples of GP(m, k), i.e., g i ∼ GP(m, k) holds for i = 1, . . . , N x .
Choosing an appropriate kernel k(·, ·) requires a priori knowledge of the system. However, the assumptions required for choosing a kernel are generally far less restrictive than for parametric models, since they only pertain to features such as smoothness and periodicity. Furthermore, in some cases, error bounds can be obtained if the kernel is poorly chosen [22].

B. Predicting control performance
Assumption 3 implies that, for a fixed set of parameters ϑ, the expected state of the true system (1) at an arbitrary time step t is given by where the integrand is computed recursively using and p(ζ s i ) = N (0, I 2Nx ). The corresponding cost function is given by Lemma 1: Let Assumptions 2 and 3 hold. Furthermore, let Proof: Since k(·, ·), c t (·, ·), u s t (·) are continuously differentiable with respect to their arguments, T t=0 c t (x s t , u s t (ϑ)) is a composition of continuously differentiable functions. Hence it is continuously differentiable with respect to the control parameters ϑ. Due to Leibniz's rule, this implies that (15) is also continuously differentiable with respect to ϑ. (15) is generally intractable. Hence, we compute an estimate of the minimizer of (15) by employing a sample average approximation (SAA) of (15),  The steps required to compute a minimizer of (16) yield the AntLer algorithm, which is presented in Algorithm 1.

IV. SAMPLE AVERAGE APPROXIMATION Computing the integral
Remark 10: Despite being mainly designed with online learning-based control laws in mind, AntLer can also be Algorithm 1 Anticipating learning (AntLer) Output: ϑ * M employed in the special case where the control law does not change based on the data collected online. In such settings, AntLer becomes similar in principle to model-based reinforcement learning approaches, e.g., [21].
Remark 11: The algorithm proposed in this paper can also be applied to the infinite-horizon case, e.g., by implementing it in a receding horizon fashion. This would generally require a terminal constraint to be considered, for which probabilistic guarantees can be derived, e.g., as in [23].
We now aim to prove that a solution ϑ * M obtained with AntLer approximates an optimum ϑ * ∈ Θ * of the exact problem arbitrarily accurately for a sufficiently high number of samples M . To achieve this, we show that both the approximate and exact cost functions C M (·, ·), C(·), satisfy some regularity conditions. Lemma 2: Let Assumptions 1-3 hold, and chooseΘ as in Assumption 1. Then C(·) is finite-valued and continuously differentiable onΘ, and C M (·, Z M ) converges to C(·) with probability 1 uniformly inΘ as M → ∞.
To prove Lemma 2, we make use of the following result, which corresponds to [24,Theorem 7. Using Lemma 2, we are able prove that Algorithm 1 approximates an optimal solution ϑ * ∈ Θ * arbitrarily accurately with probability 1 for large enough M . This corresponds to our main result, and is stated in the following theorem.
Proof of Theorem 1: We show that the conditions of Lemma 4 hold for the compact subsetΘ from Assumption 1.
Conditions ii) and iii) are satisfied due to Lemma 2. Hence, it remains to be shown that i) and iv) hold.
We begin by showing that the set Θ * is nonempty. To this end, consider an arbitrary sequence of control parameters ϑ i , i = 1, . . . , ∞, with lim i→∞ C(ϑ i ) = C * . Due to Assumption 1 and the continuity of C(·) (i.e., Lemma 1), there exists an I ∈ N, such that ϑ i ∈Θ holds for all i ≥ I. Since ϑ 1 , . . . , ϑ I are finite-valued, this implies that the sequence ϑ i , i = 1, . . . , ∞ belongs to a compact set. Due to the Bolzano-Weierstrass theorem, ϑ i contains a convergent subsequence with limit ϑ * ∈Θ. Hence, Θ * is nonempty and Θ * ⊆Θ, i.e., Condition i) of Lemma 4 is satisfied.
Using the same argument we can show that Θ * M is nonempty. Moreover, Assumption 1 implies that Θ * M ⊆Θ holds for all Z M , i.e., Condition iv) is satisfied.
Hence, the AntLer algorithm approximates an optimal vector of data-independent parameters ϑ * with arbitrary accuracy for large enough M with probability 1.
For a control law u t (ϑ) that potentially improves its performance through online learning, Theorem 1 implies that AntLer guarantees superior control performance for large enough M compared to the case where learning is not anticipated. This is shown by comparing predictions for u t (·) to predictions for its data independent counterpart u 0 t (·) = (D 0 , ·, ϑ). We state this formally in the following.
This amounts to assuming that u t (ϑ) potentially improves its performance as new data is gathered.
Corollary 1: Let Assumptions 1-4 hold, and let C(·) be given as in (2). Furthermore, let C 0, * be the optimal cost under the data-independent counterpart, as given in Assumption 4, and let ζ  Proof: This follows directly from Theorem 1.

V. NUMERICAL EXAMPLE
We now illustrate the proposed algorithm using a simple nonlinear trajectory tracking problem. We demonstrate the convergence of the approximate optimal parameters computed by AntLer as the number of samples M grows, and compare the computed parameters and predictions to those obtained without anticipating learning. Furthermore, by preforming Monte Carlo simulations of the true system, we showcase the superior performance of the approximate optimal parameters compared to the case where learning is not anticipated.
The source code of the experiments presented in this section is available at https://git.lsr.ei.tum.de/acapone/antler.

A. System description
We consider the one-dimensional system with initial state x 0 = 0, process noise w t ∼ N (0, 0.01 2 ), and state transition functions We aim to design an online learning-based control law that tracks the trajectory x ref t = 4 sin(t/2π) as accurately as possible, while simultaneously accounting for any potential tracking errors due to the unknown function g(·). To this end we choose the control law where ϑ 1 acts as a control gain, and ϑ 2 scales the reference trajectory and enables to avoid regions of high model uncertainty. The term µ t (x t ) is a GP mean, which is updated online as new data points are collected. We compute µ t (x t ) using the same kernel as for AntLer, which we specify in the sequel. Employing the same kernel both for predictions and control is reasonable, since we assume that it faithfully represents the unknown function g(·). We quantify control performance by employing the cost function where the immediate cost terms c t := (x t − x ref t ) 2 penalize deviations from the reference trajectory.
We now describe the kernel used for AntLer predictions and the online learning-based control law (21). We assume to know that g(·) depends only on the state x t , and that it corresponds to a smooth function. This information is encoded into the GP by employing a squared exponential kernel that takes only the state as argument, i.e., where the signal variance σ 2 k ∈ R + and length scale l ∈ R + are obtained by training the GP using log marginal likelihood optimization [20]. To this end, we assume to have 100 measurements of (18), which were obtained using a control law that attempts to minimize the distance of the true system (18) to the origin. Squared exponential kernels are dense within the space of continuous functions on compact sets, i.e., they can approximate any continuous function uniformly and arbitrarily well on compact subsets of X [25]. Moreover, the posterior mean µ t (·) of a GP obtained with a squared exponential kernel exhibits smooth behavior [20]. Hence, (23) is an appropriate choice for this setting.
It can easily be shown that, in a setting where g(x t ) is known, i.e., µ t (x t ) = g(x t ), the system trajectory is optimal for ϑ 1 = ϑ 2 = 1. Since the control law (21) learns g(·) online, it is reasonable to expect that the optimal parameters ϑ * for unknown g(·) lie within a neighborhood of ϑ 1 = ϑ 2 = 1, provided that g(·) is learned correctly. Hence, we assume that the optimal parameters lie within the compact subsetΘ = [−1, −1] × [2,2]. In the following, we employ this assumption to restrict the feasible region of the optimization problem toΘ.

B. Approximate optimal solutions using AntLer
We demonstrate the convergence of the approximate optimal solution ϑ * M to ϑ * as M grows by applying AntLer using M ∈ {2, 10, 50, 100, 200} samples. Additionally, in order to illustrate Corollary 1, we make predictions and optimize the parameters ϑ without anticipating learning, i.e., by using the data-independent counterpart of the control law u 0 To optimize the parameters of u 0 t (ϑ), we employ AntLer with M = 200. We are able to do so, since u 0 (·) is a special case of an online learning-based control law. Hence, approximate optimal parameters can also be obtained using AntLer. For simplicity of exposition, we henceforth refer to ϑ * ,0 as the optimum of the data-independent counterpart.
To solve the SAA problem in AntLer, we employ a gradient-based method with different starting values, which are sampled from the uniform distribution onΘ. A solution is found after at most 17 gradient-descent steps. In Table I, we display the approximate optimal parameters ϑ * M computed by Antler. In Fig. 1a, we present AntLer predictions   for M = 200 and the approximate optimal online learningbased law u t (ϑ * M ). Furthermore, in Fig. 1b we show predictions for the optimal data-independent counterpart u 0 t (ϑ 0, * ). The value of the approximate optimal parameters is ϑ * M ≈ (1, 0.9) T for all 50 < M < 200. This indicates that ϑ * M has converged to a small neighborhood of the optimal parameters ϑ * , as expected from Theorem 1.
AntLer predicts that, by scaling the reference trajectory with ϑ 2 = 0.9, an optimal trade-off is achieved between the information of the collected data and the error caused by model uncertainty. In other words, if the control law were to attempt to fully enforce the reference trajectory, i.e., ϑ 2 = 1, then AntLer predicts that too many measurements need to be collected before good tracking performance is achieved. However, if ϑ 2 = 0.9 is chosen, then AntLer predicts that the unknown dynamics will be learned quickly enough to achieve good tracking performance within the time horizon T = 150. This becomes apparent in the predictions in Fig. 1a. Therein, the variance of the state x t and the expected immediate cost c t under the approximate optimal control law u t (ϑ * M ) decrease over time. After t ≈ 70, they become approximately zero.
The parameters ϑ 0, * = (0.9, 0.2) T of the optimal dataindependent counterpart u t (ϑ 0, * ) attempt to keep the system close to the origin. This is because predictions for u 0 t (ϑ) do not anticipate learning. In other words, they only yield low tracking errors in regions where model uncertainty is already low. As measurement data at the origin was collected prior to the control design, model uncertainty is high in the whole state space except for a neighborhood of the origin. Hence the approximate optimal parameters ϑ 0, * = (0.9, 0.2) T attempt to keep the system within this region. This is reflected in the predictions in Fig. 1b, where the tracking error exhibits little variance compared to Fig. 1a.
For M = 200, the predicted cost C(ϑ * M ) = 212 under the approximate optimal control law u t (ϑ * M ) is lower than the predicted cost C(ϑ 0, * ) = 731 under the data-independent counterpart u t (ϑ 0, * ). Assuming that the GP specified by the kernel (23) correctly captures the model uncertainty due to g(·), Corollary 1 implies that control performance will be superior if ϑ * M is applied to the true system instead of ϑ 0, * . This indeed is the case, as shown in the following.

C. Monte Carlo simulations of true system
The parameters ϑ * M = (1, 0.9) T computed by AntLer for M = 200 are employed to control the true system (18) in 100 Monte Carlo runs. Moreover, we compare the results to the Monte Carlo simulation using the optimal parameters obtained without anticipating learning ϑ 0, * = (0.9, 0.2) T . The respective results are shown in Fig. 2a and Fig. 2b.
As shown in Fig. 2a, the variance of the state is high for ϑ = ϑ * M at the beginning of the Monte Carlo simulation. This is due to the initially unknown system dynamics g(·). After approximately t = 80, enough measurement data has been gathered to adequately track the reference trajectory. Despite differences in overall variance and learning time, the results agree qualitatively with the AntLer prediction shown in Fig. 1a, which indicates that the kernel (23) was chosen adequately.
For ϑ = ϑ 0, * , the variance of the state is very low throughout the simulation. This is because the parameters ϑ = ϑ 0, * steer the system to a region of low model uncertainty. These results are in agreement with the predictions presented in Fig. 1b.
The average cumulative cost for ϑ = ϑ 0, * is 859. This is higher than for ϑ = ϑ * M , which achieves an average cost of 37. This was expected from the AntLer predictions and Corollary 1.

VI. CONCLUSION
We have presented AntLer, a control design approach that anticipates the effect of online learning and optimizes dataindependent parameters accordingly. By expressing model  uncertainty with a Gaussian process model, we have formulated the parameter optimization problem as a stochastic optimal control problem, which AntLer solves approximately using sample average approximation. We have shown that AntLer approximates an optimal solution arbitrarily accurately with probability one for a sufficiently large number of samples. We have applied AntLer to a nonlinear system. The results have shown that model learning is correctly anticipated, which leads to a better choice of control parameters compared to the case where learning is not anticipated.
In future work, we aim to apply AntLer to complex online learning-based control laws, such as learning-based model predictive control and online reinforcement learning.