{'fraction_non_alphanumeric': 0.04731333292764559, 'fraction_numerical': 0.014969877685145743, 'mean_word_length': 4.022463325183374, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Meta Learning in the Continuous Time Limit

In this paper, we establish the ordinary differential equation (ODE) that underlies the training dynamics of Model-Agnostic Meta-Learning (MAML). Our continuous-time limit view of the process eliminates the influence of the manually chosen step size of gradient descent and includes the existing gradient descent training algorithm as a special case that results from a specific discretization. We show that the MAML ODE enjoys a linear convergence rate to an approximate stationary point of the MAML loss function for strongly convex task losses, even when the corresponding MAML loss is non-convex. Moreover, through the analysis of the MAML ODE, we propose a new BI-MAML training algorithm that significantly reduces the computational burden associated with existing MAML training methods. To complement our theoretical findings, we perform empirical experiments to showcase the superiority of our proposed methods with respect to the existing work.


Introduction
In machine learning, an ideal learner is able to speed up the learning of new tasks based upon previous experiences. This goal has been shared among different but highly related approaches such as few-shot learning [34], domain adaptation [43,17], transfer learning [24], and meta learning (a.k.a. learning to learn) [37,23,30,13]. In particular, meta-learning addresses a general optimization framework that aims to learn a model based on data from previous tasks, so that the learned model, after fine-tuning, can adapt to and perform well on new tasks. This idea has been successfully applied to different learning scenarios including reinforcement learning [39,4,32], deep probabilistic models [6,16,10], language models [25], imitation learning [5], and unsupervised learning [14], to name a few. Previous works have also investigated meta learning from a variety of perspectives and methods, including memory-based neural networks [29,22], black-box optimization [4,40], learning to design an optimization algorithm [1], attention-based models [38], and LSTM-based learners [28].
One of the gradient-based meta-learning algorithms that has been widely used and enjoys great empirical successes is the model-agnostic meta-learning proposed in [8]. Rather than minimizing directly on a combination of task losses, maml meta-trains by minimizing the loss evaluated at one or multiple gradient descent steps further ahead for each task. The intuition is as follows: First, this meta-trained initialization resides close to the best parameters of all training tasks. Second, for each new task, the model can then be easily fine-tuned for that specific new task with only a small number of gradient descent steps. Following this work, many experimental and theoretical studies of maml have been carried out [35,3,10,42,27]. In particular, Fallah et al. [7] and Mendonca et al. [20] investigated the convergence of maml to first order stationary points. Moreover, Finn et al. [9] studied an online variant of maml and Antoniou et al. [2] proposed various modifications to maml to improve its performance. Empirically, Raghu et al. [26] found that the success of maml can be primarily associated to feature reuse, given high quality representations provided by the meta-initialization.
maml assumes a shared parameter space R d among all the tasks and learns an initialization w ∈ R d from a batch of tasks at meta-learning time, such that performing a few steps of gradient descent from this initialization at meta-testing time minimizes the loss function for a new task on a smaller dataset. More specifically, in a task pool with total M candidate tasks, each task T i is sampled from the pool according to a distribution p(T ) and has a corresponding risk function f i : R d R that is parameterized by a shared variable w ∈ R d . This function measures the performance of the parameter w on task T i . In practice we estimate this risk function from a set of training data for each task. However, for simplicity, we directly work with the risk function in this paper and assume that we can acquire the knowledge of the task loss f i from an oracle. At meta-learning time, maml looks for a warm start w via solving an optimization problem that minimizes the expected loss over f i after one step of gradient descent, i.e., (1.1) where E i∼p denotes the expectation over sampled tasks and α represents the maml step size (also referred to as the maml parameter). To ease the burden on notation, we define the maml loss function of task T i to be F i (w) := f i (w − α∇f i (w)). Further, we define some shorthand for the expected loss f (w) := E i∼p f i (w) and the expected maml loss F (w) := E i∼p F i (w). Now we are able to represent the optimization problem (1.1) in a more concise form: arg min w F (w).
Solving (1.1) yields a solution w that in expectation serves as a good initialization point, such that after a step of gradient descent it achieves the best average performance over all possible tasks. However, because the objective function F (w) is non-convex in general, there is no guarantee that one is able to find a global minimum. Therefore it is common to instead consider finding an approximate stationary point w where ∇F ( w) ≤ ε for some small ε [7].
In this paper, we study the maml algorithm proposed in [8], in which maml meta-learns the model by performing gradient descent on the maml loss. We begin by establishing a smooth approximation of this discrete-time iterative procedure and considering the continuous-time limit by taking an infinitesimally small step size. With any initialization of the parameter, the problem is transformed into an initial value problem (ivp) of an ode, which is the underlying dynamic that governs the training of a maml model. Specifically, We consider the convergence of the gradient norm ∇F (w) and the function value F (w) under the continuous-time limit of maml.
Our contributions can be summarized as follows: • We propose an ode that underlies the training dynamics of a maml model. In this manner, we eliminate the influence of the manually chosen step size. The algorithm in [8] can be viewed as an forward Euler integration of this ode.
• We prove that the aforementioned ode enjoys a linear convergence rate to an approximate stationary point of F for strongly convex task losses. In particular, it achieves an improved convergence rate of O(log 1 ε ), compared to O( 1 ε 2 ) in [7]. Note that the maml loss F may not be convex even if the individual task losses f i are strongly convex.
• We show that for strongly convex task losses with moderate regularization conditions, the maml loss F has a unique critical point that is also the global minimum.
• We propose a new algorithm named bi-maml for strongly convex task losses that is computationally efficient and enjoys a similar linear convergence rate to the global minimum compared to the original maml. It also converges significantly faster than maml on a variety of learning tasks.

Preliminaries
Before turning to the discussion of the continuous-time limit of maml, we briefly introduce a widelyused approach for taking the continuous-time limit of discrete-time algorithms and the approach we use later for its analysis.

Optimization through lens of ODE.
There is an extensive literature on the topic of understanding discrete-time algorithms through the lens of odes [31,11,18], and recent developments in this field offer novel perspectives for looking at discrete-time optimization algorithms [36,21]. For example, Shi et al. [33] developed a first-order optimization algorithm by performing discretization on odes that correspond to Nesterov's accelerated gradient descent. Krichene et al. [15] proposed a family of continuous-time dynamics for convex functions where the corresponding solution converges to the optimum value at an optimal rate. However, there can be multiple odes that correspond to the same discrete-time algorithm, and it oftentimes requires strong mathematical intuitions when it comes to taking the continuous limit. In this paper, we take the most intuitive approach by letting the step size of the gradient descent on the maml loss F go to zero, and the resulting ode is a gradient flow on F .
Lyapunov's direct method. One of the most commonly used approaches for analyzing the convergence of odes is Lyapunov's direct method [19,12,41], which is based on constructing a positive definite Lyapunov function E : R d R that decreases along the trajectories of the dynamicṡ w: This method is a generalization of the idea that measures the "energy" in a system, and the existence of such a continuously differentiable Lyapunov function guarantees the convergence of the dynamical system.

Main Results
In this paper, we analyze the maml algorithm on strongly convex functions with its continuoustime limit and establish a linear convergence rate. In addition, we propose a new algorithm named Biphasic MAML (bi-maml) as an alternative to the original maml. Unlike maml where we always minimize the function value on F , the optimization process of bi-maml can be divided into two phases. In the first phase, bi-maml optimizes the expected task loss f until it reaches its approximate global minimum. In the second phase, it runs maml until it finds an approximate critical point. We show that bi-maml also enjoys the same O(log 1 ε ) iteration complexity on strongly convex functions. While the iteration complexity of maml and bi-maml share the same order, bi-maml has a lower computational complexity, because it performs gradient descent on f rather than F in the first phase and thereby avoids computing the Hessian. In contrast, maml performs gradient descent on F , which involves computing the Hessian of f . We will elaborate further later in this section. Moreover, we will present empirical results in Section 4 to show that bi-maml significantly outperforms vanilla maml.
Both our new analyses on maml and the new bi-maml algorithm are based on our analysis of the landscape of F . Fig. 1 illustrates an example where F is indeed non-convex. This example has two tasks where we take f 1 (w) = 0.505w 2 − sin(w) and f 2 (w) = 0.505w 2 − 0.0001 sin(100w). Both functions are 0.01-strongly convex and 2.01-smooth. We observe that F (w) is not monotone increasing and that F (w) is not always positive. These imply that F is non-convex. While F is non-convex in general as illustrated, we prove that F has a unique critical point and is strongly convex on a convex set around the critical point, which implies that it is the global minimum. Of course, the function F can be non-convex outside the convex set.   Here we let f 1 (w) = 0.505w 2 − sin(w) and f 2 (w) = 0.505w 2 − 0.0001 sin(100w). Notice that both f 1 (w) ≥ 0.01 and f 2 (w) ≥ 0.01 for all w ∈ R d . Both functions are 0.01-strongly convex and 2.01-smooth. It is not hard to see also that Assumptions 3.4 and 3.5 are both satisfied for some finite σ and κ. Taking the maml step size as α = 0.4, we have a non-convex maml loss F with its first-and second-order derivatives as indicated in Figs. 1a and 1b.

Original and Biphasic MAML (BI-MAML) Algorithms
In this section, we present the original maml and bi-maml algorithms. We begin with the original maml. In particular, we investigate maml under a continuous-time limit. Recall that the update rule of maml on w follows a gradient descent on F (w), i.e., where w denotes the iterate input, w + denotes the iterate output, β represents the step size, and A i (w) := I d −α∇ 2 f i (w) is a shorthand for the Hessian correction term. Here we see that computing the gradient of F requires the Hessian of f . As mentioned above, bi-maml runs faster because it avoids computing the Hessian of f by performing gradient descent on f in the first phase. In this paper we consider maml where the step size α of the task-specific gradient descent remains constant while the step size β for the maml gradient descent goes to zero. Proposition 3.1 presents the continuous-time limit for maml, which we term as the maml ode.

Proposition 3.1 (Proof in Appendix A).
If the losses f i are twice differentiable, the continuoustime limit for maml isẇ The first term on the right-hand side of (3.2) represents a gradient flow on f , and the second term that follows is the key term that differentiates a maml gradient descent on the maml loss function F from a vanilla gradient descent on the expected loss function f . Due to the compositional nature of the maml loss F (w), the second-order information is required to evaluate its gradient.
Recall that by definition, we have To reduce such cost on computing the Hessian, Finn et al. [8] proposed the first-order model-agnostic metalearning (fo-maml), which is a first-order approximation of maml that replaces the second-order term I d − α∇ 2 f i (w) with an identity matrix. However, this may cause a failure in convergence, as mentioned in [7]. In comparison, our new bi-maml algorithm achieves a similar computational efficiency by evaluating only first-order information while at the same time still being able to converge to an approximate stationary point.
Inspired by our convergence analysis on maml ode, we propose a different dynamic with gradient flow in two stages called bi-maml ode. The first stage of bi-maml ode is a gradient flow on the expected loss function f that converges to one of its stationary point, i.e.,ẇ = −∇f (w). This is followed by the second stage, which is a maml ode starting from an approximate stationary point provided by the first stage. maml is a forward Euler integration of its continuous-time limit (3.2): a gradient flow on the maml loss F . Analogously, the bi-maml algorithm is a forward Euler integration of bi-maml ode: a gradient descent on f followed by a gradient descent of F . More detailed descriptions of bi-maml and bi-maml ode are given in Algorithm 1 and Algorithm 2.

Summary of Assumptions, Results, and Techniques
In this section, we establish our theoretical results for maml and bi-maml when the loss functions f i (w) are strongly convex and smooth with bounded gradient variance among the tasks. Formally, we make the following assumptions.
To simplify the notation, we denote L : We note that Finn et al. [9] assumed all the assumptions above, except Assumption 3.4 because they considered online meta-learning where functions can be selected in an adversarial manner. On the other hand, they assumed that the functions are Lipschitz [9, Assumption 1.1], which may contradict the strong convexity assumption [9, Assumption 2] in their paper. Similarly, Fallah et al. [7] assumed all but Assumption 3.3. We remark that Assumption 3.3 implies the boundedness of the maml loss F from below, but it does not guarantee the convexity of F . See Fig. 1 for an example of non-convex maml loss F with corresponding task losses f i satisfying Assumptions 3.2 to 3.5. In other words, while f i are strongly convex, F can still be non-convex. Hence, minimizing F is challenging as we are dealing with a non-convex optimization problem.
As we mention above, we make an additional assumption (Assumption 3.3), compared to the set of assumptions in [7]. They showed that the maml algorithm outputs a solution that guarantees Under this additional assumption, we significantly improve the result in two aspects. First, we show that our proposed algorithm finds a solution such that ∇F (w) ≤ ε in only O(log 1 ε ) iterations (Theorem 3.6). This is indeed an exponential improvement on [7] in terms of iteration complexity. Second, we characterize the landscape of F . While F is nonconvex in general, we prove that its stationary point is also the global minimum (Theorem 3.10). Therefore, the solution returned by our algorithm is close to not only a critical point but also the global minimum.
Our main results in Theorem 3.6 and Theorem 3.7 show that the maml ode and bi-maml ode achieve linear convergence in finding a critical point on the maml loss F .
then the maml ode finds a solution w such that ∇F ( w(t)) ≤ ε after at most running for then the bi-maml ode finds a solution w such that ∇F ( w(t)) ≤ ε after at most running for Theorem 3.6 says that if the maml parameter α is small enough, then the maml ode finds an approximate stationary point of the maml loss F in O(log 1 ε ) time. This approximate stationary point is at the same time an approximate global minimum, as implied by Theorem 3.10 later. Similarly, Theorem 3.7 states that whenever the maml parameter α is small enough so that F is strongly convex for every point w such that ∇f (w) ≤ ε 0 , the bi-maml ode finds an approximate global minimum of F in O(log 1 ε ) time. We prove Theorem 3.6 and Theorem 3.7 with a two-phase analysis where the transition between two phases depends on the norm of ∇F (w). In the first phase we conduct a Lyapunov function analysis on the Lyapunov candidate function ∇f (w) 2 and under the dynamics defined by the odes. The second phase follows as a landscape characterization of the maml loss F . Intuitively, ∇F (w) can be large at initialization and will become smaller over the course of the gradient flow. We note that ∇F (w) is close to ∇f (w) if α is small due to the fact that ∇F (w) = (I d − α∇ 2 f i (w))∇f i (w − α∇f i (w)), and we therefore choose to analyze the gradient norm on f instead of F . When ∇f (w) is large and we are far from the stationary point, the analysis on the Lyapunov function E(w(t)) = ∇f (w(t)) 2 helps establish a linear convergence rate for ∇f (w) to be in the order of O(σ). This is due to the proof technique shown in Lemma 3.8.
When the gradient norm ∇f (w) is small enough, ∇F (w) is also controlled. Then we enter the second phase, in which we follow a gradient flow inside a convex set where the maml loss is strongly convex. This establishes another linear convergence rate from O(σ) down to ε. Combining the above two phases gives us the overall linear rate. In the following subsections, we will explain these two phases in more detail with an emphasis on the proof of Theorem 3.6. The proof for Theorem 3.7 is similar, cf. Appendix B.

Large Gradient Phase: Linear Convergence via Lyapunov Analysis
When ∇F (w) is large, its behavior under the maml ode (3.2), namely d dt ∇F (w(t)) , is not easily tractable due to the non-convex nature of the maml loss F . Hence, we consider instead a Lyapunov candidate function E(w(t)) = ∇f (w(t)) 2 in the first phase, where (3.4) Even though we are primarily interested in the convergence of ∇F (w(t)) , the convergence analysis of ∇f (w(t)) is still helpful. We show that when α is small, an upper bound on ∇f (w) gives an upper bound on ∇F (w) , and vice versa. Hence we are able to keep track of an upper bound on ∇F (w(t)) while only having one on ∇f (w(t)) . However, we need to remark that the convergence of ∇f (w) to zero does not imply the convergence of ∇F (w) . In fact, the global minimum of f may not even be a stationary point of F .
Note that the Lyapunov candidate function ∇f (w) 2 turns into a true Lyapunov function for the dynamic (3.2) if d dt ∇f (w) 2 ≤ 0 for every w ∈ R d . To get a linear rate on ∇f (w) 2 , we need to characterize the right-hand side of (3.4). Notice that the first term is a quadratic form of ∇f (w), while the second term is less tractable due to the expectation. We build an upper bound in Lemma B.1 to tackle this second term, and it is achieved through "pulling out" the integrand B i (w) from the expectation and forming a quadratic form that is more friendly to the spectral analysis to follow. Lemma B.1 then leads to a more tractable upper bound for the right-hand side of (3.4), as illustrated in Lemma 3.8. If we replace the inequality in (3.3) with an equality, the resulting ode on ∇f (w) 2 is subject to a closed-form solution, which converges to a constant smaller than σ 2 /4 when α is small. This solution serves as an upper bound on ∇f (w(t)) 2 for any t ≥ 0. By making sure that the upper bound in (3.3) is strictly less than zero, it enables us to provide sufficient conditions on the maml step size α so that the Lyapunov function is convergent in linear rate to a small constant, as explained in Theorem B.2.
However, the upper bound on the Lyapunov function ∇f (w) 2 does not converge to zero, we can only guarantee in phase one that ∇F (w) goes below a constant. This issue will be resolved in phase two of the analysis.

Small Gradient Phase: Unique Global Minimum via Landscape Analysis
Due to the aforementioned limitations of the Lyapunov method, we propose a landscape analysis that complements the above argument and guarantees the linear rate of the maml ode when the gradient norm ∇F (w) on the maml loss is small enough. Recall that the maml ode is a gradient flow on F , thus the landscape of F determines the behavior of the maml ode. If a function is strongly convex, then its gradient flow converges linearly to its unique minimizer. Even though the maml loss F is not convex in general, we are able to show in Theorem 3.9 that for any point w ∈ R d with a bounded gradient norm ∇F (w) , the maml loss F is both smooth and strongly convex in its neighborhood. This provides us with a powerful tool that enables us to show the global convergence of the maml ode, as indicated in Theorem 3.10.
contains the unique critical point of F .
We remark that even though Theorem 3.10 concludes that the maml loss F is strongly convex within V , F can be non-convex outside V (recall our example in Fig. 1). Being a sublevel set of a strongly convex function f , the set V is convex. Moreover, it is also closed and bounded. Again, by the strong convexity of f , its minimum min w ∈R d f (w ) exists and is finite. Theorem 3.10 implies that there is no critical point outside V . Since F is strongly convex within V , the unique critical point inside the convex sublevel set V is consequently the global minimizer of F .

Numerical Experiments
Our experiments evaluate the bi-maml algorithm proposed in Section 3 against the maml algorithm on a series of learning problems. More specifically, we compare both methods on two different tasks: linear regression and binary classification with a Support Vector Machine (svm). They correspond to two different types of task loss f i : strongly convex and convex. For both types of problem we compare the methods on both synthetic and real data.
Linear regression. For the linear regression problem with synthetic data, we meta-learn the model parameter γ ∈ R d from a set of generated data that correspond to M = 10 individual linear regression tasks, each with dimension d = 20. A ground truth vector γ i ∈ R d is generated independently for each individual task. Each γ i has its coordinates drawn from i.i.d. standard normal distributions. From each task, we generate n = 100 training samples {x j , y j } ∈ R d × R, where each entry of x j is subject to an i.i.d. standard normal distribution and y j = x j γ i + σZ j where Z j is a standard normal variable and σ = 1 in our case. We also initialize the model parameter γ ∈ R d randomly in the way we generate γ i , and the maml step size α = 0.3 is fixed for all tasks. Each linear regression task i has a strongly convex loss function f i (γ) = y − Xγ 2 , where X = [x 1 , . . . , x n ] and y = [y 1 , . . . , y n ] . We take the gradient descent step size β = 0.05 for both bi-maml and maml, and we present the numerical results in Figs. 2a and 2e. For the linear regression problem on the Diabetes data set, we divide the original data into M = 4 tasks with dimension d = 8 according to male/female and whether the person is older than mean age of the data. We initialize the model parameter β randomly with i.i.d. normal entries, and the maml step size α = 0.5 is fixed for all tasks. We choose β = 0.1 and run the algorithms. Numerical results are presented in Figs. 2b and 2f. Compared to maml, our bi-maml algorithm in Fig. 2a converges to a neighborhood of stationary point of F in 25 steps, where maml is far from convergence. After 50 iterations, bi-maml also shows its superiority in computational efficiency over maml by having 37% and 36% decrease in computation time on synthetic and real data, respectively. Support vector machine. For binary classification with an svm, we meta-learn from M = 50 binary classification tasks in d = 20 dimensional space, where each one of the individual tasks is composed of n = 300 samples {x j , y j } ∈ R d × {±1}, evenly split between the positive and negative classes. We generate these training data sets with the default data generating function from scikit-learn. For each task i, it is equipped with a convex (but not strongly convex) hinge loss f i (w) = ReLu(y − Xw), where X = [x 1 , . . . , x n ] and y = [y 1 , . . . , y n ] . During the experiment, we take the maml parameter α = 0.3 for all tasks, and the gradient descent step size β = 0.1 is used for both the bi-maml and the maml. The corresponding numerical results are presented in Figs. 2c and 2g. For the binary classification on MNIST, we design M = 5 tasks, each to be a classification on different digits (classify 1, 3, 5, 7, 9 against 2, 4, 6, 8, 0), which has dimension d = 784. Each task has n = 200 samples, and the maml parameter α = 0.3 is fixed for all tasks. We take β = 0.01 for the experiment, and the result is reported in Figs. 2d and 2h. Compared to maml, our bi-maml algorithm in Fig. 2g reaches the region with small gradient norm ∇F (w) in less than 30 steps, where maml still has a much larger gradient. In these experiments, bi-maml shows its superiority in computational efficiency over maml by having a 19% and 37% decrease in computation time on the synthetic and real data, respectively. It is noteworthy that even if our method only provably works for strongly convex loss functions f i , it empirically achieves good performance on real world data for many strongly convex and even convex loss functions, such as svm. We remark that the gradient norm for the bi-maml shown in Figs. 2e to 2h represents different quantities in different stages of the algorithm. Specifically, it denotes ∇F (w k ) when the bi-maml descends on F at the k-th iteration; analogously, it represents ∇f (w k ) when the bi-maml descends on f . The jumps in Figs. 2d, 2g and 2h show clearly the transitions between two phases.

Conclusions
In this paper we analyze the maml ode, a continuous-time limit of maml, and establish a linear convergence rate to the global minimum of the maml loss function for strongly convex task losses. We also propose a computationally efficient algorithm bi-maml where its continuous-time limit bimaml ode has the same linear convergence guarantee under milder conditions. We experimentally show that the bi-maml method outperforms maml in a variety of learning tasks.

A Proof of Proposition 3.1
Proof. Recall the maml algorithm with update Eq. (3.1), i.e., α∇f i (w)). Expand the terms to get where the first equality follows from definition, the third equality follows from mean value theorem. Herew i is a value between w and w − α∇f i (w) such that mean value theorem holds. The formula can be further recast into If we think of the infinitesimal step size β 0, we obtain an ode that represents the gradient flow on F (w):ẇ We define a shorthand B i (w) for notational convenience.

B Proof of the Convergent Upper Bound
Proof. Another upper bound for the third term on the right-hand side of Eq. (3.4) can be derived through relaxing its difference with the quadratic form where the last inequality follows from Young's inequality. This provides yet another upper bound after rearranging the terms as follows: The first and second inequality are due to Assumptions 3.2 and 3.4. Recall that and it is not hard to see that max i B i (w) ≤ 2αL + α 2 L 2 . Hence we conclude that where the last inequality follows from α < 1 2L .
We derive sufficient conditions for the quadratic inequality 1 2 The sufficient conditions reduce to α < min and we have

Lemma B.3. Suppose the loss function f i (w) satisfies Assumptions 3.2 and 3.4, then for any
). Apply mean value theorem to ∇f i (w − α∇f i (w)) to get wherew i lies between w and w − α∇f i (w). Consequently, The second equality follows from separating the difference between ∇F (w) and ∇f (w). The third inequality is due to Eq. (B.2) and triangular inequality. The last inequality is due to Cauchy-Schwarz inequality, and the product of the two norms can be handled seperately. Expand A i (w), A i (w i ) and bound the first term by a constant to get The remaining term can be bounded by variance σ and gradient norm ∇f (w) : The second inequality follows from Jenson inequality. Combining the upper bounds together yields

Proof of Theorem 3.7
Proof. Since the expected loss f is µ-strongly convex, we always have in the first stage that It reaches a tolerant level at ∇f (w) ≤ ε 0 , as long as Let us denote By Lemma B.3 and the assumption α ≤ 1 2L we have Let us denote K = max{ 9 4 ε 0 + 5 4 σ, L µ +1 σ}, and Theorem 3.9 implies that if α ≤ min{ 1 2L , µ 8κ(2K+σ) } the maml loss F (w) is µ 8 -strongly convex at w, and the maml ode (3.2) after time t 1 is a gradient flow on a µ 8 -strongly convex loss F (w). This dynamics then converges exponentially fast to an approximate stationary point w where ∇F ( w) ≤ ε. Similar to the proof of Theorem 3.6, a sufficient condition for the approximate stationary point w writes e −µτ /8 ∇F (w(t 1 )) 2 ≤ ε, which means w(τ + t 1 ) is an approximate stationary point if Combine two parts together to get the major result that the bi-maml ode converges to an approximate stationary point w(t) within

C Proof of Strong Convexity
where the first inequality follows from triangular inequality and the last inequality is due to Assumption 3.2 and α < 1 4L . Similarly, we have where the first inequality is due to triangular inequality and the second one is due to Assumption 3.4. Rearrange the terms under the assumption α < 1 4L to get

Lemma C.2. Suppose f i (w) satisfies Assumptions 3.2, 3.3 and 3.5. For any
Proof. Consider w, u ∈ U (G), we have where the inequality follows from triangular inequality. For the first term, we have an upper bound where the first inequality is due to Cauchy-Schwarz inequality, the second inequality is due to Assumption 3.5, and the second equality follows from mean value theorem, and the last inequality is due to the fact that A(w) = I d − α∇ 2 f (w) ≤ 1 − αµ. Similarly, we bound the second part as where the last inequality follows from mean value inequality. Putting the pieces together to get, when α ≤ min{ 1 2L , µ 8κG }, and therefore Hess(F (w)) 9L 8 I d . The corresponding lower bound similarly follows from triangular inequality where When α ≤ min{ 1 2L , µ 8κG }, the first term is lower bounded as where the first inequality follows from λ min (A(u)) ≥ 1 − αL, the second inequality follows from Assumption 3.3, the third inequality is due to triangular inequality, and the last inequality follows from α ≤ 1 2L . Hence, it holds that where the last inequality follows from α ≤ µ 8κG . Thus we obtain Hess(F (w)) ≥ µ 8 .

D Proof of Theorem 3.10
For Proof. Let us pick w ∈ R d such that ∇F (w) ≤ K. Lemma C.1 implies that there exists a constant C 1 = 2K + σ such that ∇f (w) ≤ C 1 . Since f is µ-strongly convex, we have Setting x to the global minimizer x * of f yields Therefore, we have w ∈ V 1 2µ (2K + σ) 2 .

Lemma D.2. Under Assumptions 3.2 to 3.4, we have
Proof. Let us rewrite ∇F (w) as below where the second inequality is because of the mean value theorem. Since f is L-smooth, we have Since f is µ-strongly convex, there exists a unique global minimum x * with ∇f (x * ) = 0. Therefore, we obtain f (w) ≥ f (x * ) + 1 2L ∇f (w) 2 .
Combining Lemmas D.1 and D.2 gives the following corollary. Proof. First we show that F is bounded from below. Since every f i is strongly convex, it is bounded from below. Recall that F (w) = E i∼p f i (w − α∇f i (w)). Therefore F is also bounded from below. Let F * := inf w∈R d F (w). Pick any v(0) ∈ R d and consider the dynamic defined by .

2L
is convex and non-empty. Thus U (K) is non-empty and closed. Next, we show that U (K) is bounded. Lemma D.1 implies U (K) ⊆ V 1 2µ (2K + σ) 2 := V 0 . Since V 0 is a sublevel set of f and f is strongly convex, therefore we get the boundedness of V 0 , which implies the boundedness of U (K). Thus U (K) is compact. Define a sequence w n = u n + E(0) K , where n = 1, 2, 3, . . . . We have w n ∈ U (K). By Bolzano-Weierstrass theorem, there exists a convergent subsequence w n i . Let w 0 ∈ U (K) be the limit of w n i . We have Therefore we conclude that w 0 is a critical point of F .

2L
is convex and non-empty. Theorem 3.9 implies that Hess(F (w))

2L
. In fact, the unique critical point is the global minimizer of F .