{'fraction_non_alphanumeric': 0.03736044408407659, 'fraction_numerical': 0.014875569138651531, 'mean_word_length': 4.940533530937384, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Training Stronger Baselines for Learning to Optimize

Learning to optimize (L2O) has gained increasing attention since classical optimizers require laborious problem-specific design and hyperparameter tuning. However, there is a gap between the practical demand and the achievable performance of existing L2O models. Specifically, those learned optimizers are applicable to only a limited class of problems, and often exhibit instability. With many efforts devoted to designing more sophisticated L2O models, we argue for another orthogonal, under-explored theme: the training techniques for those L2O models. We show that even the simplest L2O model could have been trained much better. We first present a progressive training scheme to gradually increase the optimizer unroll length, to mitigate a well-known L2O dilemma of truncation bias (shorter unrolling) versus gradient explosion (longer unrolling). We further leverage off-policy imitation learning to guide the L2O learning, by taking reference to the behavior of analytical optimizers. Our improved training techniques are plugged into a variety of state-of-the-art L2O models, and immediately boost their performance, without making any change to their model structures. Especially, by our proposed techniques, an earliest and simplest L2O model can be trained to outperform the latest complicated L2O models on a number of tasks. Our results demonstrate a greater potential of L2O yet to be unleashed, and urge to rethink the recent progress. Our codes are publicly available at: https://github.com/VITA-Group/L2O-Training-Techniques.


Introduction
Optimizer Optimizee (e.g. RNN) Variables Error Signal (e.g. gradient) Parameter Update Learning to optimize (L2O) [1][2][3][4][5][6][7][8][9][10], a rising sub-field of meta learning, aims to replace manually designed analytical optimizers with learned optimizers, i.e., update rules as functions that can be fit from data. An L2O method typically assumes a model to parameterize the target update rule. A trained L2O model will act as an algorithm (optimizer) itself, that can be applied to training other machine learning models, called optimizees, sampled from a specific class of similar problem instances. The training of the L2O model is usually done in a meta-fashion, by enforcing it to decrease the loss values over sampled optimizees from the same class, via certain training techniques.
Earlier L2O methods refer to black-box hyper-parameter tuning approaches [11][12][13][14][15][16] but these methods often scale up poorly [17] when the optimizer's parameter amounts grow large. The recent mainstream works in this vein [1,[6][7][8][9] leverage a recurrent network as the L2O model, typically long-short term memory (LSTM). That LSTM is unrolled to mimic the behavior of an iterative optimizer and trained to fit the optimization trajectory. At each step, the LSTM takes as input some optimizee's current measurement (such as zero-and first-order information), and returns an update for the optimizee. A high-level overview of L2O workflow is presented in Figure 1.
Although learned optimizers outperform hand-designed optimizers on certain problems, they are still far from being mature or practically popular. Training LSTM-based L2O models is notoriously difficult and unstable; meanwhile, the learned optimizers often suffer from poor generalization. Both problems arise from a well-known dilemma, on whether the LSTM should be unrolled shorter or longer during L2O training [17][18][19][20]. For training modern models, an optimizer (either hand-crafted or learned) can take thousands or more of iterations. While naively unrolling the LSTM to this full length is impractical, and most LSTM-based L2O methods [1, 6-9, 17, 21, 22] take advantage of truncating the unrolled optimization; as a result, the entire optimization trajectory is divided into consecutive shorter pieces, where each piece is optimized by applying a truncated LSTM. However, choosing the unrolling/division length faces a well-known dilemma [7]: on one hand, a short-truncated LSTM implies too early termination of the iterative solution, causing the so-called "truncation bias" that the learned optimizers exhibit instability and yield poor-quality solutions when applied to training optimizees; on the other hand, although a longer truncation is favored from the optimizer sense, it incurs common training pitfalls for LSTM such as gradient explosion [19].
Our Contributions To tackle those challenges, a number of solutions [7][8][9]17] were proposed, mainly devoted to designing more sophisticated architectures of L2O models (LSTM variants), as well as enriching the input features for L2O models [7][8][9]. In contrast, this paper seeks to improve L2O from an orthogonal perspective, that is: can we train a given L2O model better? We offer a toolkit of novel training techniques, that can be plugged in training existing L2O models: • We propose a progressive training scheme to gradually unroll the learned optimizer, based on an exploration-exploitation view of L2O. We find it to effectively mitigate the dilemma between truncation bias (shorter unrolling) v.s. gradient explosion (longer unrolling).
• We introduce off-policy imitation learning to further stabilize the L2O training, by taking reference to the behavior of hand-crafted optimizers. The L2O performance gain by imitation learning endorses the implicit injection of useful design knowledge of good optimizers.
• We report extensive experiments using a variety of L2O models, optimizees, and datasets. Plugging in our improved training techniques immediately boosts all state-of-the-art L2O models, without any other change. Especially, the earliest and simplest LSTM-based L2O baseline [1] can be trained to outperform latest and much more sophisticated L2O models.
We summarize our take-home message for the L2O community: besides developing more complicated models, training existing simple baselines better is equally important. Only by fully unleashing "simple" models' potential, can we lay a solid and fair ground for evaluating the L2O progress.

Related Work
Learning to Optimize L2O uses a data-driven learned model as the optimizer, instead of handcrafted rules (e.g., SGD, RMSprop, and Adam). [1] was the first to leverage an LSTM as the coordinate-wise optimizer, which is fed with the optimizee gradients and outputs the optimizee parameter updates. [6] instead took the optimizee's objective value history, as the input state of a reinforcement learning agent, which outputs the updates as actions. To train an L2O with better generalization and longer horizons, [7] proposes random scaling and convex function regularizers tricks. [8,23] introduce a hierarchical RNN to capture the relationship across the optimizee parameters and trains it via meta learning on the ensemble of small representative problems. Besides learning the full update rule, L2O was also customized to automatic hyperparamter tuning in specific tasks [24][25][26].
Curriculum Learning The idea [27] is to first focuses on learning from a subset of simple training examples, and gradually expanding to include the remaining harder samples. Curriculum learning often yields faster convergence and better generalization, especially when the training set is varied or noisy. [28] unifies it with self-paced learning. [29] automates the curriculum learning by employing a non-stationary multi-armed bandit algorithm with a reward of learning progress indicators. [30][31][32][33] describe a number of applications where curriculum learning plays important roles.
Imitation Learning Imitation learning [34,35], also known as "learning from demonstration", is to imitate an expert demonstration instead of learning from rewards as in reinforcement learning.
In our work, L2O imitates multiple hand-crafted experts, such as Adam, SGD, and Adagrad. A relevant work is the "Lookahead Optimizer" [36], which first updates the "fast weights" k times using a standard optimizer in its inner loop before updating the "slow weights" once. Another work of self-improving [37,38] chooses several optimizers for hybrid training, including the desired L2O and several other auxiliary (analytical) optimizers, per a probability distribution. It then gradually anneals the probability of auxiliary optimizers and eventually leaves only the target L2O model in training.

Strengthening L2O Training with Curriculum Learning
Problem Setup for L2O The goal is to learn an optimizer parameterized by φ, which is capable of optimizing a similar class of optimizee functions f (θ). In the case of machine learning, θ is the model parameter and f (θ) is the objective loss. Usually, the learned optimizer outputs ∆θ t and the optimizee parameter is updated as θ t+1 = θ t + ∆θ t . The inputs of the learned optimizer can be ∇f (θ t ) or other optimizee features. The learned optimizer is trained by a loss L f (φ) = Ntrain t=1 ω t f (θ t ; φ) where N train is called the horizon of optimization trajectory (i.e., the unrolling length).

An exploration-exploitation view for L2O
Training an L2O model follows the basic idea of meta learning, by applying the L2O model to training a number of sampled optimizees, and enforcing it to decrease their loss values as much as possible. We refer the readers to [1] for the standard training setting in details.
During training L2O, we recognize that the the truncated optimization (e.g., unrolling LSTM to a limited number of iterations) is one of the main culprits, for the instability and poor quality when applying trained L2O to optimizees. That is because for each update, L2O can only learn from a short segment of optimization trajectory, so that it sacrifices generalization ability to longer horizons [7].
One plausible remedy for alleviating this truncation bias is to augment L2O training with more optimizees, as well as longer optimization trajectory for each optimizee. This can be interpreted as addressing an exploration-exploitation problem, a typical challenge in reinforcement learning [39]. On one hand, the agent (L2O) has to exploit policies (i.e., optimization trajectory for each optimizee currently being trained) that are previously better rewarded, in hope to gain more rewards at present. On the other hand, the same L2O agent also needs to explore unfamiliar state-action choices (here more optimizees) to discover other potentially (more) favorable policies.
Exploration: training with more optimizees A good learned optimizer needs to explore and see various optimizees' landscapes, in order to generalize to tackle unseen optimizees with more diverse landscapes. More exploration could be considered as a special form of data augmentation, which was found beneficial for L2O training [7].
One straightforward option is sampling more optimizees. In practice, we find an even easier-toimplement alternative to be the same effective: we re-use sampled optimizees, by starting from their different random weight initializations. It is similar to the "exploring starts" in reinforcement learning. Note that this also artificially enlarges our meta-training set and thus calls for meta-epochs.
Exploitation: training longer for each optimizee A good learned optimizer also shall aim to be fully executed, so as to reach low optimizee loss and high-precision solution. That is especially important for training the L2O model to see the landscapes close to the minimum. Moreover, just like any other iterative optimizer, a learned optimizer predicts its current update from the past history, and therefore should also benefit from more training exposure to longer-term dependency.
In view of the above two, the first (basic) improvement we take is to simply train the L2O model with more optimizees, and longer optimization trajectories for each. Naive as it might look like, in section 5.1 we will show that they clearly benefit existing L2O models, and solidify our starting point.

Curriculum learning to adaptively balance exploration and exploitation
Inspired by the exploration-exploitation view, we next introduce a new training strategy based on curriculum learning [27], for the first time to L2O models. The curriculum is defined by gradually unrolling the optimizer more, w.r.t. the number of epochs. Specifically, we denote the number of training unrolling steps by N train , and the number of validation unrolling steps by N valid . We progressively increase N train and N valid during training, until the validation loss stops to decrease. valid as the baseline validation loss of the (i + 1) th training stage. If none of the N period validation loss in the (i + 1) th training stage is lower than the baseline loss, we stop training and export the best model of the i th training stage, as our final trained L2O model. The overall procedure is summarized in Algorithm 1.
Generate an optimization trajectory by L2O φ; 10 Update φ with respect to The above curriculum training naturally enforces an adaptive balance between exploration and exploitation. We begin L2O training with a small number of optimizees sampled (exploration), and each trained with small unrolling steps (exploitation). As the exploration grows (more training epochs), each optimizee is simultaneously exploited more, with a larger number of steps unrolled for optimization. In this way, we effectively avoid long chaotic trajectories generated by optimizers that have not been trained sufficiently at the beginning. Moreover, the optimizer can focus on learning simpler, short-horizon trajectory patterns firsts, and gradually grow to covering longer dependency, without overfitting difficult corner-cases. It is shown to alleviate the intrinsic bias caused by truncated optimization at different L2O training phases. As a result, the L2O model converges quickly and more stably, and ends up reaching lower validation losses.

Imitation learning: multi-task regularization by analytical optimizers
In this section, we propose another L2O training method based on imitation of analytical optimizers behaviours, through a multi-task learning form, which is found to further stabilize our training, prevent overfitting, and improve the trained L2O models' generalization.
The learned optimizer may "overfit" some training optimizees if it learns some optimization policies that are only applicable to some but not generalize to more unseen optimizees. To enforce a generalizability constraint, we regularize our L2O training by imitating some general optimization policies, which are known to generalize well and agnostic to data. For this purpose, we let the L2O model follow optimization trajectories generated by analytical optimizers such as Adam [40]. That imitation also help prevent the LSTM model's "spiking" or other highly non-smooth update predictions sometimes, stabilizing the optimization trajectories.
Generally, the learned optimizer outputs the optimizee parameter updates ∆θ t and use θ t+1 = θ t + ∆θ t to calculate the optimizee f (θ t+1 ). The optimizer is trained by a loss L f (φ) = Ntrain t=1 ω t f (θ t ; φ) on an optimization trajectory of length T generated by the current learned optimizer φ. Now given another analytical optimizer O such as Adam optimizer [40] as a "teacher", we first use O to generate an optimization trajectory of length N train on the training optimizee f . The trajectory is denoted by T The relation between the inputs (g t ) t∈ [1,Ntrain] and the outputs (∆θ O t ) t∈ [1,Ntrain] is determined by the optimization policies of O. Now we calculate another training objective which is the weighted square error between the label updates (∆θ O t ) t∈ [1,Ntrain] given by O and the updates given by the current learned optimizer φ as (∆θ φ t = φ(g t )) t∈ [1,Ntrain] as shown in equation (1): Note that it is analogous to off-policy learning [41] in reinforcement learning where the agent is trained on a trajectory generated by another behavior policy. In our case, the learned optimizer is trained on the trajectory generated by the typically more stable analytical optimizer O. We adjust the weight between the two losses L f and L O for multi-task learning as shown in algorithm 2. Note that in reinforcement learning, off-policy is usually aimed at boosting exploration, while our goal differs here, i.e., to stabilize L2O training learn more generalizable optimizers.

Imitation Learning versus Self Improving
A self-improving approach [37,38] was lately adapted for L2O training, by sampling and mixing different optimizers in one training pass (i.e., each iteration may adopt a different optimizer's update), according to a probability distribution. Then, the probability of choosing other optimizers will be gradually annealed, so that only the desired optimizer will remain at the end. Compared with imitation learning, self-improving technique only produce a single optimization trajectory T s which consists of mixture update from different optimizers. Following the notation in the Section 4, the generated trajectory is presented as T s = [(g 1 , ∆θ Oi 2 2 ), ..., (g Ntrain , ∆θ Oi train [1,Ntrain] are the corresponding update rule. Let {O 0 , · · · , O k } is the candidate pool of optimizers, where O 0 is the desired L2O and {O 1 , · · · , O k } are the auxiliary optimizers. At the beginning, {O i k } train 1 are sampled from {O 0 , · · · , O k } respect to a multinomial distribution M(p 0 , p 1 , · · · , p k ), p 0 = p 1 = · · · = p k = 1 k+1 . Then, along with the increase of training epochs, p 1 = · · · = p k decay to zero and p 0 grow to 1. In this way, the desired L2O is steadily improved with the assistance of auxiliary optimizers.
In this section, we conduct systematic experiments to evaluate our proposed training techniques. Our experiments are organized into three parts, respectively showing: • The earliest and simplest LSTM-based L2O baseline [1] can surpass some latest sophisticated L2O models, by training with our proposed techniques. • Our improved training techniques can be further plugged into previous state-of-the-art L2O methods and yield extra performance boosts for them all. • All the components in our proposal matter, i.e., an extensive ablation study for validating the respective gain of each technique.
Optimizee Settings For the fair comparison purpose, in our all experiments, we train the optimizer on the same single optimizee as in [1], which uses the cross-entropy loss on top of a simple Multi-layer Perceptron (MLP) with one hidden layer of 20 dimensions and the sigmoid activation function on the MNIST dataset. Validation also follows to use the same optimizee as in [1]. We consider five optimizee problems during testing, to evaluate the generalization ability of the learned optimizer: • v) Conv-CIFAR: the above CNN trained on CIFAR-10, with all identical architecture configurations except using stride 2.
Optimizee i) is designed for sanity check and to exclude randomness factors. Optimizees ii)-iv) are for evaluating the generalization of L2O across network architectures. Finally, optimizee v) evaluates the generalization of L2O across both network architectures and datasets.
Training Details and Evaluation L2O are trained by default meta optimizers with the best hyperparameters provided by each baseline: L2O-DM [1] and RNNprop [7] are optimized by Adam with an initial learning rate of 1 × 10 −3 ; L2O-Scale [8] and its variants are optimized by RMSProp with an initial learning rate of 1 × 10 −6 . The optimizee parameters are initialized by a random normal distribution of standard deviation of 0.01. The batch size for optimizees is 128. The unroll length is fixed to 20 except for the meta learning baseline of L2O-Scale where both the number of optimization steps and unroll lengths are sampled from long-tail distributions [8]. In the testing phase of L2O, we evaluate learned optimizers on unseen testing optimizees for 10, 000 steps every time, and then report the training loss of testing optimizees. Other hyperparameters are strictly controlled to be fair. Our results come with multiple independent runs, and the error bars are reported in the Appendix.

Training the L2O-DM baseline to surpass the state-of-the-arts
In this section, we equip the "simple baseline" L2O-DM with our improved training techniques, including the curriculum learning scheduler (CL) and the imitation learning regularizer (IL). L2O-DM-CL-IL denotes the enhanced L2O model. Learned optimizers are evaluated on five representative optimizees and the corresponding optimizee training loss are collected in Figure 2.
From the results in Figure 2, we observe that the previously noncompetitive L2O-DM, that initially even cannot stably converge on the Optimizee i) at long horizons, now consistently and largely outperforms over all previous SOTA L2O methods: RNNprop, L2O-Scale, and L2O-Scale-Meta, by decreasing the objective loss value much lower. Compared to the vanilla L2O-DM, CL and IL are also  In this section, we validate that the power of our proposed techniques can extend to improving previous SOTA L2O methods. CL and IL are applied to L2O-Scale and RNNprop, leading to corresponding strengthened models, named L2O-Scale-CI-IL and RNNprop-CI-IL, respectively 3 .
As shown in Figure 3, our training techniques also immediately improve the two L2O models on all five optimizees, in terms of both objective value and the convergence stability.

Ablation study of our proposed techniques
For simplicity, we take L2O-DM as an illustrative example in the ablation case study. More results and details about our case study are referred to Appendix A1.
Exploration and Exploitation As shown in section 3.1, we can better balance exploration and exploitation by training L2O with more instances and longer for each instance. We set the number of training/validation steps to 100, 200, 500, 1000 respectively. We pick the model with the lowest validation loss during each 1, 000 epochs so that we save 10 models for each experiment. We then evaluate the models by running these learned optimizers on MLP-orig optimizees while extending to 10, 000 training steps. The average performance of 20 independent runs with different random seeds are reported, and the error bars are collected in the Appendix A1. From the results in Figure 4, we gain the following two observations: • With extra training epochs, the evaluation curve diverges, or ends up with larger losses. It suggests that redundant training epochs incur overfitting since it may amplify the bias on certain unrolling length ranges. • The evaluation curves are less likely to diverge or degrade when increasing training epochs with more training optimization steps. The possible explanation is that longer exploitation helps with a more thorough landscape exploration and hence alleviates overfitting.
The best L2O model in Figure 4, training 5, 000 epochs with optimization step size 1, 000, is adopted as a strong augmented baseline (L2O-DM-AUG) for next-step comparisons. Curriculum Learning We then assess the proposed curriculum learning [27] as in section 3.2. As shown in Figure 5, L2O-DM-CL surpasses L2O-DM-AUG with a significant performance margin, while only needs less than 1/14 training iterations of L2O-DM-AUG, thanks to its strategical focus on simpler training cases first. More details of CL and its cost analysis are referred to the Appendix. Conv-MNIST Baseline Self Improving Imitation Learning Figure 6: Imitation Learning v.s Self-Improving Imitation Learning We introduce off-policy imitation learning (IL) of analytical optimizers (Adam, SGD and Adagrad) as described in section 4 on L2O-DM. As shown in Figure 5, we observe that L2O-DM-IL converge better, especially on MLP-ReLU, Conv-MNIST and Conv-CIFAR where even L2O-DM-AUG tends to diverge. It suggests that optimization behaviors imitated from the analytical optimizers are in general beneficial, which adds extra effectiveness on top of CL as indicated by Figure 2 and 5. Besides, IL improves the stability of optimizee training. Appendix offers more analysis and more detailed settings for IL.

Imitation Learning versus Curriculum Learning
We compare the performance across L2O-DM-CL-IL, L2O-DM-CL and L2O-DM-IL, as presented in Figures 2 and 5. We observe: i) On MLP optimizees, curriculum learning (CL) improves more than imitation learning (IL); On CNN optimizees, IL contributes more to the performance gain. Here is a possible explanation. The curriculum learning (CL) technique mainly helps alleviate the notorious L2O truncation bias, while the imitation learning (IL) approach mainly helps L2O refer to the generally applicable optimization rules and hence avoid overfitting specific optimizee structures. Thus, IL appears to be a main contributor for L2O generalizing across different optimizees. ii) Combining CL and IL techniques always enjoys extra performance boost, compared to using either alone.
Imitation Learning versus Self-Improving We also compare our proposed imitation learning regularizer with the self-improving techniques [38] which also take Adam, SGD, and Adagrad as their auxiliary optimizers to mix with. The probability of choosing L2O gradually grows to 1, while other probabilities linearly decay from 0.33 to 0 in 100 epochs of optimzier training. Figure 6 reports the previous SOTA L2O, L2O-Scale, as the baseline, and apply the two techniques (IL, and self-improving) on the top of it. Results demonstrate that while self-improving also demonstrates to be helpful in our case, IL shows to be certainly superior. We believe that the key reason lies in the long-term coherency of optimization trajectories. The imitation learning technique allows L2O to learn from the entire trajectories, with hand-crafted optimizers serving as end-to-end guidance. In comparison, self-improving breaks each optimization trajectory into mixed local "pieces" of applying either analytical or learned optimizers, therefore restricting learned optimizers to capturing only the dependency within local segments (e.g., a few iterations).

Comparison against Hand-designed Optimizers
We also conduct comparison against handdesigned optimizers on MLP-orig and Conv-MNIST optimizees with MNIST dataset, as presented in Figure 7 [a] and [b]. Plugging our proposed enhanced training techniques immediately boosts L2O to outperform analytical optimizers (i.e., Adam, RMSProp, and SGD) whose hyperparameters have been optimally tuned via a grid search, while the vanillan L2O-DM [1] quickly collapses and diverges in the same setting.

Effectiveness on Complex Optimizees
To show the effectiveness of our proposal on complex optimizees, we consider a non-conventional architecture sampled from a neural architecture search benchmark, and a larger LeNet model [42]), both on the CIFAR-10 dataset. The former, termed as NAS-CIFAR, is taken from the popular NAS-Bench-201 search space [43]. It 4 consists of multiple skip connections, convolutions, and average pooling operations, which is significantly more complicated than previously used simple instances of MLPs and CNNs. As shown in Figure 7 [c], our proposed techniques enable L2O trained on single-layer MLPs to generalize robustly to the much more sophisticated NAS-CIFAR, where vanillan L2O fails. We further validate our proposed training techniques can also successfully scale up to the LeNet case, as evidenced in Figure 7 [d].

Conclusion
Learning to optimize (L2O) is a promising field of meta learning that has so far been a bit held back by unstable L2O training and the poor generalization of learned optimizers. This work provides practical solutions to push this field forward. We propose a set of improved training techniques to unleash the great potential of L2O models. We apply our techniques to existing state-of-the-art L2O methods and consistently obtain performance boosts on a number of tasks. The contributions made in this work are of practical nature; we hope them to lay a solid and fair evaluation ground by offering strong baselines for the L2O community.

Broader Impact
This work mainly contributes to AutoML in the aspect of discovering better learning rules or optimization algorithms from data. As a fundamental technique, it seems to pose no substantial societal risk. This paper proposes several improved training techniques to tackle the dilemma of training instability and poor generalization in learned optimizers. In general, learning to optimize (L2O) prevents laborious problem-specific optimizer design, and potentially can largely reduce the cost (including time, energy and expense) of model training or tuning hyperparameters.

Exploration and Exploitation
As shown in Figure A1, The average performance of 20 independent runs with different random seeds are reported, and the error bars are also collected.   iterations while L2O-DM-CL is only trained for (100 + 200 + 500 + 1, 000) × 300 = 360, 000 iterations (the best model is saved during N train = 500). It demonstrates that L2O-DM-CL surpasses L2O-DM-AUG with a significant performance margin, while only needs less than 1/14 training iterations of L2O-DM-AUG, thanks to its strategical focus on simpler training cases first.
From the view of the potential advantages of curriculum learning as hypothesized in [?], the easy training examples represent a smoother global picture of the objective landscape. Therefore, the learner can avoid being trapped in local minimum at the beginning and converges to a better minimum. Furthermore, learning from a smoother objective landscape can also reduce the time wasted by noise examples and thus accelerate the convergence. In our specific case of learning to optimize, a training example input is a trajectory of optimizee parameters with corresponding gradients and updates. Smaller N train means shorter trajectories. As a result, the starting training example inputs are in the neighborhood of the initialization distribution of the optimizee parameters. Longer trajectories will cover more complex landscapes. Besides, a longer trajectory means longer horizontal dependency for the adaptive optimizer to learn. In addition, our proposed algorithm also provides a stop criterion. We stop when training on a new larger N train does not bring further decrease on the validation loss within N period periods, which indicates the L2O has been saturated with optimization knowledge.

A1.3 Imitation Learning
L2O-DM-IL donates the enhanced L2O-DM with our proposed imitation learning regularizer, mimicking the optimization behaviors of analytical optimizers. We choose (O i ) i∈I as the ensemble of Adam, SGD, and Adagrad. The hyperparameters of Adam β 1 , β 2 are set to the default value 0.9 and 0.999 in Tensorflow. The learning rate of Adam, SGD, Adagrad, and the multi-task ratio r are chosen by grid search as 0.01, 0.01, 0.01, and 0.3, respectively. The ω t in equation (1) are set to 1. We train and validate both by 100 optimization steps with 5, 000 epochs.

Results and Analysis
The results are shown in figure A2. we observe that L2O-DM-IL converge better, especially on MLP-ReLU, Conv-MNIST, and Conv-CIFAR where even L2O-DM-AUG tends to diverge. The multi-task learning of imitation of analytical optimizers forces the L2O to learn some general optimization policies, which improves the stability of optimizee training. According to [?], multi-task learning has the effect of regularization by inducing bias so that it helps alleviating overfitting on random noise. Figure A3 also indicates that our proposed imitation learning technique has extra effectiveness on top of curriculum learning.

A1.4 Training the L2O-DM baseline to surpass the state-of-the-arts
As shown in Figure A3, The average performance of 10 independent runs with different random seeds are reported, and the error bars are also collected.

A2 More Training Details
We adopt the official training/testing split of MNIST and CIFAR-10 in our experiments.
• All of our experiments are conducted on NVIDIA GTX 1080-Ti GPUs.