{'fraction_non_alphanumeric': 0.03121962782979732, 'fraction_numerical': 0.011259926514163802, 'mean_word_length': 5.02141021981159, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data. To thoroughly study our assumptions on how to design such an AutoDL system, we additionally introduce a new benchmark on learning curves for DNNs, dubbed LCBench, and run extensive ablation studies of the full Auto-PyTorch on typical AutoML benchmarks, eventually showing that Auto-PyTorch performs better than several state-of-the-art competitors on average.


INTRODUCTION
In recent years, machine learning (ML) has experienced enormous growth in popularity as the last decade of research efforts has lead to high performances on many tasks. Nevertheless, manual ML design is a tedious task, slowing down new applications of ML. Consequently, the field of automated machine learning (AutoML) [1] has emerged to support researchers and developers in developing highperformance ML pipelines. Popular AutoML systems, such as Auto-WEKA [2], hyperopt-sklearn [3], TPOT [4], autosklearn [5] or Auto-Keras [6] are readily available and allow for using ML with a few lines of code.
While initially most AutoML frameworks focused on traditional machine learning algorithms, deep learning (DL) has become far more popular over the last few years due to their strong performance and their ability to automatically learn useful representations from data. While choosing the correct hyperparameters for a task is of major importance for DL [1], the optimal choice of neural architecture also varies between tasks and datasets [7], [8]. This has lead to the recent rise of neural architecture search (NAS) methods [9]. Since there are interaction effects between the best architecture and the best hyperparameter configuration for it, AutoDL systems have to jointly optimize both [10].
Traditional approaches for AutoML, such as Bayesian optimization, evolutionary strategies or reinforcement learning, perform well for cheaper ML pipelines but hardly scale to the more expensive training of DL models. To achieve better anytime performance, one approach is to use multi-fidelity optimization, which approximates the full optimization problem by proxy tasks on cheaper fidelities, e.g.
training only for a few epochs. Although there is evidence that correlation between different fidelities can be weak [11], [12], we show in this paper that it is nevertheless possible to build a robust AutoDL system with strong anytime performance.
To study multi-fidelity approaches in a systematic way, we also introduce a new benchmark on learning curves in this paper, which we dub LCBench. This new benchmarks allows the community and us to study in an efficient way the challenges and the potential for using multi-fidelity approaches on DL and combining it with meta-learning, by providing learning curves of 2 000 configurations on a joint optimization space for each of 35 diverse datasets.
Finally, we introduce Auto-PyTorch Tabular, an AutoML framework targeted at performing multi-fidelity optimization on a joint search space of architectural parameters and training hyperparameters for neural nets. Auto-PyTorch, the successor of Auto-Net [13] (part of the winning system in the first AutoML challenge [14]), combines state-of-the-art approaches from multi-fidelity optimization [15], ensemble learning [5] and meta-learning for a data-drive selection of initial configurations for warmstarting Bayesian optimization [16].
Specifically the contributions of this paper are: 1) We address the problem of jointly and robustly optimizing hyperparameters and architectures of deep neural networks. 2) We introduce LCBench 1 , a new benchmark for studying multi-fidelity optimization w.r.t. learning curves on a joint optimization space of architectural and training hyperparameters across 35 datasets. 3) We study several characteristics of LCBench to gain insights on how to design an efficient AutoDL framework, including correlation between fideli-ties and evaluation of hyperparameter importance across different fidelities. 4) We propose to combine BOHB, as a robust optimizer for AutoDL, with automatically-designed configuration portfolios and ensembling, showing the superior performance of this combination. 5) To achieve state-of-the-art performance on tabular data, we combine tuned DNNs with simple traditional ML baselines (i.e., RFs, catboost, extra trees, knn and lgb). 6) We introduce our Auto-PyTorch Tabular 2 tool to make deep learning easily applicable to new datasets for the DL framework PyTorch. 7) We show that Auto-PyTorch Tabular outperforms other common AutoML frameworks like Au-toKeras, AutoGluon, auto-sklearn and hyperoptsklearn.
In contrast to the main-stream in DL, here we do not focus on image data, but on tabular data, since we feel that DL is understudied for many applications of great relevance that feature tabular data such as climate data, classical sensor data in manufacturing or medical data files. Although we only study tabular data in this paper, the Auto-PyTorch approach is not limited to tabular data, but can in principle also be applied to other data modalities and tasks, such as image classification, semantic segmentation, speech processing or natural language processing.

RELATED WORK
In the early years of automated machine learning, AutoML tools, such as Auto-WEKA [2], hyperopt-sklearn [3], autosklearn [5] and TPOT [4], focused on traditional machine learning pipelines because tabular data was and still is important for major applications. Popular choices for the optimization approach under the hood include Bayesian optimization, evolutionary strategies and reinforcement learning [17]. In contrast to plain hyperparameter optimization, these tools can also address the combined algorithm selection and hyperparameter (CASH) problem by choosing several algorithms of a ML pipeline and their corresponding hyperparameters [2]. Although we focus on deep neural networks, our approach also addresses mixed configuration spaces with hierarchical conditional structures, e.g., the choice of the optimizer as a top-level hyperparameter and its sub-hyperparameters.
Another trend in AutoML is neural architecture search [9], which addresses the problem of determining a well-performing architecture of a DNN for a given dataset. Early NAS approaches established new state-of-the-art performances [7], but were also very expensive in terms of GPU compute resources. To improve efficiency, state-of-theart systems make use of cell-search spaces [18], i.e., configuring only repeated cell-architectures instead of the global architecture, and use gradient based optimization [19]. Although gradient-based approaches are very efficient on some datasets, they are known for their sensitivity to hyperparameter settings and instability [20]. Recent practices 2. https://github.com/automl/Auto-PyTorch for doing hyperparameter optimization only as a postprocessing step of NAS ignore interaction effects between hyperparameter settings and choice of architecture and also are an expensive additional step at the end.
Instead, we propose to jointly optimize architectures and hyperparameters by combining a robust approach based on multi-fidelity optimization and an efficiently designed configuration space, where we make use of repeated blocks and groups and minimize the number of hyperparameters by only describing shapes of these.
There is also a growing interest in well-designed Au-toML benchmarks to take reproducibility and comparability of AutoML approaches into account. For example, HPOlib [21] provides benchmarks for hyperparameter optimization, ASlib [22] for meta-learning of algorithm selection and NASBench101, 1Shot1 and 201 [11], [23], [24] for neural architecture search. However, to the best of our knowledge, there is yet no multi-fidelity benchmark on learning curves for the joint optimization of architectures and hyperparameters. We address that by our new LCBench, which also provides data on 35 datasets in order to allow combinations with meta-learning.
Last but not least, several papers address the problem of understanding the characteristics of AutoDL tasks [12], [25]. A typical finding for example is that the design space is over-engineered, leading to very simple optimization tasks where even random search can perform well. As we show in our experiments, this does not apply to our tasks. In addition to study the characteristics of LCBench and also the full design space of Auto-PyTorch Tabular, we study hyperparameter importance from a global [25], [26], [27], but also from a local point of view [28], showing that both provide different insights.

AUTO-PYTORCH
We now present Auto-PyTorch, an AutoDL framework framework utilizing multi-fidelity optimization to jointly optimize NN architectural parameters and training hyperparameters. As the name suggests, Auto-PyTorch relies on the PyTorch framework [29] as DL framework. Auto-PyTorch implements and automatically tunes the full DL pipeline, including data preprocessing, network training techniques and regularization methods. Furthermore, it offers warmstarting the optimization by sampling configurations from portfolios as well as an automated ensemble selection. In the following, we provide more background on the individual components of our framework and introduce two search spaces of particular importance for this work.

Configuration Space
The configuration space provided by Auto-PyTorch Tabular contains a large number of hyperparameters, ranging from preprocessing options (e.g. encoding, imputation) over architectural hyperparameters (e.g. network type, number of layers) to training hyperparameters (e.g. learning rate, weight decay). Specifically, the configuration space Λ of Auto-PyTorch consists of all these hyperparameters and is structured by a conditional hierarchy, such that top-level hyperparameters can activate or deactivate sub-level hyperparameters based on their settings. Auto-PyTorch utilizes the ConfigSpace [30] package for that.
We propose to study two configuration spaces: (i) a small space that contains only few important design decisions, allowing for a thorough, fast and memory-efficient study; (ii) the full space of Auto-PyTorch Tabular which allows to achieve state-of-the-art performance.

Configuration Space 1
For our smaller configuration space, we consider 7 hyperparameters in total, see Table 1. Out of these, 5 are common training hyperparameters when training with momentum SGD. For the architecture, we deploy shaped MLPs [31] with ReLUs and dropout [32]. Shaped MLP nets offer an efficient design space by avoiding layer-wise hyperparameters. We use a funnel shaped variant that only requires a predefined number of layers and a maximum number of units as input. The first layer is initialized with the maximum amount of neurons and each subsequent layer contains neurons, where n i−1 is the number of neurons in the previous layer and n out is the number of classes. An example is depicted in Figure 1. To design a search space capable of finding high performance solutions, we extend the search space to include another training algorithm, Adam [33] and include mixup [34], Shake-Shake [35] and ShakeDrop [36] to increase the choice of training techniques. We add preprocessing options with truncated SVD to deal with sparse datasets and add funnelshaped residual networks (ResNets) [37], allowing to train deeper networks. Like shaped MLPs, shaped ResNets allow for an efficient hyperparametrization by repeating groups with a predefined number of ResNet blocks. The output dimension of each group is determined identically to a shaped MLP with the equivalent number of layers. An example is provided in Figure 1. All parameters of the larger search space are listed in Table 2.

Multi-Fidelity Optimization
Since training DNNs can be expensive, achieving good anytime performance is a challenging task. Auto-PyTorch follows a multi-fidelity optimization approach to tackle this,   using BOHB [15] to find well-performing configurations over multiple budgets. BOHB combines Bayesian optimization (BO) [38] with Hyperpand (HB) [39] and has been shown to outperform BO and HB on many tasks. It also achieves speed ups of up to 55x over Random Search [15]. Similar to HB, BOHB consists of an outer loop and an inner loop. In the inner loop, configurations are evaluated first on the lowest budget (defined by the current outer loop step) and well-performing configurations advance to higher budgets via SuccessiveHalving (SH) [40]. In the outer loop, the budgets are calculated from a given minimum budget b min , a maximum budget b max and a scaling factor η such that for two successive budgets bi+1 bi = η.

Baselines
Unlike HB, BOHB fits a kernel density estimator (KDE) as a probabilistic model to the observed performance data and uses this to identify promising areas in the configuration space by trading off exploration and exploitation. Additionally, a fraction of configurations is sampled at random to ensure convergence. Before fitting a KDE on budget b, BOHB samples randomly until it has evaluated as many data points on b as there are hyperparameters in Λ to ensure that the BO model is well-initialized. For sampling a configuration from the KDE, BOHB uses the KDE on the highest available budget. Since the configurations on smaller budgets are evaluated first (according to SH), the KDEs on the smaller budgets are available earlier and can guide the search until better informed models on higher budgets are available.
An important choice in setting up multi-fidelity optimization is the type of budget used to create proxy tasks. There are many possible choices such as runtime, number of training epochs, dataset subsamples [41] or network architecture-related choices like number of stacks [19]. We choose the number of training epochs as our budgets over the runtime because of its generality and interpretability. Epochs allow a simple comparison between performance of configurations across datasets and transfer of configurations (e.g. learning rate schedules).

Parallel Optimization
Since BOHB samples from its KDE instead of meticulously optimizing the underlying acquisition function, a batch of sampled configurations has a certain degree of diversity. As shown by Falkner et al. [15], this approach leads to efficient scaling for a parallel optimization. We make use of this by deploying a master-worker architecture, such that Auto-PyTorch can efficiently use additional compute resources, see Figure 2.

Model Selection
To ensure model generalization, a good model selection strategy is required in any AutoML framework. To that end, Auto-PyTorch implements a variety of options. It supports the hold-out protocol, allowing for input of user-defined splits or automated splitting, or cross-validation with any of scikit-learn's [42] cross-validation iterators. The model performance is then evaluated via a predefined, or userspecified metric, on the validation set and used to select well-performing configurations. Auto-PyTorch also offers using early stopping via the performance on the validation set to further improve generalization.

Ensembles
The iterative nature of Auto-PyTorch implies that many models are trained and evaluated. This naturally allows for further boosting the predictive performance by ensembling the evaluated models. Auto-PyTorch uses an approach for ensembling inspired by auto-sklearn [43], implementing an automated post-hoc ensemble selection method [44] which efficiently operates on model predictions stored during optimization. Starting from an empty set, the ensemble selection iteratively adds the model that gives the largest performance improvement until reaching a predefined ensemble size. Allowing multiple additions of the same model results in a weighted ensemble. Since it has been shown that regularization improves ensemble performance [45], and following auto-sklearn [43] we only consider the k best models in our ensemble selection (k = 30 in our experiments).
An advantage of our post-hoc ensembling is that we can also include other models besides DNNs easily. As we show in our experiments, this combination of different model classes is one of the key factors to achieve state-of-the-art performance on tabular data.

Portfolios
BOHB is geared towards good anytime performance on large search spaces. However, it starts from scratch for each new task. Therefore, we warmstart the optimization to improve the early performance even further. auto-sklearn tried to use task meta-features to determine promising configurations for warmstarting [5]. In this work, we follow the approach of the newer PoSH-Auto-Sklearn to utilize portfolios in a meta-feature free approach. Auto-PyTorch simply starts BOHB's first iteration with a set of complementary configurations that cover a set of meta-training datasets well; afterwards it transitions to BOHB's conventional sampling.
To construct the portfolio P, offline, we performed a BOHB run on each of many meta training datasets D meta , giving rise to a set of portfolio candidates C. The incumbent configurations from the individual runs are then evaluated on all D ∈ D meta , resulting in a performance meta-matrix, such that the portfolio can simply be constructed by table look-ups. From the candidates λ ∈ C, configurations λ are iteratively and greedily added to the portfolio P in order to minimize its mean relative regret R over all meta datasets D meta . Specifically, motivated by Hydra [46], [47], we successively add λ * i to the previous portfolio P i−1 in iteration i: where P 0 is the empty portfolio and the relative regret R is calculated w.r.t. the best observed performance over the portfolio candidates. Therefore, in the first iteration the configuration that is best on average across all datasets is added to the portfolio. In all subsequent iterations, configurations are added that tend to be more specialized to subsets of D meta for which further improvements are possible.
Configurations are added in this manner until a predefined portfolio size is reached. Limiting the size of the portfolio in this manner balances between warmstarting with promising configurations and the overhead induced by first running the portfolio.
This approach assumes (as all meta-learning approaches) that we have access to a reasonable set of meta-training datasets that are representative of meta-test datasets. We believe that this is particularly possible for tabular datasets because of platforms such as OpenML [48], but sizeable dataset collections also already exist for several other data modalities, such as images and speech (see, e.g. https://www.tensorflow.org/datasets/catalog).

LCBENCH: COMPREHENSIVE AUTODL STUDY FOR MULTI-FIDELITY OPTIMIZATION
To gain insights on how to design multi-fidelity optimization for AutoDL, we first performed an initial analysis on the smaller configuration space (see Section 3.1.1). Starting on a small configuration space allows us to sample configurations more densely and with many repeats to obtain information about all regions of the space. We study (i) the performance distribution of configurations across datasets to show that single configurations can perform well across datasets, (ii) the correlation between budgets for adaptive and non-adaptive learning rate schedules to justify the use of multi-fidelity optimization and (iii) the importance of architectural design decisions and hyperparameters across budgets. Based on our findings we suggest a number of design choices for the full Auto-PyTorch Tabular design space. In particular, we argue for the benefit of portfolios and justify the use of multi-fidelity optimization in our setting.

Experimental Setup
We collected data by randomly sampling 2 000 configurations and evaluating each of them across 35 datasets and three budgets. Each evaluation is performed with three different seeds on Intel Xeon Gold 6242 CPUs with one core per evaluation, totalling in 1 500 CPU hours.

Datasets
We evaluateed the configurations on 35 out of 39 datasets from the AutoML Benchmark [49] hosted on OpenML [48]. As we designed LCBench to be a cheap benchmark, we omit the four largest datasets (robert, guillermo, riccardo, dilbert), in order to ensure low runtimes and memory footprint. Nevertheless, the datasets we chose are very diverse in the number of features (5 − 1 637), data points (690 − 581 012) and classes (2 − 355) (also see Figure 7) and cover binary and multi-class classification, as well as strongly imbalanced tasks. Whenever possible, we use the test split defined by the OpenML task with a 33 % test split and additionally use fixed 33 % of the training data as validation split. In case there is no such OpenML task with a 33 % split available for a dataset, we create a 33 % test split and fix it across the configurations.

Configuration Space and Training
We searched on the search space 1 introduced in Section 3.1.1 and Table 1, which comprises 7 hyperparameters (3 integer, 4 float), two of which describe the architecture and five of which are training hyperparameters. We used SGD for training and cosine annealing [50] as learning rate scheduler. We fixed all remaining hyperparameters to their default value.

Budgets
We used the number of epochs as our budgets and evaluated each configuration for 12, 25 and 50 epochs. These budgets correspond to the ones BOHB selects with the parameters (b min , b max , η) = (12,50,2). For each of these evaluations, the cosine annealing scheduler was set to anneal to 10 −8 when reaching the respective budget. Since we also log the full learning curve, we hence are able to compare the adaptive learning rate scheduling with a seperate cosine schedule for each budget with the non-adaptive one using a single cosine schedule for 50 epochs, also evalutated at 12 and 25 epochs.

Data
Throughout these evaluations we logged a vast number of metrics, including learning curves on performance metrics on train, validation and test set, configuration hyperparameters and global as well as layer wise gradient statistics over time like gradient mean and standard deviation. Although we only use the performance metrics and configurations in this study, the full data is publicly available and we hope that it will be very helpful for future studies by the community. We dub this benchmark LCBench.

Results
We study the following questions on LCBench in order to inform our design of Auto-PyTorch Tabular.

RQ1
Are there configurations that perform well on several datasets?

RQ2
Is it possible to cover most datasets based on a few complementary configurations?

RQ3
Is there a strong correlation between the budgets if we use number of epochs as budgets?

RQ4
Is the importance of hyperparameters consistent across datasets such that models on different budgets can be efficiently used? Figure 3 illustrates the performance of all 2000 configurations across all datasets. We show the mean accuracy on the validation set, as well as the mean relative regret with respect to the best observed validation accuracy, averaging over the 3 runs. As can be seen, the performance distribution varies strongly across different tasks. On some, most configurations perform well (e.g. APSFailure) such that even a random sampling quickly yields good solutions. On the other hand, there are tasks where good configurations are sparse and random search is very inefficient (e.g. dionis). While the intermittent stripe pattern indicates that an increase in mean performance does not strictly indicate better performance on each individual task, there is a clear trend to this end. Always selecting the best configuration for each dataset would improve the average absolute error by 3% compared to selecting the configuration that is best on average across for all datasets. The standard deviation across the datasets is at 15.3%, indicating that some datasets benefit much more from selecting the best configurations than others. Interestingly, selecting the best of the 2 000 evaluated configurations for each dataset shows that some of them are best for several datasets. Overall the final set of best configurations contains 22 unique configurations from 35 datasets with up to 7 occurrences of a single configuration. This provides evidence that transferring configurations to other datasets is very promising if the configuration is well selected. This is an argument for the use of well designed portfolios, as proposed in Section 3.6.

RQ2: Performance of Portfolios with Different Sizes
The results of RQ1 already indicate that portfolios should be a simple but promising approach for achieving good performance with little effort. In Figure 4, we show how the coverage of the portfolio (in terms of regret compared to the full portfolio of 2000 configurations) depends on the size of the portfolio. Whereas a single configuration does not perform well on all datasets, a portfolio of size 10 has an average accuracy regret of less than 0.1% and thus covers all datasets to a sufficient degree.

RQ3: Correlation Between Budgets
To enable multi-fidelity optimizaion, there has to be mutual information between different budgets. We thus study the correlation between budgets utilizing the Spearman rank correlation coefficient. More precisely, we choose a dataset D ∈ D, and compute τ for a budget pair (b, b ) over all configurations λ ∈ Λ to obtain τ (D, b, b ). We perform this for all datasets and budget pairs. Figure 5 shows the corresponding results. Generally, the correlations between budgets are rather high as required for multi-fidelity optimization. As expected, the adjacent budget pairs (12,25) and (25, 50) exhibit a larger correlation than the more distant budget pair (12,50). Table 3 lists the mean correlation across all tasks. Additionally, we consider learning rate schedules with and without adaption to the budget at hand. For adaptive scheduling, we consider cosine annealing, where the epoch at which the minimum is reached is set to the current budget. We refer to the case where the minimum is always set to be reached at the maximum budget as non-adaptive scheduling. Although, Ying et al. [11] used an adaptive scheduling for their NAS-Bench 101, these adaptive schedules exhibit a slightly worse correlation compared to nonadaptive strategies on the larger budgets.

RQ4: Hyperparameter Importance
The analysis of hyperparameter importance offers methods to analyse which hyperparameters are crucial to set properly to achieve optimal performance. This is of particular interest (i) when jointly searching for the architecture and hyperparameters and (ii) when using multi-fidelity optimization to prevent the importance changing substantially across budgets.
We chose two approaches: a global analysis based on fANOVA [27] and Local Hyperparameter Importance (LPI) [28]. Both approaches quantify importance as the variation caused by changing a single hyperparameter; fANOVA marginalizes over all other parameters in the configuration space, whereas LPI fixes all other values to a given (incumbent) configuration. Both utilize a random forest as an empirical performance model [27] fitted on the configurations and their observed performances. Generally, fANOVA is a global method, as it reasons about the entire configuration space by marginalizing out other effects. Results of an fANOVA analysis can be seen in Figure 6. We see that the design of the architecture and the hyperparameters significantly influences the performance of a configuration. Surprisingly, the number of layers (num layers) is the most important hyperparamter, even more important than learning rate or weight decay. However, the maximum number of neurons (max units) is less important, which can be explained by the fact that more layers also leads to overall more neurons by using our shaped networks; however more neurons (max. units) do not imply more layers.
As we expected, the importance of the learning rate slightly increases if we train for longer (i.e. larger budgets).    The weight decay and batch size have a small trend to decrease in importance instead. All other importance scores are fairly constant across budgets. Overall, we can conclude that the hyperparameter importance according to fANOVA is quite stable across budgets. LPI is inspired by the human approach in searching for performance improvements. It defines the local importance of a parameter p as the variance caused by p at an incumbent configuration normalized by the sum of the variances over all individual hyperparameters at that configuration. However, in considering only the neighbourhood of one configuration, LPI marks an extremely local method along one axis.
In contrast to fANOVA, the more local LPI assigns the largest overall importance to the learning rate, see Figure 6. This means that for a fixed incument network changing the learning rate will lead to more catastrophic performance losses, compared to changing e.g., the number of layers. While the number of layers and choice of weight decay are still ranked highly, the importance is more distributed overall. These importance scores again are quite stable across budgets.
Since the number of layers is of major importance, the addition of ResNets to the configuration space of Auto-PyTorch might yield better solutions, as they allow to train much deeper DNNs more easily.

RESULTS OF AUTO-PYTORCH
We now move to evaluate the full Auto-PyTorch system, which combines the full configuration space (Section 3.1), multi-fidelity optimization (Section 3.2), ensembles (Section 3.5) and portfolio generation (Section 3.6). To study the improvement effects of each of these, we study them isolated in a thorough ablation study.

Construction of the Portfolio
To collect candidate configurations for a portfolio, we ran Auto-PyTorch on 100 meta datasets D meta from OpenML [48]. These were chosen by sampling datasets from clusters of similar metafeatres (e.g. number of instances, classes) using the OpenML-Python API [51]. Only datasets with at least two attributes and between 500 and 1 000 000 data points were sampled. Sparse and synthetic datasets, as well as datasets containing time or string type attributes were not considered. Finally, we checked for overlaps between D meta and D test . We used the same budgets as before (12, 25 and 50 epochs) and searched in 24 hours for at most 300 BOHB iterations. To prevent long runtimes on large datasets with poor performances, we also enabled early stopping. We ran on the same hardware as described in Section 4.1 with 3 workers and 2 cores per worker, totalling in about 900 CPU hours. The configuration space is the large space described in Section 3.1.2.
As described in Section 3.6, we obtained a performance matrix with 100 × 100 entries by evaluating the incumbent configurations of each individual run on all other datasets from which we create a greedy portfolio of size 16. To have another portfolio baseline, we additionally include all 100 incumbent configurations in a portfolio, dubbed simple portfolio. Fig. 7. Distribution of meta-train datasets ("meta") and meta-test datasets. Meta-train datasets are further categorized by datasets used in Section 4 ("LCBench") and additional datasets ("new")

Evaluation Setup of Auto-PyTorch
To evaluate the benefit of our framework components, we ran Auto-PyTorch on test datasets in an ablating fashion. We use the same number of cores, workers and epoch budgets as for the portfolio construction, but run up to 72 hours or 300 BOHB iterations. When running plain BO, we use the TPE-like BO implementation based on KDEs in BOHB, as described in Section 3.2. We choose 8 test datasets from OpenML roughly covering the feature and class distribution of the meta datasets, see Figure 7. For comparison, we evaluate a multitude of baselines on the test datasets. Following Erickson et al. [52], we choose LightGBM [53], CatBoost [54], Random Forests, Extremely Randomized Trees and k-nearest-neighbours, and use the same hyperparameter setting. For the latter three, we use the scikit-learn [42] implementations. For training the baselines, 6 CPU cores were allocated to match the resources of an Auto-PyTorch run. We later also consider ensembles built from Auto-PyTorch models and these baselines. Finally, we compare to other common AutoML frameworks. We report the error obtained from the accuracy on the test set for all experiments.

Studying the Large Configuration Space
We now extend the analysis previously presented on the smaller configuration space to the configuration space of Auto-PyTorch to check which of the previous hypothesis also hold here. We do this study on the meta-train datasets, on which we collected a lot of meta-data for the portfolio construction, leaving the test datasets for a final evaluation. Figure 8 shows the Spearman rank correlation between budgets for all meta training datasets. Whereas the correlation was consistently quite high on the small configuration space, the correlation is quite low on some datasets when using the larger configuration space. This indicates that there are specific hyperparameters in our larger configuration space whose settings are more sensitive to budgets and thus lead to smaller correlations. In future work, we will investigate how to automatically and efficiently extract which these are. Although the correlations are weaker compared to our study on LCBench, we will show in the up-coming experiments that the correlations suffice for Auto-PyTorch to perform well.
We also performed an fANOVA analysis and show the 10 most important hyperparameters in Figure 9. Overall, the importance is more evenly distributed than in the previous results. Both architecture design and hyperparameters are both very important to be optimized, which indicates that both should be jointly optimized. Consistent with our previous results, the hyperparameter importance is quite stable across budgets, indicating that this is not the reason for the smaller correlations observed on some datasets.

Ablation Results
To understand the contribution of the individual components of Auto-PyTorch, we study its performance on the meta-test datasets by successively adding one component at a time. We start from plain Bayesian Optimization (BO) and move to multi-fidelity optimization with BOHB, then further add portfolios and finally ensembles. We show two datasets at a time in the following and refer to Figure 14 in the appendix for all datasets and trajectories.

Multi-fidelity Optimization
On all meta-test datasets, BOHB performed as well or better than plain BO in terms of anytime performance.
In particular on the larger datasets, BOHB outperformed BO. Figure 10 illustrates that BOHB can find a strong network earlier than BO by leveraging the lower budgets. Furthermore, BOHB manages to transfer to higher budgets without loosing performance compared to BO even in the convergence limit.

Portfolios
Anytime performance further improves when we add wellconstructed portfolios to Auto-PyTorch. While the simple portfolio (with all incumbents of the 100 meta-train datasets) sometimes performs worse than plain BOHB because of the induced overhead, the greedily constructed portfolio reliably improves performance, see Figure 11. In particular, it matches the first few steps of BOHB and then shows a strong improvement once the first configurations are evaluated on the largest budget. This is expected, since the portfolio is constructed from configurations that perform well on the highest budget.  BOHB + simple PF BOHB + greedy PF

Ensembles
Finally, we build ensembles from the evaluated configurations and compare to common baselines for tabular data. Figure 12 shows that building ensembles from different DNNs and fidelities improves the performance in the long run, sometimes substantially. Furthermore, pure DL can outperform traditional baselines, but not all the time.
To get the best out of both worlds, we therefore propose to build diverse ensembles based on the trained DNNs and in addition based on the baselines, including Random Forest, Extra Trees, LightGBM, Catboost and KNN. Implementation-wise, we first train the baselines and start with the portfolio and BOHB afterwards. By doing this, we can show that the robustness of Auto-PyTorch (i.e., BOHB + greedy portfolio + diverse ensembles) substantially improved s.t. it outperforms the baselines on all but one metatest datasets (apsfailure), see Figure 14 in the appendix.

Parallelization
One of the advantages of Auto-PyTorch and its work horse BOHB is that it nicely parallelizes to using more available compute resources. In Figure 13, we show that parallel Auto-PyTorch makes efficient use of three parallel workers compared to its sequential version and can achieve speedups of 3 and more in particular on larger datasets.

Comparison against Common Baselines and Other AutoML Systems
We compare Auto-PyTorch to several state-of-the-art Au-toML frameworks, i.e. Auto-Keras [6], Auto-Sklearn [43], AutoGluon [52] and hyperopt-sklearn [55]. Whenever possible we chose a runtime of 1h with the same compute TABLE 4 Accuracy and standard deviation across 5 runs of different AutoML frameworks after 1h. "-" indicates that the system crashed and has not returned predictions.  Table 4. We note that some frameworks ran into memory errors (AutoKeras, AutoGluon, auto-sklearn) and some could not handle un-imputed data (hyperopt-sklearn). After 1h, Auto-PyTorch performs best on all but one dataset, where it performs nearly as good as the best system.

DISCUSSION & CONCLUSION
In this paper, we introduced Auto-PyTorch Tabular, a robust approach for jointly optimizing architectures and hyperparameters of deep neural networks. To make educated design decisions, we first studied the challenges of the task on a newly proposed learning curve benchmark, dubbed LCBench. Although LCBench only covers a small configuration space, most of the insights gained on it actually transferred to the more complex configuration space of Auto-PyTorch. In the future, we will extend LCBench to larger datasets and more fidelities.
Our main insights can be summarized as: (i) a well and efficiently designed configuration space for joint architecture search and hyperparameter optimization allows to find very well performing models for a wide range of datasets, (ii) multi-fidelity optimization combined with Bayesian optimization can efficiently search in such spaces, (iii) complementary portfolios found on meta-train datasets improve the performance substantially, in particular in the early phase of the optimization, (iv) by constructing ensembles out of the trained DNNs and strong baselines for tabular data, Auto-PyTorch Tabular achieves state-of-theart performance and outperforms several state-of-the-art competitors.
Although Auto-PyTorch already performs quite well, we believe that there are several ways to push further in the future. First of all, the ensembles overall improve the performance, but it is known that they can also lead to overfitting. By using better generalization performance estimates [56], [57], we plan to address this in the future. Furthermore, we are currently building ensembles from Auto-PyTorch NNs and strong baselines on their default hyperparameter settings. A combination of Auto-PyTorch with Auto-Sklearn [43], [58] could therefore lead to an even stronger performance. Furthermore, we studied the importance of hyperparameters, but have not yet made use of the results to improve the performance of Auto-PyTorch further. We envision that this information can be integrated to improve anytime performance. Last but not least, our approach of Auto-PyTorch Tabular is not specific to any data modality, but we focused on purpose on tabular data in this paper. As a next step, we are planning to design configuration spaces for vision tasks and to study the performance of Auto-PyTorch on these. We are currently in the process of packaging the contributions of this paper cleanly to update the public release of Auto-PyTorch at https://github.com/automl/Auto-PyTorch. Figure 14 shows the results of the ablations study on all meta-test datasets. For the baselines we used scikit-sklearn version 0.23.0 (Random Forest, Extra Trees, KNN), Light-GBM version 2.3.1 and Catboost version 0.23.1. Figure 15 shows the results of Auto-PyTorch and other common AutoML frameworks as well as our baseline models on all meta-test datasets. For all experiments, the same ressources were allocated (i.e. 6 Intel Xeon Gold 6242 CPU cores, 6 GB RAM per core). For Auto-PyTorch the logging post-hoc ensemble selection allows the construction of the full trajectory. For other frameworks, the trajectories were created by running for 1, 3, 5, 10, 30, 50, ..., 10 000 trials and using the timestamp of the trial finish as timestamp for the point on the trajectory.

APPENDIX B COMPARISON AGAINST COMMON BASELINES AND AUTOML FRAMWORKS
We used Auto-Keras 1.0.2 and the default settings. For AutoGluon, we used version 0.0.9 in the "optimize for deployment" setting with the same validation split as Auto-PyTorch uses. For hyperopt-sklearn, we allowed any classifier and preprocessing to be used and used the tpe alogrithm. Finally, we used auto-sklearn 0.7.0 with an ensemble size of 50, initializing with the 50 best, model selection via 33 % holdout, 3 workers. All other settings were set to their default value.