{'fraction_non_alphanumeric': 0.027938931297709923, 'fraction_numerical': 0.010101781170483461, 'mean_word_length': 5.00099251794167, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AMLB: an AutoML Benchmark

Comparing different AutoML frameworks is notoriously challenging and often done incorrectly. We introduce an open and extensible benchmark that follows best practices and avoids common mistakes when comparing AutoML frameworks. We conduct a thorough comparison of 9 well-known AutoML frameworks across 71 classification and 33 regression tasks. The differences between the AutoML frameworks are explored with a multi-faceted analysis, evaluating model accuracy, its trade-offs with inference time, and framework failures. We also use Bradley-Terry trees to discover subsets of tasks where the relative AutoML framework rankings differ. The benchmark comes with an open-source tool that integrates with many AutoML frameworks and automates the empirical evaluation process end-to-end: from framework installation and resource allocation to in-depth evaluation. The benchmark uses public data sets, can be easily extended with other AutoML frameworks and tasks, and has a website with up-to-date results.


Introduction
To create useful machine learning (ML) models from data, data scientists must prepare the data (for example, by encoding categorical features and processing text data), select an ML algorithm, and tune its hyperparameters. This requires extensive expertise, such as knowing which hyperparameters to tune and how (Probst et al., 2019;Weerts et al., 2020). Even with this knowledge, it is a time-consuming task, since the best choices are unique to each data set and can be interdependent on each other (Van Rijn and Hutter, 2018).
The field of automated machine learning (AutoML) is focused on automating the design and optimization of ML pipelines in a data-driven way . Neural Architecture Search (NAS) is an important part of AutoML that automates the design decisions of deep neural networks. AutoML aims to free up valuable time for experts to perform other tasks and allow novice users to train well-performing ML models.
Many different AutoML approaches have been proposed, including sequential modelbased optimization (Hutter et al., 2011;Snoek et al., 2012), hierarchical task planning (Erol et al., 1994), and genetic programming (Koza and Koza, 1992). Novel systems are being developed in both academia and industry, and a recent survey by Van der Blom et al. (2021) showed that 69% of 307 practitioners (at least partially) adopt automated model selection and hyperparameter configuration.

The Need for Standardized Benchmarks
With considerable effort being spent on developing and improving AutoML frameworks as well as increased usage by practitioners, there is a need for systematic and in-depth comparisons of the the various approaches to track progress in the field. However, the comparison of AutoML frameworks is prone to several types of error. First, selection bias may be introduced, even accidentally, when authors decide which data sets to use in their evaluation. For example, too few data sets may be selected to accurately evaluate the framework's strengths and weaknesses, and the chosen data sets may no longer be challenging for current AutoML frameworks. Without a standard suite of data sets to use for evaluation, the selection of data sets is often not reasonably justified and motivated. Moreover, issues may arise from errors in the installation, configuration, or use of 'competitor' frameworks. Typical examples are misunderstanding memory management and/or using insufficient compute resources (Balaji and Allen, 2018), or failing to use comparable resource budgets (Ferreira et al., 2021).
Several suites of benchmark data sets have been used in the aforementioned papers, but none have become a standard in the AutoML community. The original selection of data sets by Thornton et al. (2013) was used in several earlier papers (Feurer et al., 2015a;Mohr et al., 2018), but fails to highlight differences between current AutoML frameworks. Most published AutoML papers use a self-selected suit of data sets on which their methods are evaluated. For example, Drori et al. (2018), Rakotoarison et al. (2019) and Gil et al. (2018) all published at the same time but feature different suites on which they evaluate their contributions. This inconsistency makes it impossible to directly compare results across papers and track progress over time. It also potentially allows for presenting cherry-picked results.

Our Contributions
We introduce a novel AutoML benchmark following best practices to avoid these common pitfalls while stimulating progress towards more standardized benchmarking. 1 To ensure reproducibility 2 , we provide an open-source benchmarking tool 3 that allows easy integration with AutoML frameworks, and performs end-to-end evaluations thereof on carefully curated sets of open data sets. Our focus is on tabular data. Unstructured data are out of scope for this paper, since they are best tackled with NAS, and benchmarking NAS frameworks imposes different practical constraints, as discussed in Section 2.2.
Our benchmarking tool can be used to perform evaluations of AutoML frameworks in a fully automated way. The AutoML frameworks are integrated with benchmarking tool in direct agreement and jointly with the framework's developers to ensure correctness. We carry out a large-scale evaluation of 9 well-known AutoML frameworks across 71 classification and 33 regression tasks. To better understand how these systems perform across many tasks, we also introduce techniques for detailed comparison of AutoML frameworks, including final model accuracy, inference time trade-offs, and failure analysis. Finally, we provide an interactive visualization tool that may be used for further exploration of all our results or to reproduce the analyses performed in this paper.
In the remainder of this paper, we first discuss related work in Section 2 and cover several key open-source AutoML frameworks in Section 3. Next, we provide an overview of our proposed benchmarking tool in Section 4 and motivate our benchmark design choices in Section 5. The results obtained by running this benchmark are analyzed in Section 6. Finally, Section 7 summarizes our main conclusions and sketches directions for future work.

Related Literature
In this section, we motivate why we need benchmarks specifically designed for AutoML, review other work evaluating AutoML frameworks, and finally discuss the relevant ML benchmarking literature. Several benchmark suites have been developed in ML (Van Gestel et al., 2004;Wu et al., 2018;Bischl et al., 2021). The data sets in these suits often do not include problematic data characteristics found in real world tasks (for example, many missing values), as many ML algorithms are not natively able to handle them. By contrast, in order to be applicable to a wide range of data, AutoML frameworks should be designed to handle these problematic data sets. As such, there is opportunity to allow more such problematic data sets in ML benchmarks and study how well AutoML frameworks handle these issues. Moreover, runtime budgets are crucial in an AutoML benchmark, as most AutoML frameworks are designed to run until a given time budget is exhausted. However, these runtime budgets are often not specified beforehand in traditional ML benchmarks, since algorithms must usually run to completion (one exception are performance studies, such as Kotthaus et al., 2015). Consequently, new benchmarks that have been designed specifically for AutoML frameworks are needed.

Evaluation of Automated Machine Learning
To establish new best practices for AutoML benchmarking, it is beneficial to study the shortcomings encountered in prior benchmarks as well as lessons learned. Balaji and Allen (2018) conducted one of the first benchmark studies on AutoML frameworks. They evalu-2. Using the ACM definition: an independent group can obtain the same result using the author's own artifacts (https://www.acm.org/publications/policies/artifact-review-and-badging-current). 3. Code, results, and documentation at: https://openml.github.io/automlbenchmark/ ated four open-source frameworks on both classification and regression tasks sourced from OpenML (Vanschoren et al., 2014), optimized for weighted F1 score and mean squared error, respectively. Unfortunately, they encountered technical issues with most AutoML frameworks that led to a questionable experimental evaluation. For example, H2O AUTOML (LeDell and Poirier, 2020) was configured to optimize to a different metric (log loss as opposed to weighted F1 score) and ran with a different setup (unlike the others, H2O AUTOML was not containerized), and AUTO ML (Parry, 2018) had its hyperparameter optimization (HPO) disabled, making for incomparable results. This highlights the need for careful configuration of all AutoML frameworks involved.
A study on nearly 300 data sets across six different frameworks was conducted by Truong et al. (2019). Each experiment consisted of a single 80/20 holdout split on a 15-minute training time budget, which was chosen so that most tools returned a result on at least 70% of the data sets. We postulate it is reasonable to assume that the data sets for which no result is returned by a framework are most often those data sets for which optimization is hard. For example, a large data set might cause one framework to conduct only few evaluations while it completely halts another. Unfortunately, this makes the resulting comparisons uninterpretable, as a framework could seemingly demonstrate better performance simply because it failed to return models on data sets for which optimization was difficult. Hence, it is key that failures must be avoided as much as possible, and any remaining failures should be analysed and taken into account in subsequent analysis.
On the positive side, Truong et al. (2019) present their results across different subsets of the benchmark-for example, few versus many categorical features-which helps to highlight differences between different frameworks. The authors also conduct small-scale experiments to analyze performance over time by running the tools on multiple time budgets on a subset of data sets as well as the 'robustness', which denotes the variance in final performance given the same input data. Unfortunately, both experiments were conducted on only one data set per sub-category, which does not lend to generalizing the results. Still, studying the impact of data characteristics and budget sizes should ideally be part of AutoML benchmark design. Zöller and Huber (2021) present a survey on AutoML and combined algorithm selection and HPO frameworks (CASH, Thornton et al., 2013). Six CASH and five AutoML frameworks are compared across 137 classification tasks, with the former limited to 325 iterations and the latter constrained to a one hour time budget. Among the CASH frameworks, hyperopt  performed best, although absolute differences were small between all optimizers. The AutoML frameworks are compared as they are, which might reflect common use. However, by not controlling their settings, it becomes impossible to draw conclusions about the effectiveness of individual parts of AutoML systems. A number of errors of AutoML frameworks are observed, including memory constraint violations, segmentation faults, and Java server crashes. The authors also find that most frameworks construct rather modest pipelines (with few preprocessing operators). As such, it is recommended that AutoML frameworks are controlled carefully, use a wide operator search space, and are evaluated on datasets that require non-trivial preprocessing.
Kaggle 4 , a platform for data science competitions, is sometimes used to compare Au-toML frameworks to human data scientists (Zöller and Huber, 2021;Erickson et al., 2020). Zöller and Huber, 2021 found that the rankings of AutoML frameworks on benchmark datasets are very different from the rankings on competitions. Furthermore, humans still find better solutions than the examined AutoML frameworks. However, it is hard to interpret such results. Submissions on Kaggle leaderboards range from serious attempts to random test runs, and a significant portion of them do not outperform a simple baseline. Finally, most Kaggle results are several years old and possibly no longer represent state-ofthe-art human-made models because of advances in available hardware or algorithms.
A recent AutoML benchmark for multi-label classification Wever et al. (2021) proposed a general tool with a configurable search space and optimizer, which allows for the inclusion of new methods and ablation studies. Unfortunately, this approach requires that existing AutoML frameworks be re-implemented within this tool, which is difficult in such a rapidly developing field. As such, we need a simpler way to include existing and new AutoML frameworks into benchmarking tools while still allowing control over their configuration.
In addition to AutoML benchmarks, a series of competitions for tabular AutoML was hosted (Guyon et al., 2019). The first two competitions focused on tabular AutoML, where data is assumed to be independent and identically distributed. In these competitions, participants submitted code that automatically builds a model on given data and produced predictions for a test set. During the development phase, competitors could make use of a public leaderboard and several validation data sets. After the development phase, the latest submissions of each participant would be evaluated on a set of new data sets to determine the final ranking. These data sets consisted of a mix of both new data and data taken from public repositories, although they were reformatted to conceal their identity. In their analysis Guyon et al. (2019) reveal, most methods failed to return results on at least some data sets due to practical issues (such as running out of memory).

Other (Benchmark) Literature
Instead of benchmarking whole (Auto-) ML frameworks, it is often useful to focus on various sub-parts and optimize them step-by-step. One part of such a system is HPO, and its available algorithms are as numerous and diverse as the learning algorithms themselves. Consequently, there exist various benchmarking suites to foster research by comparing those black box optimization algorithms.
Like non-black-box optimization, black box optimizers are typically evaluated on synthetic test functions or on real-world tasks. A recent benchmarking suite for continuous optimization is COCO (Hansen et al., 2021), which includes a collection of various synthetic black-box benchmark functions. kurobako (Ohta and Yamazaki, 2022) provides various general black-box optimizers and benchmark problems, while LassoBench (Šehić et al., 2021) is suitable for benchmarking high-dimensional optimization problems. Especially for Bayesian Optimization, Bayesmark (Turner, 2022) combines several benchmarks on real-world tasks.
One of the first benchmarks for empirically evaluating HPO algorithms was HPOlib (Eggensperger et al., 2013), which allows accessing real-world HPO tasks, tabular and surrogate benchmarks, and synthetic test functions using a common API. This bench-4. https://www.kaggle.com/ mark was used by Bergstra et al. (2014) in their empirical benchmark studies. HPOBench (Eggensperger et al., 2021) is a similar successor of HPOlib that concentrates on reproducible containerized benchmarks and multi-fidelity optimization problems. Based on OpenML (Vanschoren et al., 2014), Arango et al. (2021) recently introduced HPO-B, a large-scale reproducible benchmark for transfer-HPO methods. In contrast, PROFET  uses a generative meta-model to generate synthetic but realistic benchmark instances.
A widely related benchmark is NAS-Bench-101 (Ying et al., 2019), which is a tabular data set that maps convolutional neural network architectures to their trained and evaluated performance on CIFAR-10. Its goal is to make NAS more accessible, despite the tremendous demand of computational resources. Additionally, there exists a series of NAS-Bench that includes NAS-Bench-1shot1 , NAS-Bench-301 , and others.

AutoML Tools
Automated ML pipeline design was first explored by Escalante et al. (2009), but the first prominent AutoML framework was AUTO-WEKA (Thornton et al., 2013). AUTO-WEKA used Bayesian optimization to select and tune the algorithms in an ML pipeline based on WEKA (Hall et al., 2009). Since then, a plethora of new AutoML frameworks have been developed, either by iteratively improving on old designs or using novel approaches. In this section, we will discuss the tools we considered for our benchmark.
Unfortunately the cost of evaluating all frameworks is prohibitive, so we selected only 9 of them to evaluate in this work. Only open source tools were considered. From those, we made selections to cover a variety of different approaches. We considered both frameworks developed by industry as well as academia and included packages whose authors proactively integrated their AutoML framework.
The most notable omission is AUTO-WEKA, which we decided to exclude based on the performance in our 2019 evaluation and its lack of updates since (Gijsbers et al., 2019). Some frameworks integrated with the benchmark tool are not evaluated in this paper. The authors of AUTOXGBOOST (Thomas et al., 2018) opted out of an evaluation because it is built on deprecated software with no plans for updates. ML-PLAN (Mohr et al., 2018), MLR3AUTOML 5 , and MLNET 6 are excluded because we encountered significant technical problems evaluating the systems. We hope to publish results from these frameworks at a later date. There are still many tools not yet integrated with the AutoML benchmark, including OBOE (Yang et al., 2018), AUTO-KERAS (Jin et al., 2019), and AUTOPYTORCH (Zimmer et al., 2021) among others. Table 1 offers an overview of the AutoML tools evaluated in this paper alongside a simplified description of their AutoML design, which are elaborated below. We refer the interested reader to the original publications and our website, which has links to the tools' source code and documentation.

Integrated Frameworks
5. Source and documentation of MLR3AUTOML at https://github.com/a-hanf/mlr3automl/. 6. Source and documentation of MLNET at https://github.com/dotnet/machinelearning/.  (Erickson et al., 2020), but we will refer to it as AUTOGLUON. In contrast to other AutoML systems discussed here, AUTOGLUON does not perform a pipeline search or hyperparameter tuning. Instead, it has a predetermined set of models that are combined through multi-layer stacking and ensembling.
AUTOGLUON's ensemble consists of three layers. The first layer consists of models from a range of model families trained directly on the data. In the second layer, the same type of models are considered but rather as a stacking learner trained with both the input data and the predictions of the first layer. In the final layer, the predictions of the second-layer models are combined into an ensemble (Caruana et al., 2004), an ensemble method first used in AutoML by AUTO-SKLEARN (Feurer et al., 2015a).
To adhere to time constraints, AUTOGLUON may stop iterative algorithms prematurely or forgo training certain models altogether. Given more time, AUTOGLUON will train additional models using the same algorithms and hyperparameter configurations on different data splits, which further improves the generalization of the stacking layer.

auto-sklearn
Based on the design of AUTO-WEKA, AUTO-SKLEARN (Feurer et al., 2015a) also uses Bayesian optimization but is instead implemented in Python and optimizes pipelines built with SCIKIT-LEARN (Pedregosa et al., 2011). Additionally, it warm-starts optimization through meta-learning, starting pipeline search with the best pipelines for the most similar data sets (Feurer et al., 2015b). After pipeline search has concluded, an ensemble is created from pipelines trained during search using the procedure described by Caruana et al. (2004Caruana et al. ( , 2006. AUTO-SKLEARN has won two AutoML challenges (Guyon et al., 2019), although for both entries, AUTO-SKLEARN was customized for the competition, and not all changes are found in the public releases (Feurer et al., 2018).
Based on experience from the challenges, AUTO-SKLEARN 2 was subsequently developed (Feurer et al., 2020). The most notable changes include reducing the search space to only iterative learning algorithms and excluding most preprocessing, use of successive halving (Jamieson and Talwalkar, 2016), adaptive evaluation strategies, and replacing the data-specific warmstart strategy with a data-agnostic portfolio of pipelines. Because these changes make version 2.0 almost entirely different from 1.0, and 1.0 has been updated since our last evaluation, we evaluate both auto-sklearn versions in this paper. However,  does not yet support regression, and its heavy use of meta-learning made it impossible for us to perform a 'clean' evaluation at this time (see Section 5.3.3).

FLAML
The Fast and Lightweight AutoML Library (FLAML, Wang et al., 2021) optimizes boosting frameworks (XGBOOST, Chen and Guestrin, 2016, CATBOOST, Prokhorenkova et al., 2018, and LIGHTGBM, Ke et al., 2017) and a small selection of SCIKIT-LEARN algorithms through a multifidelity randomized directed search (Wu et al., 2021). This search is based on an expected cost for improvement, which tracks the expected computational cost of improving over the best found model so far for each learner. Only after choosing which learner to tune, hyperparameter optimization proceeds by a randomized directed search, sampling a new configuration from a unit sphere around the previous sample point. After evaluating its validation performance, the next sample point is moved to that direction (if better) or the opposite direction (if worse). FLAML positions itself as a fast AutoML framework that is designed to work for small time budgets (Wang et al., 2021).

GAMA
Designed as a modular AutoML tool for researchers, GAMA's search method and post processing are easily configurable (Gijsbers and Vanschoren, 2021). By default, GAMA uses genetic programming to optimize linear ML pipelines with an arbitrary amount of preprocessing algorithms. Similar to TPOT, GAMA's evolutionary algorithm uses NSGA-II to perform multi-object optimization (Deb et al., 2002), maximizing performance while minimizing the number of components in the pipeline. By contrast, GAMA's evolutionary algorithm is asynchronous and does not work with distinct generations, which allows for higher resource utilization. The final model is constructed through ensemble selection (Caruana et al., 2004(Caruana et al., , 2006, similar to AUTO-SKLEARN.

H2O AutoML
Built on the H2O ML platform (H2O.ai, 2013), H2O AUTOML (LeDell and Poirier, 2020) evaluates a portfolio of algorithm configurations and also performs a random search over the majority of the supervised learning algorithms offered in H2O. To maximize model performance, H2O AUTOML also trains two types of stacked ensemble models at various stages during the run: an ensemble using all available models at time t, and an ensemble with only the best models of each algorithm type at time t. H2O AUTOML aims to cover a large search space quickly and relies on stacking to boost model performance. Additionally, H2O AUTOML uses a predefined strategy for imputation, normalization, and categorical encoding for each algorithm and does not currently optimize over preprocessing pipelines. The H2O AUTOML algorithm is designed to generate models that are very fast at inference time, rather than strictly focusing on maximizing model accuracy, with the goal of balancing these two competing factors to produce practical models suited for production environments.

LightAutoML
LIGHTAUTOML is specifically designed with applications in the financial services industry in mind (Vakhrushev et al., 2021). In this framework, pipelines are designed for quick inference and interpretability. Only linear models and boosting frameworks (CATBOOST and LIGHTGBM) are considered. Expert rules are used to evaluate likely good hyperparameter configurations and design the search space. Tree-structured Parzen Estimators (Bergstra et al., 2011) are used to optimize hyperparameters of the boosting frameworks, while warm-starting and early stopping are used to optimize linear models with grid search. Different models are combined in either a weighted voting ensemble (binary classification and regression) or with two levels of stacking (multi-class classification). In a special "compete" mode for larger time budgets, the AutoML pipeline is run multiple times with different configurations and their resulting models are ensembled with weighted voting, which allows for a more robust model.

MLJar
MLJAR (P lońska and P loński, 2021) starts its search with a set of predetermined models and a limited random search-similar to H2O AUTOML. This is followed by a feature creation and selection step, after which a hill climbing algorithm is used to further tune the best pipelines. After search, the models can be stacked, used in a voting ensemble, or both. Learners from SCIKIT-LEARN are considered, and boosting packages XGBOOST, CATBOOST, and LIGHTGBM. MLJAR features multiple modes for different use cases, including exploratory data analysis or finding a fast model, but also a 'compete' mode, which aims to find the best possible model and is used in our experiments.

TPOT
Tree-based Pipeline Optimization Tool , or TPOT, optimizes pipelines using genetic programming. Using a grammar, ML pipelines can be expressed as trees where different branches represent distinct preprocessing pipelines. These pipelines are then optimized through evolutionary optimization. To reduce overfitting that may arise from the large search space, multi-objective optimization is used to minimize for pipeline complexity while optimizing for performance . It is also possible to reduce the search space by specifying a pipeline template (Le et al., 2018), which dictates the high-level steps in the pipeline (for example, a "Selector-Transformer-Classifier" template will result in pipelines with only those three steps, in that order). Development has focused around genomic studies, providing specific options for dealing with this type of high dimensional data for which prior knowledge may be present (Sohn et al., 2017). While TPOT supports neural networks in its search (Romano et al., 2021), the default search space uses SCIKIT-LEARN components and XGBOOST only.

Baselines
In addition to the integrated frameworks, the benchmark tool allows for running several baselines. The constant predictor always predicts the class prior or mean target value, regardless of the values of the independent variables. The Random Forest baseline builds a forest 10 trees at a time, until one of two criteria is met: we expect to exceed 90% of the memory limit or time limit by building 10 more trees, or 2000 trees have been built.
The Tuned Random Forest baseline improves on the Random Forest baseline by using an optimized max features value. The max features hyperparameter defines how many features are considered when determining the best split and is found to be the most important hyperparameter (Van Rijn and Hutter, 2018) 7 . The value is optimized by evaluating up to 11 unique values for the hyperparameter with 5-fold cross-validation before training a final model with the best found value. The Tuned Random Forest is our strongest baseline and aims to mimic a typical first approach for modeling by a human.
Recently, Mohr and Wever (2021) proposed to introduce a baseline that aims to emulate the optimization that a human expert might perform. The baseline includes several steps, including feature scaling, feature selection, hyperparameter tuning, and model selection. We omit this baseline here because it does not support regression and it was published late into our preparation for the experiments.

Software
We developed an open source benchmark tool that may be used for reproducible AutoML benchmarking. 8 This tool features robust automated experiment execution and supports multiple AutoML frameworks, many of which are evaluated in this paper. The benchmark tool is implemented as a Python application consisting mainly of an amlb module and a framework folder hosting all the officially supported extensions, which have been developed together with AutoML framework developers. The main consideration for the design of the benchmark tool is to produce correct and reproducible evaluations; the AutoML tools are used as intended by their authors with little to no room for user error, and the same evaluation conditions (including framework version, data set, and resampling splits) and controlled computational environments can easily be recreated by anyone. The amlb module provides the following features: • a data loader to retrieve and prepare data from OpenML or local data sets.
• various benchmark runner implementations: a local runner that runs the experiments directly on the machine. This is also the runner to which each runner below delegates the final execution.
container runners (docker and singularity are currently supported), which allow preinstalling the amlb application together with a full setup of one framework 7. The hyperparameter min. samples leaf is more important, but not significantly. Although it may not be immediately obvious, the absolute values used for min. samples leaf also transfer to our data sets as the relative values used in max features. 8. Source code and documentation under MIT license at https://github.com/openml/automlbenchmark. and consistently run all benchmark tasks against the same setup. This implementation also makes it possible to run multiple container instances in parallel.
an aws runner that allows the user to safely run the benchmark on several EC2 instances in parallel. Each EC2 instance can itself use a pre-built docker image, as used for this paper, or can configure the target framework on the fly, which is useful for experiments in a development environment.
• a job executor responsible for running and orchestrating all the tasks. When used with the aws runner, this allows distribution of the benchmark tasks across hundreds of EC2 instances in parallel, with each one being monitored remotely by the host.
• a post-processor responsible for collecting and formatting the predictions returned by the frameworks, handling errors, and computing the scoring metrics before writing the information needed for post-analysis to a file. Figure 13 in Appendix C provides an architecture overview and description of the flow of the benchmarking tool, as used in the experiments for this paper.

Extensible Framework Structure
To ensure that the benchmark tool is easily extensible for new AutoML frameworks, we integrate each tool through a minimal interface. Each of the current tools requires less than 200 lines of code across at most four files (most of which is boilerplate). The integration code handles installation of the AutoML framework as well as its software stack and provides the framework with data and recording predictions. The integration requirements are minimal, as both input data and predictions can be exchanged both in Python objects and common file formats, which makes integration across programming languages possible (currently integrated frameworks are written in C # , Java, Python and R). By keeping the integration requirements minimal, we hope that AutoML framework authors are encouraged to contribute integration scripts for their framework and, at the same time, avoid influencing the methods or software used to design and develop new AutoML frameworks (as opposed to providing a generic starter kit, which may bias the developed AutoML frameworks, Guyon et al., 2019). Frameworks may also be integrated completely locally to allow for private benchmarking. 9

Extensible Benchmarks
Benchmark suites define the data sets and one or more train/test splits, which should be used to evaluate the AutoML frameworks. The benchmark tool can work directly with OpenML tasks and suites (Bischl et al., 2021), allowing for new evaluations without further changes to the tool or its configuration. This is the preferred way to use the benchmark tool for scientific experiments, as it guarantees that the exact evaluation procedure can be reproduced easily by others. However, it is also possible to use data sets stored in local files with manually defined splits, e.g., to benchmark private use cases. 10

Using the Software
To benchmark an AutoML framework, the user must first identify and define: • the framework against which the benchmark is executed, • the benchmark suite listing the tasks to use in the evaluation, and • the constraint that must be imposed on each task. This includes: the maximum training time.
the amount of CPU cores that can be used by the framework; not all frameworks respect this constraint, but when run in aws mode, this constraint translates to specific EC2 instances, therefore limiting the total amount of CPUs available to the framework.
the amount of memory that can be used by the framework; not all frameworks respect this constraint, but when run in aws mode, this constraint translates to specific EC2 instances, therefore limiting the total amount of memory available to the framework.
the amount of disk volume that can be used by the framework (only respected in aws mode).
Those constraints must then be declared explicitly in a constraints.yaml file (also in the resources folder or as an external extension).

Commands
Once the previous parameters have been defined, the user can run a benchmark on the command line using the basic syntax: Additional options may be used to specify the mode, the parallelization, and other details of the experimental setup. For example, the following command may be used to evaluate the random forest baseline on the regression benchmark suite across 100 8-core aws instances in parallel with a time budget of one hour.
$ python runbenchmark . py r a n d o m f o r e s t openml / s /269 1 h8c −m aws −p 100

Benchmark Design
In this section, we discuss both the design of the benchmark suite (that is, the chosen data sets and evaluation procedures, Bischl et al., 2021) and the experimental setup, as well as their limitations.

Benchmark Suites
To facilitate a reproducible experimental evaluation, we make use of OpenML Benchmark suites (Bischl et al., 2021). An OpenML benchmark suite is a collection of OpenML tasks, which each reference a data set, an evaluation procedure (such as k-fold cross-validation) and its splits, the target feature, and the type of task (regression or classification). The benchmark suites are designed to reflect a wide range of realistic use-cases for which the AutoML tools are designed. Resource constraints are not part of the task definition. Instead, we define them separately in a local file so that each task can be evaluated with multiple resource constraints. Both the OpenML benchmark suite (and tasks) and the resource constraints are machine-readable to ensure automated and reproducible experiments.

Data Sets
We created two benchmarking suites, one with 71 classification tasks, and one with 33 regression tasks. The data sets used in these tasks are selected from previous AutoML papers ( Thornton et al., 2013), competitions (Guyon et al., 2019), and ML benchmarks (Bischl et al., 2021) according to the following predefined list of criteria: • Difficulty of the data set must be sufficiently high. If a problem is easily solved by almost any algorithm, it will not be able to differentiate the various AutoML frameworks. This can mean that simple models (such as random forests, decision trees or logistic regression) achieve a generalization error of zero, or that the performance of these models and all evaluated AutoML tools is identical.
• Representative of real-world data science problems to be solved with the tool. In particular, we limit artificial problems. We included a small selection of such problems, either based on their widespread use (kr-vs-kp) or because they pose difficult problems, but we do not want them to constitute a large part of the benchmark. We also limit computer vision problems on raw pixel data because those problems are typically solved with dedicated deep learning solutions. However, since they still make for real-world, interesting, and hard problems, we did not exclude them altogether.
• No free form text features that cannot reasonably be interpreted as a categorical feature. Most AutoML frameworks do not yet support feature engineering on text features and will process them as categorical features. For this reason, we exclude text features, even though we admit their prevalence in many interesting real-world problems. A first investigation and benchmark of multimodal AutoML with text features has been carried out by Shi et al. (2021).
• Diversity in the problem domains. We do not want the benchmark to skew towards any application domain in particular. There are various software quality problems in the OpenML-CC18 benchmark (jm1, kc1, kc2, pc1, pc3, pc4), but adopting them all would lead to a bias in the benchmark to this domain.
• Independent and identically distributed (i.i.d.) data are required for each task.
If the data are of temporal nature or repeated measurements have been conducted, the task is discarded. Both types of data are generally very interesting but are currently not supported for most AutoML systems, and we plan to extend the benchmark in the future in this direction.
• Freely available and hosted on OpenML. Data sets that can only be used on specific platforms, like Kaggle, or are not shared freely for any reasons are not included in the benchmark.
• Miscellaneous reasons to exclude a data set included label-leakage and near-duplicates of other tasks in independent variables (for example, different only in categorical encoding or imputation) or dependent variable (most commonly the binarization of a regression of multi-class task).
To study the differences between AutoML systems, the data sets vary in the number of samples and features by orders of magnitude and vary in the occurrence of numeric features, categorical features, and missing values. Figure 1 shows basic properties of the classification and regression tasks, including the distributions of the number of instances and features, the frequency of missing values and categorical features, and the number of target classes (for classification tasks). Properties of the tasks are shown in Appendix A and can be explored interactively on OpenML. 11 While the selection spans a wide range of data types and problem domains, we recognize that there is room for improvement. Restricting ourselves to open data sets without text features severely limits options, especially for big data sets.
All data sets are available in multiple formats for the AutoML frameworks, either as files (parquet, arff, or csv) or as Python object (pandas dataframe, numpy array), and the used format depends on the AutoML framework. All frameworks have access to meta-data, such as the datatype of the columns, either directly from the chosen data format or as separate input, so that each AutoML framework has the same information available regardless of data format.

Performance Metrics
In our evaluation, we use area under the receiver operating characteristic curve (AUC) for binary classification, log loss for multi-class classification and root mean-squared error (rmse) for regression to evaluate model performance. 12 We chose to use these metrics because they are generally reasonable, commonly used in practice, and supported by most AutoML tools. The latter is especially important because it is imperative that AutoML systems optimize for the same metric on which they are evaluated. However, our tool is not limited to only these three metrics, and a wide range of performance metrics can be specified by the user.

Missing Values In Experimental Results
As will be discussed in more detail in Section 6.4, not all frameworks are equally wellbehaved. There are situations where search time budgets are exceeded or the AutoML frameworks crash outright, which results in missing performance estimates. There are multiple strategies to consider on how to deal with these missing data.
One naive approach may be to ignore missing values and aggregate over the obtained results. However, we observe that failures do not occur at random. Failures correlate with data set properties, such as data set size and class imbalance, which may be correlated with "problem difficulty" and thus performance. Ignoring missing values thus means that AutoML frameworks may fail on harder tasks or folds and consequently obtain higher average performance estimates. Imputing missing values with performance obtained by the same AutoML framework on other folds is subject to the same drawback. Moreover, in case a framework fails to produce predictions on all folds of a task, neither method specifies how to deal with missing values.
Instead, we propose to impute the missing values with an interpretable and reliable baseline. An argument may be made for using the random forest baseline, since this may be a strong fallback that AutoML frameworks could realistically implement. However, we observe that training a random forest (of the size used in the baseline) requires an insignificant amount of time on some data sets. Automatically providing this fallback by means of imputation would provide an unfair advantage to the AutoML frameworks that are not well-behaved. Moreover, many failures would not be remedied by having a random forest to fall back on, since the AutoML frameworks crash irrecoverably, e.g., due to segmentation faults.
Instead, we impute missing values with the constant predictor, or prior. This baseline returns the empirical class distribution for classification and the empirical mean for regression. This is a very penalizing imputation strategy, as the constant predictor is often much worse than results obtained by the AutoML frameworks that produce predictions for the task or fold. However, we feel this penalization for ill-behaved systems is appropriate and fairer towards the well-behaved frameworks and hope that it encourages a standard of robust, well-behaved AutoML frameworks.

Experimental Setup
We execute the experiments on commodity-level hardware with AutoML frameworks generally in their default configurations.

Hardware
For comparable hardware and easy expandability, we opt to conduct the benchmark on standard m5.2xlarge 13 instances available on Amazon Web Services (AWS). These represent current commodity-level hardware with 32 GB memory, 8 vCPUs (Intel Xeon Platinum 8000 series Skylake-SP processor with a sustained all core Turbo CPU clock speed of up to 3.1 GHz). 100 GB of gp3-SSD storage is available for storage, which can be necessary for storing a larger number of evaluated pipelines. The use of AWS also enables others to fully reproduce and extend our results, as funding permits, since the results do not depend on private computing infrastructure. As discussed in Section 4.1, the benchmark is not limited to AWS but can be run on any machine.

Framework Configuration
All AutoML frameworks are instantiated with their default configuration, with the following exceptions: • Runtime for the search. Additionally, there is one hour leeway for data loading, making predictions, and cleanup operations, but this is not communicated to the AutoML frameworks.
• Resource constraints that specify the number of CPU cores and amount of memory available.
• Target metric to use for optimization. This is the same metric that is used for evaluation in the benchmark.
• 'mode' to declare the user intent. For example, obtaining the best possible model versus finding an interpretable (less complex) model. The mode used to evaluate each AutoML framework is chosen by their developers.
• 'output directory', where any artefacts of the AutoML framework may be stored.
The experiment design intentionally prohibits further customization of other AutoML system configuration parameters to reflect how these systems are usually applied in practice as closely as possible (note that the benchmark tool does allow for this type of customization). An overview of the exact framework versions used and their 'mode' configurations is shown in Table 10 in the appendix.

Limitations
Both the design of the benchmark and the setup for the experiments described in this paper have some limitations with regard to the interpretation of their results. Limitations in the design stem from the desire to keep the use of the tools as close as possible to the original vision and usage intended by developers, whereas the limitations in the experiments are caused by resource constraints and may be alleviated by running additional experiments with the benchmark software. In this section, we highlight some important limitations and stress that this paper and the results within do not state which AutoML tool is ultimately the best.

Limitations of the Design
Perhaps the biggest limitation of the design is the inability to attribute the performance of an AutoML tool to any one aspect of its build, as is often done with ablation studies. The evaluated AutoML tools differ among multiple design choices, such as underlying ML library, search space, preprocessing, and search algorithm. Concretely, a performance difference between, e.g., AUTO-SKLEARN and TPOT could be caused by TPOT's built-in stacking, AUTO-SKLEARN's ensembles, the difference in Bayesian optimization versus genetic programming, the difference in how multiprocessing is employed, or a combination of these or any other difference between them. Software that would allow for such conclusions essentially requires each AutoML tool to be reimplemented on a shared set of algorithms for building models, search, and evaluation. We acknowledge that this would be incredibly valuable for the research community. However, it would also no longer resemble the software as used in practice and thus would be different work altogether. Note that it is possible to perform ablation studies with the benchmark tool for a specific AutoML framework, e.g., by comparing different framework configurations as done by Erickson et al. (2020).
Another limitation stems from only recording results produced by the final model. Anytime performance, where information about the performance during optimization is captured as if they were final models, can be very insightful. This method allows for the distinction between a tool that converges quickly from one that does not. This may be especially important for users who are interested to use the systems with a human-in-theloop, such as when designing a search space or data features. Unfortunately, many tools do not support collection of anytime performance and-depending on how they are recordedmight interfere with resources used during search. We hope to be able to record anytime performance in the future, but in this work, we only approximate it by evaluating the tools under two different time constraints (1 and 4 hours).
Finally, the qualitative comparison of the tools is also limited. Certain quality of life features like analysis of the pipeline via interpretable ML methods, reports, usability, or support are not evaluated here but are important to many users. For a qualitative analysis of those characteristics, we refer the reader to one of the many existing overview papers on AutoML (Zöller and Huber, 2021;Truong et al., 2019). We also provide an overview of various AutoML frameworks with links to their documentation on our website 14 .

Limitations of the Experiments
Most tools are highly configurable and allow the user to configure the search algorithm or its hyperparameters, among other aspects that affect the AutoML performance. Some tools even provide different configuration presets for different use cases, such as a performance-14. https://openml.github.io/automlbenchmark/frameworks.html oriented competition mode-which we use-and a mode that produces fast or interpretable models at the cost of some performance. However, comparing the effect on model performance of tuning AutoML hyperparameters or using different presets quickly carries prohibitive costs. For this reason, we must limit our experiments to use only one mode specified by the frameworks' developers. The most performance-oriented setting was selected as the mode for each tool. It is likely that better results may be achieved by carefully meta-tuning the AutoML tool, or that the tool with the best performance in competition mode has relatively poor performance in interpretable mode. While it is cost prohibitive for us to evaluate many different scenarios, it easy to run the benchmark with custom configurations for the various AutoML tools. This allows users to evaluate AutoML systems in a setting that reflects their interest.

Meta-learning
Many AutoML tools make use of meta-learning to better initialize and speed up the search (Yang et al., 2018;Feurer et al., 2015aFeurer et al., , 2020. Since all data in the benchmark is publicly available and many of them are well known in the AutoML community, it is likely that there is a substential overlap between data used by the developers for metalearning and the data used in the benchmark. This is a very intricate problem, as we consider AutoML tools as black boxes. Removing the effect of the data set that is to be evaluated from the meta-learning procedure is not solvable in general. In this paper specifically, both AUTO-SKLEARN and AUTO-SKLEARN 2 use meta-learning. AUTO-SKLEARN's meta-learning uses 208 data sets from OpenML, each associated with well-performing ML pipelines. The search is initialized with a 25-nearest-data set (KND) lookup using 38 meta-features (Reif et al., 2012). AUTO-SKLEARN can exclude data sets by name from the lookup-which we make use of in the benchmark and ensured that there is no overlap to the 39 data sets from Gijsbers et al. (2019). Even so, it cannot be guaranteed that identical data sets with a different name might be used for meta learning out-of-the-box.
AUTO-SKLEARN 2's meta-learning model is more complicated and consists of: a) a static pipeline portfolio for warm-starting the search, which is computed across hundreds of data sets using a greedy forward selection, and b) a meta-model to predict the internal model selection strategy and budget allocation strategy. Single data sets cannot be excluded from these meta-learning procedures, and it is not feasible to retrain the meta-models and pipeline portfolio for each data set in our benchmark. This ultimately means that the result of AUTO-SKLEARN and especially AUTO-SKLEARN 2 must be considered very carefully. More research is required to address these issues and allow for the correct evaluation of AutoML systems that use meta-learning in common benchmarks.

Overfitting the Benchmark
One last issue that plagues any widely-adopted benchmark is the potential of algorithms overfitting on the data sets used in the benchmark. Since freely available, interesting, and usable (refer to Section 5.1.1 for our selection criteria) data sets are scarce, many AutoML developers use these data sets to benchmark and then improve their systems iteratively. While this is not as direct of an issue as with meta-learning, these data sets can in general not be assumed to be truly unseen. The only practical way to avoid this is to collect a novel set of data sets for the benchmark, which would entail a prohibitive effort. Moreover, after publishing such a benchmark, the new data sets are published, which again gives developers the possibility to use them to improve their systems. On the other hand, should the benchmarking data sets be kept private to avoid this issue, the benchmark is no longer entirely reproducible by independent researchers. We hope that the size of our benchmarking suites is large enough and their design general enough that overfitting is less of an issue, but this is difficult to guarantee, and a study as outlined above may be useful to evaluate this phenomenon in the future.

Results
In this section, we provide an overview and analysis of the results obtained. This section is accompanied by an interactive visualization tool 15 , additional information in Appendix B, and all data artifacts generated from these experiments 16 . For a more comprehensive comparison than what we can provide here, we strongly encourage the reader to explore the data with the interactive visualization tool. This tool enables users to select any subset of frameworks, task types, performance measures, or data characteristics iteratively and interactively. The tool is created with RSHINY (Chang et al., 2017) and includes overview plots for different task types as well as detailed visualizations for individual data sets. Moreover, a statistical analysis of the results by critical difference plots and Bradley-Terry trees are implemented.
In addition to the limitations outlined in section 5.3, we also want to stress that these experimental results are obtained by running the frameworks as they were in September of 2021. Some of these frameworks are still under active development, and results from experiments run on later versions will almost certainly differ. We strongly encourage people to run additional experiments that match their use case with up-to-date frameworks, and use the results in this section as a reference on how to analyze the results. Table 10 in Appendix C.2 shows an overview of the versions benchmarked for each framework as well as the most current version.

Performance
To report on the results for many AutoML frameworks across whole benchmarking suites, we propose using critical difference (CD) diagrams (Demšar, 2006). In a CD diagram, the average rank of each framework as well as which ranks are statistically significantly different from each other are shown. To calculate the average rank per task, we first impute any missing values with the constant predictor and then average the performance over all folds. We may then test for the presence of statistically significant differences in the average rank distributions using a non-parametric Friedman test at p < 0.05 (here, p ≈ 0 for every diagram) and use a Nemenyi post-hoc test to find which pairs differ. For each benchmarking suite and time budget, the CD diagrams are shown in Figure 2, which displays the rank of each framework (lower is better) averaged over all results from the given benchmarking suite and budget. AUTO-SKLEARN 2 is excluded in this comparison due to the meta-learning 15. Results can be interactively explored at https://compstat-lmu.shinyapps.io/ AutoML-Benchmark-Analysis/. 16. Experiment artifacts can be found at: https://test.openml.org/amlb/. issues discussed in Section 5.3.3 and because its inclusion would affect the critical difference as well as the rank of other frameworks. Overall, we observe that AUTOGLUON and TPOT respectively achieve the best and worst rank among AutoML frameworks in each setting with respect to model accuracy, although never by a statistically significant margin. In almost all cases, the baselines obtain lower ranks than any AutoML framework, although the tuned random forest is a strong baseline that is often not significantly worse than many of the AutoML frameworks. All AutoML frameworks except AUTOGLUON and TPOT are generally ranked close to each other, with small differences across the various suites and budgets.
To complement the CD diagrams-which obfuscate the relative performance differenceswe show box plots of obtained results (after imputation) across all tasks in Figure 3. Because the performances are not commensurable across tasks, we first scale all results per task between the random forest performance (-1) and the best observed performance (0) (that is, higher scores correspond to better performance). This also makes the scaled value interpretable. Furthermore, the value scales based on the improvement over the baseline that is observed to be achievable. While the boxplots are calculated over performance data on  Even if ranks are similar, the performance distribution might be noticeably different. For example, GAMA, H2O AUTOML and LIGHTAUTOML achieve very similar average ranks on the one hour binary classification tasks. However, we observe from the boxplots that while H2O AUTOML achieves lower median normalized performance in this segment, its worst observed performances are much better than that of GAMA and LIGHTAUTOML. Similarly, while TPOT's average rank is generally close to that of the Tuned Random Forest baseline, TPOT exhibits much higher variance in its prediction quality.

Bradley-Terry Trees
Bradley-Terry (BT) trees (Strobl et al., 2011) can be used to statistically analyse benchmark experiments based on data set characteristics (Eugster et al., 2014). These trees use data set characteristics-such as the number of instances, the number of features, the ratio of missing values, and others-to split paired performance comparisons of the framework to find statistically significant differences in performance. Bradley-Terry models originate in psychology to analyze paired comparison experiments of subjects preferring one stimulus over another. For our benchmark, such a preference ranking can be easily derived by pairwise performance comparisons of all frameworks with regard to the data sets and crossvalidation folds.
The underlying algorithm of model-based recursive partitioning of Bradley-Terry models works as follows: In each split of the BT tree, a BT model is fitted for the paired comparisons based on the underlying data set characteristics. Following Zeileis and Hornik If this test reveals a significant instability in the model parameters, the corresponding tree node splits the data according the the characteristic yielding the highest instability (lowest test p-value). The splitting cut-point is then determined such that it has the highest improvement of the model fit. This procedure is repeated until either no significant instability is left, a set tree depth is reached, or further splits would exceed a set minimum number of observations in the leaves. Numeric values in the tree leafs are worth parameters that can be interpreted as preferences for the different frameworks (Eugster et al., 2014). Since these values are in [0, 1] and sum up to 1 within a leaf, they can be understood as the probability of a framework performing best, given the data characteristics in the corresponding leaf. Figure 4 shows a Bradley-Terry tree for classification tasks for a runtime of one hour. For simplicity, in order to obtain an easily understood tree, only the number of instances/features and the imbalance ratio was chosen as data characteristics. The first split distinguishes between data sets with more that 5832 instances and those equal to or below that cut-point. Following the left child node, the imbalance ratio of 1.034 was chosen to define the two left tree leafs. The left one (Node 3)-small and very balanced classification data sets-indicates that in such situations, GAMA is preferred over all other frameworks. Even though AUTOGLUON is less preferred than GAMA for those kinds of data sets, AUTOGLUON is still preferable to all other frameworks. On small, more imbalanced data sets (Node 4), AUTOGLUON is preferable to all other frameworks, followed by AUTO-SKLEARN 2.
The right half of the tree is again divided into medium and large data sets at a splitting value of 130064 observations. While on the left leaf (Node 6) AUTOGLUON is clearly the preferred framework, the same applies on large classification data sets to FLAML (Node 7), followed by MLJAR and AUTOGLUON.
Simpler Bradley-Terry trees with only the number of instances and features as data set characteristics can be found in the appendix. Note that the findings from the BT trees are essentially the same as those from Section 6.1, as AUTOGLUON is overall the preferred framework in most tree leafs. Moreover, the reader is strongly invited to explore the aforementioned interactive visualization tool, with which deeper BT trees based on several more data set characteristics can be constructed on various task types.

Model Accuracy vs. Inference Time Trade-offs
Model accuracy plays a central role in evaluating performance of machine learning models. However, maximizing accuracy can come at the cost of added model complexity. One practical way to consider the complexity of the model is to measure the inference speed of the resulting model. Some of the integrated frameworks offer a "compete" mode (AUTOGLUON, MLJAR, LIGHTAUTOML and GAMA) that maximizes accuracy, typically at the cost of increased model complexity, similar to competing in a Kaggle competition. This can lead to models being built that are highly accurate but are extremely slow at inference time and are therefore not practical in many real-life use-cases.
Fortunately, some frameworks provide multiple presets that allow the user to balance of the trade-off between accuracy and inference time differently. Results in this section used presets which prioritize accuracy over inference time, and performing additional experiments with other presets is advised when inference time is important 17 . On the other hand, we also recognize that there are applications for which inference time is not important.
In order to evaluate the limitations of the models produced by each framework (in "compete" mode), we also measured "prediction duration," or how long was needed to produce predictions for the test set for each data set in the benchmark. This metric provides important insight into the trade-offs that tool authors make in their algorithm designs. Figure 5 shows aggregated inference times across all models, including total time to score the test set (predict duration) as well as the per-row prediction speed (predict duration divided by the number of rows in each test set). Outliers have been removed from both plots for visibility, as there are a handful of very extreme outliers. Both AUTOGLUON and MLJAR stand out as orders of magnitude slower than the other AutoML tools, on average. 18 LIGHTAUTOML is also slower than the remaining tools. GAMA is approximately as fast as AUTO-SKLEARN and AUTO-SKLEARN 2, which can be explained by the fact that they both build optimized SCIKIT-LEARN pipelines and create an ensemble using the same algorithm (Caruana et al., 2004(Caruana et al., , 2006. H2O AUTOML, FLAML and TPOT stand out as having very fast inference times, although TPOT is much less accurate than the other tools, as seen in more detail below.  In Figure 12, we show the Pareto front for all six scenarios, demonstrating the average normalized model accuracy against corresponding average per-row prediction speeds 17. We plan to do additional experimental evaluations which may include more balanced presets later this year and publish the results on our website. 18. It is important to note that AUTOGLUON's most recent version, which was released after the experiments were performed, now has the option to explicitly constrain inference time and provide more presets that balance inference time and model accuracy. (Appendix B.2 contain plots with median per-row prediction speeds instead). Here, it is more apparent that the frameworks that achieve the highest accuracy do so at the cost of inference time performance. This demonstrates that when contextualizing any type of model accuracy results, it is important to consider any trade-offs that may have been made to achieve the extra performance and how that will affect the framework's usability in practice. In this case, measuring accuracy in isolation does not give the complete picture of the overall utility of a particular framework. It should also be noted that with sufficient computing infrastructure and effort, scoring across rows or chunks of data could be parallelized in a production system, which would reduce the overall prediction time as compared to predicting a single test set on a single machine. However, our goal in this section was to compare the inference time of the highaccuracy models derived from different AutoML frameworks to each each other rather than evaluate different techniques for speeding up the inference of any individual system.

Observed AutoML Failures
While most jobs completed successfully, we observed multiple framework errors during our experiments. In this section, we will discuss where AutoML frameworks fail, although we want to stress that development for these packages is ongoing. For that reason, it is likely that the same frameworks will not experience the same failures in the future (especially after gaining access to all experiment logs). We categorize the errors into the following categories: Memory: The framework crashed due to exceeding available memory or encountering other memory-related errors, such as segmentation faults.
Time: The framework exceeded the time limit past the leniency period.
Data: Errors due to specific data characteristics (such as imbalanced data) occurred.
Implementation: Any errors caused by bugs in the AutoML framework code occurred.
These categories are a bit crude and ultimately subjective, since from a reductive viewpoint, all errors are implementation errors. However, they serve for a quick overview and a more detailed discussion can be found in Appendix D. Figure 7 shows the errors by type on the left and by task on the right. Overall, memory and time constraints are the main cause for errors, with one major exception 19 . We observe that errors are far more common in the classification benchmark suite than the regression suite. This is largely accounted for by the difference in benchmarking suite size (33 and 71 tasks) and the fact that the largest data sets are mostly classification, both in number of instances and features. Unique to classification, we do observe several frameworks failing to produce models or predictions on highly imbalanced data sets. This is also the case for the failures on the two small classification data sets ('yeast' and 'wine-quality-white'), where careless use of internal validation splits no longer contain all classes. Interestingly, AutoML frameworks fail more frequently on a larger time budget. Both memory and 19. MLJAR has 190 'implementation errors' which are caused by only two distinct index errors. time constraint violations happen more frequently, which may potentially be explained by frameworks saving increasingly more models or building increasingly larger pipelines. Only when the framework exceeds the time budget by more than one hour do we record a time error. However, as we can see in Figure 8, not all AutoML frameworks adhere to the runtime constraints equally well, even if they finish within the leniency period. In the figure, the training duration for each individual job (task and fold combination) are aggregated, and timeout errors are shown above each framework, where missing values due to non-time errors are excluded. These plots reveal different design decisions around the specified runtime, with some frameworks never exceeding the limit by more than a few   minutes, while others violate it by a larger margin with some regularity. Interestingly, we see that a number of frameworks consistently tend to stop far before the specified runtime limit.

Conclusion
We presented a novel benchmark for measuring and comparing AutoML frameworks. To ensure reproducibility, fair comparison, and detailed analysis of the results, our open-source benchmarking tool automates the empirical evaluation of any integrated framework on any supported task, including installation of the AutoML framework, provisioning the data for training and inference, resource allocation, and processing of the results. This greatly simplifies evaluating AutoML frameworks while enhancing reproducibility and reducing errors. We worked jointly with the authors of 9 AutoML frameworks to evaluate their systems in a large-scale study on 71 classification and 33 tasks.
When analyzing the predictive performance of these AutoML systems, we find that the average ranks of the AutoML frameworks are generally not statistically significantly different from each other. Still, by using Bradley-Terry trees (Strobl et al., 2011), we find that their relative performance is affected by data characteristics-such as the data set size, dimensionality, and class balance-highlighting specific strengths and weaknesses. Overall, in terms of model performance, AUTOGLUON consistently has the highest average rank in our benchmark. Additionally, in most scenarios, the AutoML frameworks outperform even our strongest baseline.
Because inference time is an important factor in real-world applications, we also reviewed the inference time and accuracy trade-off and found large differences in inference time of the produced models, at times spanning multiple orders of magnitude. The most accurate frameworks achieve higher model accuracy at a large cost to performance in terms of inference speed. Broadly speaking, while the models with higher accuracy also have slower inference time, not all frameworks produce models that are Pareto optimal.
Finally, we analyzed scenarios in which AutoML frameworks fail to produce a model and found that the main cause for failure was data set size. In other words, not all methods scale well. To allow further analysis of our results, we provide an open-source interactive visualization tool, which includes graphical representations and statistical tests.

Limitations
These quantitative results are obtained by using AutoML frameworks in their default configuration, optionally with a 'benchmarking preset', to reflect the out-of-the-box nature of AutoML frameworks. The performance of frameworks under non-default settings can be very relevant when, for example, there is budget available to also optimize AutoML hyperparameters, or when a custom search space is used.
Next, all frameworks differ along multiple design axes, which prohibits attributing performance differences to any specific component of the AutoML framework (such as the search algorithm), without additional analysis.
Moreover, since we provide a purely quantitative comparison, it ignores qualitative aspects of AutoML frameworks that are very relevant in real-world settings, such as the produced model's interpretability or the level of support. Additionally, most of the evaluated AutoML frameworks are under active development and their performance subject to change.
Lastly, the benchmarks were executed on 8-CPU machines, which is very modest by today's standards. Therefore, frameworks with better parallelism may show even greater advantages on higher powered machines (with, for example, 50 or 100 CPUs).

Future Work
The benchmarking suites, or sets of tasks, proposed in this paper are meant as a starting point to be improved upon in collaboration with the AutoML community. We will seek to update these suites based on community discussion so that they continue to reflect modern challenges while also decreasing the risk of AutoML frameworks overfitting to the benchmark. In particular, we are interested in extending the benchmarking suites with problems that feature free-form text data. Real-world data often contain instances of text data with different semantic meaning, such as addresses or URLs, from which meaningful features can be extracted. This kind of feature engineering is typically very important for model performance on such data sets, but the current selection of data sets does not reflect this (in part because not all AutoML frameworks support text features).
We would also like to extend the benchmarking tool to support new problems, such as multi-objective optimization tasks. While model accuracy is often an important metric, 'secondary' metrics-such as a model's inference time or fairness-are often crucial for realworld applications. Multi-objective optimization can be used to convey the importance of these metrics to AutoML frameworks (for example, by using a fairness metric as secondary objective, Schmucker et al., 2021), which can subsequently provide Pareto fronts of models that optimize this trade-off. In particular for fairness related tasks, additional support to convey sensitive attributes and protected groups to the AutoML frameworks must be added. Other interesting problem types include non-i.i.d. data, such as when temporal relationships are present in the data, or semi-supervised learning, where not all instances have an associated ground truth.
Finally, the evaluations in this paper were performed on the equivalent of commoditylevel hardware using only CPU and a limited time budget. In some cases, it is more desirable to devote a large budget to building a single model, perhaps even days of compute, potentially with GPU access. As different behavior (both in robustness and model performance) on one-hour and four-hour budgets are already observed, future work may reveal different behavior when evaluating AutoML frameworks at different scales.

Parting Words
The benchmark tool presented in this work makes producing rigorous reproducible research both easier and faster. We hope that the open and extensible nature of this benchmark motivates researchers to not only use the tool, but also to contribute their own data sets, framework integrations, or feedback to the open source AutoML benchmark. We strongly encourage this participation so that the benchmark may remain useful to the community for a long time to come. This work made use of the resources and expertise offered by the SURF Public Cloud Call which is financed by the Dutch Research Council (NWO). We also made use of research credits provided by the AWS Cloud Credit for Research program.
Pieter Gijsbers and Joaquin Vanschoren would Table 2 and Table 3 contain an overview of data sets used in the regression and classification benchmarking suites, respectively. We hope to continuously update the benchmarking suites with new data sets that represent current challenges.

B.2 Model Accuracy vs. Inference Time Trade-offs
Below are the same plots as presented in the results section, but using the median inference time instead of the mean.

Appendix C. Software
This appendix contains additional details about the developed and used software.
C.1 Architecture Overview Figure 13 shows the architecture and information flow when running the benchmarking tool in the setup used in the experiments for this paper. The dotted lines indicate calls (communication) whereas the solid lines indicate transfer of files. In this paper we distribute our workload to AWS EC2 containers which run experiments within a docker environment. The experiments are initiated from a 'local' machine, shown enclosed in a blue rectangle in the bottom. This local instantiation first reads information from a local configuration file and the command line, after which it uses boto3 to connect to AWS. It uploads local files required for the experiments, such as the configuration files, to an S3 bucket accessible by the EC2 instances, deploys the jobs to EC2 instances, and tracks the instance status through CloudWatch.
The EC2 instance then installs the specified version of the benchmarking tool (here of type m5.2xlarge), retrieves the user configuration from the S3 bucket, the specified docker image from the docker hub, and the dataset (task) from OpenML. The experiment is then run in a docker container (which in turn uses the benchmarking software in 'local' mode) which has the benefits of reproducibility of the software stack and requiring much less time than installing the framework from scratch.
After the experiment has completed, results are uploaded to an S3 bucket and the EC2 instance terminates. The local machine will observe this shutdown and subsequently fetch the downloaded results from the S3 bucket.
As you can see, with this setup there are almost no requirements to the local machine. Provided the docker images are built and published, the specified benchmark tool version is available online and OpenML is used, the only data transfer between the local machine and the cloud is the upload of configuration files and the download of results.

C.2 Framework Versions
In principle the latest version of each AutoML framework as of 30 th of September in 2021 was used. 20 The specific version number of each framework can be seen in Table 10. For frameworks that are installed directly from Github, only the first 10 characters of the commit hash are shown. For each framework, the latest available version as of the 9 th of April 2022 is also shown. 20. In GAMA version 21.0.0, released in January 2021, ensembling was only available through the 'postprocessing' hyperparameter. To comply with the benchmark design of only allowing a 'preset' hyperparameter, the 21.0.1 update introduced a 'performance' preset that uses ensembling.

D.1 Class Imbalance
The two classification tasks with a large amount of failures despite being small are 'yeast' and 'wine-qualitywhite', which feature a minority class with only 5 instances. This means that within the 10-fold crossvalidation we perform in our experiments, either 4 or 5 of those instances are available in the training splits. We see that only in the case where one of those samples is in the test set failures occur. The exact error message differs per framework, though they indicate that evaluating pipelines fails. This is likely due to e.g., using 5-fold cross-validation out of the box. Failure on these specific datasets (and folds) is only observed for GAMA, LightAutoML, and TPOT.
This is specific to MLJarSupervised. While we can only guess, we assume it is related to the extensive AutoML pipeline MLJarSupervised has. It includes 10 different steps, including three steps for feature generation and selection and three steps for ensembling and stacking. These steps are not turned on by default 21 , feature engineering is only turned on for 'performance' and 'compete', and ensembling and stacking is only used in 'compete' mode, which we used. 21. https://supervised.mljar.com/features/modes/