AutoMLBench: A Comprehensive Experimental Evaluation of Automated
  Machine Learning Frameworks
  Nowadays, machine learning is playing a crucial role in harnessing the power
of the massive amounts of data that we are currently producing every day in our
digital world. With the booming demand for machine learning applications, it
has been recognized that the number of knowledgeable data scientists can not
scale with the growing data volumes and application needs in our digital world.
In response to this demand, several automated machine learning (AutoML)
techniques and frameworks have been developed to fill the gap of human
expertise by automating the process of building machine learning pipelines. In
this study, we present a comprehensive evaluation and comparison of the
performance characteristics of six popular AutoML frameworks, namely,
Auto-Weka, AutoSKlearn, TPOT, Recipe, ATM, and SmartML across 100 data sets
from established AutoML benchmark suites. Our experimental evaluation considers
different aspects for its comparison including the performance impact of
several design decisions including time budget, size of search space,
meta-learning, and ensemble construction. The results of our study reveal
various interesting insights that can significantly guide and impact the design
of AutoML frameworks.
