An Extensive Experimental Evaluation of Automated Machine Learning
  Methods for Recommending Classification Algorithms (Extended Version)
  This paper presents an experimental comparison among four Automated Machine
Learning (AutoML) methods for recommending the best classification algorithm
for a given input dataset. Three of these methods are based on Evolutionary
Algorithms (EAs), and the other is Auto-WEKA, a well-known AutoML method based
on the Combined Algorithm Selection and Hyper-parameter optimisation (CASH)
approach. The EA-based methods build classification algorithms from a single
machine learning paradigm: either decision-tree induction, rule induction, or
Bayesian network classification. Auto-WEKA combines algorithm selection and
hyper-parameter optimisation to recommend classification algorithms from
multiple paradigms. We performed controlled experiments where these four AutoML
methods were given the same runtime limit for different values of this limit.
In general, the difference in predictive accuracy of the three best AutoML
methods was not statistically significant. However, the EA evolving
decision-tree induction algorithms has the advantage of producing algorithms
that generate interpretable classification models and that are more scalable to
large datasets, by comparison with many algorithms from other learning
paradigms that can be recommended by Auto-WEKA. We also observed that Auto-WEKA
has shown meta-overfitting, a form of overfitting at the meta-learning level,
rather than at the base-learning level.
