Ranking Algorithms by Performance
  A common way of doing algorithm selection is to train a machine learning
model and predict the best algorithm from a portfolio to solve a particular
problem. While this method has been highly successful, choosing only a single
algorithm has inherent limitations -- if the choice was bad, no remedial action
can be taken and parallelism cannot be exploited, to name but a few problems.
In this paper, we investigate how to predict the ranking of the portfolio
algorithms on a particular problem. This information can be used to choose the
single best algorithm, but also to allocate resources to the algorithms
according to their rank. We evaluate a range of approaches to predict the
ranking of a set of algorithms on a problem. We furthermore introduce a
framework for categorizing ranking predictions that allows to judge the
expressiveness of the predictive output. Our experimental evaluation
demonstrates on a range of data sets from the literature that it is beneficial
to consider the relationship between algorithms when predicting rankings. We
furthermore show that relatively naive approaches deliver rankings of good
quality already.
