Statistical comparison of classifiers through Bayesian hierarchical
  modelling
  Usually one compares the accuracy of two competing classifiers via null
hypothesis significance tests (nhst). Yet the nhst tests suffer from important
shortcomings, which can be overcome by switching to Bayesian hypothesis
testing. We propose a Bayesian hierarchical model which jointly analyzes the
cross-validation results obtained by two classifiers on multiple data sets. It
returns the posterior probability of the accuracies of the two classifiers
being practically equivalent or significantly different. A further strength of
the hierarchical model is that, by jointly analyzing the results obtained on
all data sets, it reduces the estimation error compared to the usual approach
of averaging the cross-validation results obtained on a given data set.
