Using Sequential Statistical Tests to Improve the Performance of Random
  Search in hyperparameter Tuning
  Hyperparamter tuning is one of the the most time-consuming parts in machine
learning: The performance of a large number of different hyperparameter
settings has to be evaluated to find the best one. Although modern optimization
algorithms exist that minimize the number of evaluations needed, the evaluation
of a single setting is still expensive: Using a resampling technique, the
machine learning method has to be fitted a fixed number of $K$ times on
different training data sets. As an estimator for the performance of the
setting the respective mean value of the $K$ fits is used. Many hyperparameter
settings could be discarded after less than $K$ resampling iterations, because
they already are clearly inferior to high performing settings. However, in
practice, the resampling is often performed until the very end, wasting a lot
of computational effort.
  We propose to use a sequential testing procedure to minimize the number of
resampling iterations to detect inferior parameter setting. To do so, we first
analyze the distribution of resampling errors, we will find out, that a
log-normal distribution is promising. Afterwards, we build a sequential testing
procedure assuming this distribution. This sequential test procedure is
utilized within a random search algorithm.
  We compare a standard random search with our enhanced sequential random
search in some realistic data situation. It can be shown that the sequential
random search is able to find comparably good hyperparameter settings, however,
the computational time needed to find those settings is roughly halved.
