{'fraction_non_alphanumeric': 0.02835934526075371, 'fraction_numerical': 0.012228778073848496, 'mean_word_length': 4.624029970564624, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Comparison of Bayesian and particle swarm algorithms for hyperparameter optimisation in machine learning applications in high energy physics

When using machine learning (ML) techniques, users typically need to choose a plethora of algorithm-specific parameters, referred to as hyperparameters. In this paper, we compare the performance of two algorithms, particle swarm optimisation (PSO) and Bayesian optimisation (BO), for the autonomous determination of these hyperparameters in applications to different ML tasks typical for the field of high energy physics (HEP). Our evaluation of the performance includes a comparison of the capability of the PSO and BO algorithms to make efficient use of the highly parallel computing resources that are characteristic of contemporary HEP experiments.


Introduction
Machine learning (ML) methods often aid the analysis of the vast amounts of data that are produced by contemporary high energy physics (HEP) experiments. The ML algorithms often feature tunable parameters, referred to as hyperparameters, which need to be chosen by the user and often have a significant effect on the algorithm's performance. In a previous publication [1] we presented two different algorithms, particle swarm optimisation (PSO) and the genetic algorithm, for the autonomous determination of these hyperparameters. In the present paper we compare the performance of the PSO algorithm, the more promising of the two algorithms studied in our previous publication, to the performance of the Bayesian optimisation (BO) [2,3,4,5,6] algorithm. The latter is widely used for the task of finding optimal hyperparameter values in the context of ML applications since the pioneering work of Refs. [7,8,9]. The "asynchronous successive halving algorithm" (ASHA) [10] is an alternative algorithm for optimising the values of hyperparameters, which is popular in the ML community outside the field of HEP.
As in our previous publication, we formulate the task of determining the set of optimal hyperparameter values as a function maximisation problem. More specifically, given an ML algorithm A, we seek to find a point h in the space H of hyperparameters, such that the performance of the ML algorithm A attains its maximum at this point: where the objective (or "score") function (OF) s(h) quantifies the performance of the ML algorithm A, and the point h at which s(h) attains its maximum is denoted by the symbolĥ. We compare the performance of the PSO and BO algorithms on two benchmark tasks, the task of finding the minimum of the Rosenbrock function [11], and on a typical data analysis task in the field of HEP, the "AT-LAS Higgs boson machine learning challenge" [12].
An important aspect in applications of ML algorithms in the field of HEP is an algorithm's capability to make efficient use of massively parallel computing facilities. A single training of an ML algorithm on a single machine may take several hours, days, or, in extreme cases, even weeks. In the context of the hyperparameter optimisation task, such a single training corresponds to a single evaluation of the OF s(h). In order for the hyperparameter optimisation task to finish within an acceptable time, different evaluations of the OF, i.e. different ML trainings, need to be executed in parallel. The computing facilities of contemporary HEP experiments typically allow users concurrent access to hundreds, sometimes even thousands, of machines. It is therefore of high practical relevance whether the PSO and BO algorithms can organise the hyperparameter optimisation task such that hundreds of ML trainings can be executed in parallel. We find that both algorithms fulfill this requirement, but do exhibit some differences in performance compared to the case that all ML trainings are executed sequentially on a single machine.
The manuscript is structured as follows: In Section 2, we present the main concepts of the BO algorithm. The PSO algorithm is described in Ref. [1]. In Section 3, we compare the performance of the PSO and BO algorithms on the two benchmark tasks. The study of the parallelisation capability of the two algorithms is presented in Section 4. We conclude the paper with a summary in Section 5.

Bayesian Optimisation
The BO algorithm is designed to facilitate the numerical maximisation of objective functions s(h) which are time-consuming to evaluate and for which the analytical form is in general not known. The BO algorithm further allows to solve the maximisation task without using derivative information on s(h). This is achieved by performing the maximisation not on s(h) directly, but on an approximation of s(h), which is referred to as the "surrogate" function (SF). The SF is chosen such that it is fast to evaluate and its analytic form, including its derivative, is known. We use a Gaussian process [13] with the Matérn kernel [14] for the SF in this paper. For each point h ∈ H, the SF provides two values: an estimate for the value s(h) of the OF and a confidence interval. The latter represents an estimate of the accuracy of the approximation of the OF by the SF at this point.
The numerical maximisation of the OF is performed by an iterative procedure. Each iteration consists of two steps: The first step consists of finding the next point h for which to perform the timeconsuming evaluation of the OF. The task of finding this point is performed by an "acquisition function" (AF). The inputs to the AF consist of the estimate for the values s(h) provided by the SF and the estimated accuracy of the approximation of the OF by the SF at this point. In the second step, the SF is updated with the information of the value of the OF at the point h, in order to improve the accuracy of the approximation. Each evaluation of the OF thus serves two purposes: first, to find points h where the OF attains a higher value s(h) compared to previously found points and second, to improve the accuracy of the approximation of the OF by the SF. We use the expected improvement (EI) [15] for the AF in this paper. The time-consuming evaluation of the OF is performed at the point where the AF reaches its maximum. The search for the maximum of the AF is performed numerically, using the Broyden-Fletcher-Goldfarb-Shanno (BFGS) [16,17,18,19] algorithm. The numeric search for the maximum takes advantage of the fact that both the SF and the AF are fast to evaluate, which allows multiple evaluations of the SF and AF to be made in order to find the best point for for which to make the time-consuming next evaluation of the OF. The EI acquisition function has a parameter, denoted by the symbol ξ, that allows to regulate how much importance is given to finding points h where the OF attains a high value ("exploitation") versus finding points which improve the accuracy of the approximation ("exploration"). The former are typically located in the neighbourhood of previously found points, while the latter are typically located in previously unexplored regions of the space H. The update of the SF after each evaluation of the OF is the feature to which the Bayesian optimisation algorithm owes its name. The SF is referred to as "prior function" before evaluating the OF and as "posterior function" afterwards.
The BO algorithm is started by evaluating the OF at an initial set of points and fitting the SF to the values s(h) of the OF at these points. This initial set of points is chosen with the objective of populating the space H uniformly. We use the Latin hypercube [20] algorithm to obtain this initial set of points. After fitting the SF to the values s(h) of the OF at these points, the BO algorithm enters the iterative phase: Given the SF, the BFGS algorithm is used to find the location where the AF attains its maximum. The OF is then evaluated at this point and the SF is updated. These steps are repeated until either the algorithm has converged, indicated by changes in the value of the OF that fall below a given threshold, or the computing resources are exhausted, i.e. a maximum number of iterations or a computing-time limit is reached.
The operation of the BO algorithm is illustrated by means of an example in Figs. 1 and 2, which visualise the seventh and eighth iteration of the BO algorithm, respectively. The OF is represented by the solid blue line and the SF by the dashed black line in the upper part of each figure. The red diamond-shaped markers indicate the points where the OF has been evaluated previously (including 2 initial points, which were chosen randomly in this example). The shaded blue area represents the confidence interval that quantifies the accuracy of the approximation of the OF by the SF. The AF is represented by the solid red line in the lower part of the figures. The yellow circle along the red line indicates the maximum of the AF, i.e. the point h where the next evaluation of the OF is performed.  The BO algorithm has been implemented as described above by the authors. We have validated our implementation by comparing its performance to the implementation of the BO algorithm included in the scikit-optimize [21] package.
The BO algorithm that we described above allows to evaluate the OF at one point h per iteration of the algorithm. This is well suited for executing the BO algorithm sequentially on a single machine. For the purpose of optimising the hyperparameters of ML algorithms the training of which is performed on massively parallel computing facilities, the iterative phase of the BO algorithm needs to be extended to provide multiple points h per iteration of the BO algorithm. This extension of the BO algorithm is non-trivial and still a field of active research [22]. In this paper, we follow the approach described in Ref. [23], referred to as "multi-points expected improvement" (q-EI), and use the imple-mentation provided by Ref. [24] when running the hyperparameter parameter optimisation on multiple machines in parallel.

Performance
In this section, the performance of the PSO and BO algorithms are compared on the two benchmark tasks: the finding of the minimum of the Rosenbrock function and the ATLAS Higgs boson machine learning challenge.

Rosenbrock function
The Rosenbrock function [11] is a widely used trial function for evaluating the performance of function minimisation algorithms. It is defined as: It's domain is the x-y plane and it depends on two parameters a and b. The global minimum of the Rosenbrock function is located at (x, y) = (a, a 2 ). We choose the two parameters to be a = 1 and b = 10, resulting in the minimum to be located at (x, y) = (1, 1) and the value of the Rosenbrock function at the minimum to be R(1, 1) = 0. See Fig.  3 in Ref. [1] for a visualisation of the Rosenbrock function around the minimum. We treat the task of finding the minimum of the Rosenbrock function as a two dimensional hyperparameter optimisation problem. The optimal set of hyperparameters is scanned in the range of [−500, +500] × [−500, +500].
Both the BO and the PSO algorithms were executed for 30 iterations. During each iteration, 100 different hyperparameter sets were evaluated in parallel. The parameter settings for both algorithms are given in Tables 1 and 2 Table 2 Parameter settings for the PSO algorithm. The parameters are described in Ref. [1].
Performance We denote the location of the minimum found in each trial i by the symbolĥ i and the value of the Rosenbrock function at these points by the symbolR i = R(ĥ i ). The average of these values over the 1000 trials, R = 1 1000 · 1000 i=1R i , is shown in Fig. 3. For the first 10 iterations, the BO algorithm converges faster to the minimum than the PSO algorithm, but improves less rapidly than the PSO algorithm for more than 10 iterations. This difference in the rate of convergence is expected, since the BO algorithm has been developed for applications where the number of evaluations of the OF is in the order of a few hundred to a thousand [22] (here it is 3000, given by the product of 30 iterations times 100 different hyperparameter sets that are evaluated in parallel per iteration). Neither the BO nor the PSO algorithm converges to the true minimum R = 0 within 30 iterations. We have shown in Ref. [1] that the PSO algorithm converges to the true minimum after 10 4 iterations. The shaded band in Fig. 3 represents the standard deviation of theR i over the 1000 trials.

The ATLAS Higgs Boson machine learning challenge
The ATLAS Higgs Boson machine learning challenge (HBC) [12] constitutes the second benchmark task for evaluating the performance of the BO and PSO algorithms. As in our previous publication [1], we choose a boosted decision tree (BDT), implemented in the XGBoost [25] package, for the ML algorithm. The hyperparameter optimisation is performed with respect to the 7 hyperparameters given in Table 3 of Ref. [1]. During the optimisation, each of the 7 hyperparameters is restricted to be within the range given in Table 3.
The optimisation of the hyperparameters by the BO and PSO algorithms was run for 30 iterations. 70 BDTs, initialised with different random number seeds, were trained in parallel per iteration. The parameter settings used for the BO and PSO algorithm are the same as for the task of finding the minimum of the Rosenbrock function and are given in Tables 1 and 2, except for the parameter N inf o of the PSO algorithm, which was set to 7 for the HBC task. The 550000 signal and background events provided by the organisers of the HBC are split into training and test sets as described in Ref. [1]. The "modified approximate mean significance" (d-AMS) defined by Eq. (8) of Ref. [1] was used as objective function for the BDT training. The d-AMS scores were computed setting the coefficient κ that controls the penalty term against overtraining to 0.3. The hyperparameter optimisation was repeated for 100 trials.

Results
The average hyperparameter values found by the BO and PSO algorithms over the 100 trials and the standard deviation of these values are reported in Table 4 Table 4 Average values (µ) of the hyperparameters found by the BO and PSO algorithms and their standard deviation (σ) for the HBC task.
The performance of BDTs is evaluated on two separate samples of events, referred to as the public and private leaderboard samples [1,12]. Following Ref. [1], the performance is quantified by the "approximate mean significance" (AMS) score. The latter is averaged over the 100 trials. The motivation for using the d-AMS score as objective function for the training, while the (final) performance is quantified using the AMS score is detailed in Ref. [1]. The evolution of the average performance as function of the number of iterations is shown in Fig. 4. The bottom part of the figure shows the evolution of the d-AMS score, the objective function used for the BDT training. While the d-AMS score keeps increasing monotonously with more iterations, the AMS scores on the public and private leaderboard samples reach a plateau already after 5-10 iterations. In subsequent iterations, the AMS scores start to fluctuate around the plateau. The magnitude of the fluctuations is of O(10 −2 ). The differences in performance between the public and private leaderboard samples is compatible with a statistical fluctuation of the signal and background events contained in these samples [12]. The BO and PSO algorithms achieve a similar performance on the HBC task.

Parallelisation capability
The capability of the BO and PSO algorithms to make efficient use of parallel computing resources is studied on the task of finding the minimum of the Rosenbrock function. We denote by N parallel the number of points h in hyperparameter space that are evaluated in parallel per iteration of the BO and PSO algorithms. The value of N parallel represents the number of ML trainings that are executed in parallel on different machines. Ideally, the duration ("wall time") of the hyperparameter optimisation task should decrease inversely proportional to N parallel , except for some (small) overhead imposed by the BO and PSO algorithms.
Bayesian optimisation Based on the literature [22], one expects the BO algorithm to perform best when all ML trainings are performed sequentially on a single machine, with one point h in hyperparameter space evaluated per iteration. We have compared the performance, quantified by the values of R , obtained when executing our implementation of the BO algorithm (described in Section 2) sequentially on a single machine (N parallel = 1) with the performance obtained by the q-EI algorithm from Ref. [24], with the latter running either N parallel = 2, 5, 10, 25, 50, 100, 250, 500 or 1000 ML trainings in parallel. We find that the values of R are very similar in all cases and mainly depend on the total number of evaluations of the Rosenbrock function, given by the product of the number of iterations times N parallel . For 3000 evaluations, the value of R amounts to about 10 2 for all values of N parallel that we have tried.
The computing (CPU) time overhead imposed by the BO algorithm for updating the SF and finding the maximum of the AF amounts to about 50 CPU hours on a 2.30 GHz Intel ® Xeon ® E5-2695 v3 processor. The overhead does not vary much as function of N parallel . The task of updating the SF and finding the maximum of the AF needs to run (as a "supervisor" task) on a single machine. The overhead is sizeable compared to the computing time spent on performing 3000 evaluations of the Rosenbrock function, which only takes 0.06 CPU seconds, but becomes less important for "real" ML training applications: The total computing time spend on the HPC task (with 30 iterations and 70 ML trainings running in parallel) amounts to about 500 CPU hours.
Particle swarm optimisation In contrast to the BO algorithm, the capability to make efficient use of parallel computing resources is an intrinsic feature of the PSO algorithm. We find that the performance of the PSO algorithm depends on the value of N parallel , which corresponds to the number of particles in the swarm, in a non-trivial manner. Both too small and too large values of N parallel degrade the performance of the PSO algorithm compared to the optimal value. We find that the optimal value of N parallel amounts to about 2% times the total number (N tot ) of evaluations of the Rosenbrock function, i.e. the best performance of the PSO algorithm is achieved when using N parallel = 0.02 · N tot particles in the swarm and a fixed number of 50 iterations (up to 10000 evaluations of the Rosenbrock function that we have tried). We recommend to set the parameter N inf o of the PSO algorithm to 10% times N parallel .
We find that the CPU time overhead imposed by the PSO algorithm (spend on updating the positions and momenta of the particles in the swarm) is negligible for both benchmark tasks.
The CPU time overhead limits the "speedup" (reduction in wall time) that one can achieve by increasing the number of points in hyperparameter space that are evaluated in parallel. The effect is referred to as "Amdahl's law" [26] in the literature and visualized in Fig. 5. An ideal algorithm with negligible overhead would achieve a speedup factor equal to N parallel in this figure. The speedup achieved by the PSO algorithm on both benchmark tasks is very close to the ideal case. The CPU time overhead limits the speedup factor achievable by the BO algorithm to about 3 for the task of finding of the minimum of the Rosenbrock function and to about 12 for the HBC task.

Summary
We have compared the performance of two autonomous algorithms for the optimisation of hyperparameters, Bayesian optimisation (BO) and particle swarm optimisation (PSO), on two benchmark tasks typical for ML applications in the field of high energy physics: the task of finding the minimum of the Rosenbrock function and the ATLAS Higgs boson machine learning challenge.
We find that the BO algorithm performs better than the PSO algorithm when the total number of evaluations of the Rosenbrock function (equivalent to the number of ML trainings) is in the order of a few hundred to a few thousand. In case the number of evaluations (ML trainings) is large, the PSO algorithm outperforms the BO algorithm.
The capability of both algorithms to make efficient use of parallel computing resources is good. In particular, we find that the "multi-points expected improvement" of the BO algorithm provides similar performance when running on parallel computing resources compared to executing the BO algorithm sequentially on a single machine. In case the PSO algorithm, we found that the best performance is achieved when setting the number of particles in the swarm to 2% times the total number of function evaluations (ML trainings) and using a fixed number of 50 iterations.
We found that the BO algorithm may add a significant computational overhead to the task of finding the optimal hyperparameter values, while for the PSO algorithm the overhead is insignificant.