{'fraction_non_alphanumeric': 0.042118169466812475, 'fraction_numerical': 0.013586506279616927, 'mean_word_length': 4.716612994885395, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sampling Acquisition Functions for Batch Bayesian Optimization

This paper presents Acquisition Thompson Sampling (ATS), a novel algorithm for batch Bayesian Optimization (BO) based on the idea of sampling multiple acquisition functions from a stochastic process. We define this process through the dependency of the acquisition functions on a set of model parameters. ATS is conceptually simple, straightforward to implement and, unlike other batch BO methods, it can be employed to parallelize any sequential acquisition function. In order to improve performance for multi-modal tasks, we show that ATS can be combined with existing techniques in order to realize different explore-exploit trade-offs and take into account pending function evaluations. We present experiments on a variety of benchmark functions and on the hyper-parameter optimization of a popular gradient boosting tree algorithm. These demonstrate the competitiveness of our algorithm with two state-of-the-art batch BO methods, and its advantages to classical parallel Thompson Sampling for BO.


Introduction
Bayesian optimization (BO) methods deal with the optimization of expensive, black-box functions, i.e., functions with no closed-form expression or derivative information. Many such problems arise in practice in fields such as: hyper-parameter optimization of machine learning algorithms (Snoek et al., 2012), synthesis of short polymer fiber materials (Li et al., 2017), and analog circuit design (Lyu et al., 2018). Therefore, in recent years, BO gained widespread popularity to the point that its versatility was showcased on toy problems like optimizing chocolate cookie recipes (Solnik et al., 2017).
From a high-level perspective, BO works by training a probabilistic regression model (typically, Gaussian Processes) on the available function evaluations. This model is then employed to decide where to evaluate the function next, maximizing a trade-off between exploration and exploitation that is formalized through an acquisition function a(x). Many sequential BO algorithms exist, differing by the choice of the regression model, the acquisition function, or properties of the input domain (Hutter et al., 2011;Srinivas et al., 2012;Shahriari et al., 2016;Springenberg et al., 2016;Hernandez-Lobato et al., 2015). Recently, parallel BO techniques have gained popularity. These methods select multiple points to be evaluated in parallel in order to better exploit modern hardware capabilities, and to speed up the overall optimization process in the case of costly evaluations like training machine learning algorithms. In the parallel case, typically, the acquisition function must return q points at once (batch BO), these being greedily selected one after the other.
Most successful batch BO algorithms focus on parallelizing a specific acquisition function (Kandasamy et al., 2018;Wu & Frazier, 2016;Daxberger & Low, 2017;Shah & Ghahramani, 2015;Gupta et al., 2018). In this paper, instead, we will focus on methods that can be applied to any sequential acquisition function, as in general no single one consistently outperforms the others (Hoffman et al., 2011). Amongst these general methods, the prevalent approach is to modify the landscape of the acquisition function a(x) after a point is scheduled so as to avoid returning another point in its neighborhood. For instance, Local Penalization (LP) (González et al., 2016) accomplishes this by suppressing (penalizing) the value of a(x) in the neighborhood of any scheduled x using an approximation of the Lipschitz constant of the unknown function. A family of algorithms including, e.g., B-LCB (Desautels et al., 2014), instead, obtains the same effect by hallucinating the results of the points that have been scheduled but not yet completed.

Our Intuition
In general, a sequential acquisition function depends on a vector of parameters mostly belonging to the employed regression model. We will denote the parameter vector by θ, and write a(x) = a(x, θ) to refer to a member of the family of acquisition functions. The θ vector exerts a strong effect on the practical performance of BO (Snoek et al., 2012). Hence, in practice, it is either set to its maximum likelihood value, or its influence is mitigated by using an arXiv:1903.09434v1 [cs.
LG] 22 Mar 2019 acquisition function averaged over a fixed number of possible values of θ. Our intuition is that, rather than focusing on modifying the landscape of a fixed a(x, {θ i }), where {θ i } possibly denotes a set of vectors, such a dependency can be exploited to select each point in a batch from a different member of the family. In order to do so, inspired by Parallel Thompson Sampling (P-TS) (Kandasamy et al., 2018), we interpret a given sequential acquisition function a(x, {θ i }) as being sampled from a stochastic process. It then suffices to draw samples from the process to obtain as many different acquisition functions as the number of batch points to evaluate.

Contributions
• We present ATS, a novel batch Bayesian optimization algorithm based on the idea of sampling acquisition functions from a stochastic process. ATS is conceptually simple, easy to implement and different from the previously published randomized approach (Kandasamy et al., 2018), in that it can parallelize any arbitrary sequential acquisition function.
• We show how existing techniques to diversify points within a batch, i.e., jitter and hallucination, can be incorporated into our randomized framework to improve performance of ATS on highly multi-modal functions.
• We present experimental results on five different wellknown benchmark functions and on hyper-parameter tuning for XGBoost (Chen & Guestrin, 2016). We demonstrate and explain the advantages of ATS and its variants compared to P-TS, B-LCB and LP, three popular state-of-the-art batch BO methods.

Problem Statement and Background
Our goal is to find the minimum of a black-box function f : X → R, defined on a bounded set X ⊂ R d . While the function itself is unknown, we have access to the results of its evaluations f (x) at arbitrary domain points x.
We are interested in scenarios where these evaluations are costly and we seek to minimize f using as few evaluations as possible. For instance, f might represent the loss incurred by a machine learning algorithm trained with the hyper-parameter vector x.

Bayesian Optimization
Bayesian Optimization (BO) approaches this black-box optimization problem by placing a prior on f and updating the model by conditioning on the evaluations as they are collected. Once the surrogate model is updated, BO employs it to select the next single point to evaluate according to a criterion referred to as the acquisition function. This function should not only "exploit" the regions that are likely to contain the minimum, but also "explore" those where uncertainty is significant.
Gaussian Processes (GP) are a common and versatile choice for the function prior. Under a GP, any finite number of function outputs is jointly distributed according to: where the covariance K of the Gaussian process is defined through a kernel function k, parametrized by θ k as The choice of the kernel function k determines the family of functions that a GP can represent.
In the following, we will use the Matérn 5/2 kernel due to its widespread usage and its good empirical performance on tuning machine learning algorithms (Snoek et al., 2012). Moreover, we will assume the prior mean µ p to be constant (yet possibly non-zero) on the input domain. In general, while the function f is deterministic, its evaluations may be noisy. It is, therefore, common practice to assume Gaussian noise in addition to the GP, so that y i = f (x i )+N (0, σ 2 n I). We will denote the set of collected observations up to evaluation t with D t = {(x i , y i )) | i ∈ {1, . . . , t}}. A desirable property of GP's in this context is that N (µ, K|D t ) is again a GP, and its posterior mean and variance at point x * are respectively (Rasmussen & Williams, 2006): where k * denotes the vector with elements k * ,i = k(x * , x i ) and K i,j = k(x i , x j ). In the remainder of the paper, we will ignore the evaluation noise and set y i = f (x i ) to facilitate the presentation. In order to generalize to the noisy case, it suffices to substitute f (x i ) with y i and to include the noise in the kernel matrix.
Once t evaluations are available, the next point is selected by maximizing the acquisition function: x t+1 = arg max X a(x|D t ), where the conditioning explicitly denotes the dependence on the updated surrogate model taking evaluations up to t into account. Two widespread sequential acquisition functions are Expected Improvement (EI) (Jones et al., 1998) and Lower Confidence Bound (LCB) (Srinivas et al., 2012): where f (x − ) denotes the best (smallest) evaluation result recorded until evaluation t.

Parallel Bayesian Optimization
Bayesian Optimization as presented in the previous section is an inherently sequential algorithm: points are evaluated one by one, and the model is updated to reflect the evaluation results. Many works have explored the possibility of evaluating more than one point at once, either asynchronously (Snoek et al., 2012;Kandasamy et al., 2018) or synchronously (González et al., 2016;Desautels et al., 2014;Gupta et al., 2018;Kathuria et al., 2016). Next, we will focus on the latter, more studied, case, as asynchronous algorithms can also be employed in a synchronous fashion.
Some approaches, like q-EI by Wang et al. (2016), theoretically generalize sequential acquisition functions to the case in which q points must be returned at once. Unfortunately, such approaches are not computationally feasible for q > 4, as the batch of points needs to be jointly optimized (Daxberger & Low, 2017). Therefore, most of the attention has been devoted to greedy methods, where points are selected one after the other. In this context, we will call pending evaluations, the points for which the evaluation has been scheduled but not yet completed. We will denote all pending evaluations with the prime sign: if evaluation i is pending, we will write x i , meaning that x i is available, but not f (x i ).
A plethora of approaches for batch BO exists; here, we will describe those that form the background to our algorithms. A well-known heuristic is to hallucinate the results of the pending evaluations, i.e., to approximate them with known quantities. This way, the acquisition function is up- where h i denotes the hallucinated evaluation result for the pending evaluation x i . For instance, the very effective batch LCB (B-LCB) approach updates the surrogate model after each batch point has been scheduled by setting f (x i ) = µ(x * |D t ). The batch BO algorithm by Gupta et al. (2018), instead, parametrizes the LCB acquisition function to express different explore-exploit trade-offs and, arguing that different trade-offs are associated to different maxima of a(x|D t ), assigns each batch point a new trade-off.

Thompson Sampling
Thompson Sampling (TS) is a rather old heuristic to address the explore-exploit dilemma that gained popularity in the context of multi-armed bandits (Chapelle & Li, 2011). The general idea is to choose an action according to the probability that it is optimal. Applied to BO, this corresponds to sampling a function g from the Gaussian Process posterior and selecting x t+1 = arg max X g (Kandasamy et al., 2018). In fact, let p * (x) be the probability that point (action) x optimizes the GP posterior, then: TS is therefore equivalent to a randomized sequential acquisition function, one that is easy to parallelize. In fact, it suffices to sample multiple functions from the GP posterior to get a batch of points to be evaluated in parallel (Kandasamy et al., 2018). In this paper, we generalize the reasoning in (4) so as to parallelize an arbitrary acquisition function, and additionally incorporate ideas from hallucinations and intra-batch trade-off differentiation presented in the previous section.

Sampling Acquisition Functions
We first introduce Acquisition Thompson Sampling (ATS), our novel algorithm for both batch and asynchronous parallel Bayesian Optimization ( §3.1) and then discuss two extensions thereof, j-ATS ( §3.2) and h-ATS ( §3.3). The common, novel idea behind all of these methods is to sample an acquisition function from a stochastic process for each parallel evaluation to perform.

Acquisition Thompson Sampling (ATS)
In general, any sequential acquisition function a(x) explicitly depends on the surrogate model's hyper-parameters θ and can be therefore written as a(x, θ). The most common way to choose θ is to optimize the marginal likelihood under the Gaussian process. The acquisition functions are very sensitive to these hyper-parameters and it is therefore advisable to adopt a fully Bayesian approach by marginalizing over them, as suggested by Snoek et al. (2012). Specifically: This sensitivity is further confirmed by their empirical evaluations. The evaluation of the integral in (5), in practice, is approximated byã s (x), the empirical average of s samples from the data posterior of θ obtained via Markov Chain Monte Carlo (MCMC) sampling: In this paper, however, we use a different approach: we do not seek to fully marginalize over the hyper-parameters, but instead, to leverage the sensitivity to θ for selecting a diverse batch of points. The main intuition is to recognize thatã s (x) represents a sample from a probability distribution over functions that is implicitly defined by the θ data posterior, i.e., a stochastic process. We will denote such a process by a s (x|D t ), and writeã s ∼ p(ã s |D t ) to refer to the sampling in (6). We can then argue that the sequential criterion for determining the next point to evaluate, Algorithm 1 Batch iteration, Acquisition Thompson Sampling (ATS).
can be seen as sampling from p a * s (x), the probability that x maximizes the stochastic process. In fact, we can write: where δ denotes the Dirac delta distribution. In light of this interpretation, a sequential acquisition function can be employed in the parallel setting by sampling as many different acquisition functions from a s (x|D t ) (i.e., as many Monte Carlo approximations of the marginalized function a(x|D t )) as the parallel evaluations to perform. We call this procedure ATS. Our algorithm closely resembles Thompson Sampling for Bayesian Optimization (cf. §2.3): in fact, (7) can be interpreted as an adaptation of (4) to a case in which the optimal action is defined with respect to a stochastic acquisition function rather than the GP posterior. While parallel TS is an acquisition function of its own, this adaptation allows us to obtain a general approach, applicable for the parallelization of any sequential acquisition function by means of its parametrization θ. Both ATS and TS easily work in the asynchronous setting, in addition to batch setting. Moreover, ATS can be also employed on top of an existing greedy parallel approach to make it scale further. Algorithm 1 outlines the pseudo-code for a single iteration of the batch version of ATS.
In order to link theory to practice, it remains to fully specify the data posterior of θ. Making use of the Bayes rule, we can write: p(θ|D t ) = 1 Z p(D t |θ) p(θ), where 1 Z is a normalization constant. p(D t |θ) is the marginal likelihood of the collected data under the Gaussian process (Rasmussen & Williams, 2006), while p(θ) is the prior over the model hyper-parameters. We found the performance on high-dimensional functions (e.g., d = 6) to be strongly dependent on the choice of the prior and on the output range of f . To mitigate the effect of the latter, we employ z-normalization of the outputs {y i } after each model update. For what concerns the prior, we explored differ-ent options and eventually chose a modification of that employed by the GPyOpt authors (2016), as it was the one that performed the best in a variety of cases. Specifically: p(θ k ) = Γ(α = 1, β = 6) and p(µ) = Uniform(−3, 3), where θ k are the GP kernel hyper-parameters, and µ is the constant prior mean of the GP. The priors are independent as in Snoek et al. (2012).

Sampling Explore-Exploit Trade-Offs (j-ATS)
If f exhibits multiple minima, ATS might over-exploit the current posterior of the function, producing a batch of points centered around a local minimum. In the hope of ending up in different valleys of f , we encourage the exploration of regions of the input domain with varying degrees of posterior uncertainty by pursuing different explore-exploit trade-offs within a batch. This approach, devised by Gupta et al. (2018), was originally implemented by explicitly solving a multi-objective optimization problem on LCB j (x|D t ) = jσ(x|D t ) − µ(x|D t ), a parametrized version of the LCB acquisition function, and taking M different trade-offs from the resulting Pareto curve (one per batch point). In this context, the parameter j is commonly called jitter: the larger the jitter, the more explorative the acquisition function.
Rather than explicitly solving the multi-objective optimization as Gupta et al. (2018), we translate this idea into our randomized framework by sampling different tradeoffs from a distribution. In order to do this, we associate a new process a s,j (x|D t ) to the parametrization, and place an independent prior on j so as to sample jitter alongside the θ i vectors. In addition to LCB, such a parametrization is commonly employed for many popular acquisition functions; e.g., EI (the GPyOpt authors, 2016): As a consequence, sampling from a s,j (x|D t ) implies obtaining acquisition functions having different exploreexploit trade-offs. A batch iteration of this jittered variant (j-ATS) of the ATS algorithm is outlined in Algorithm 2.
It now remains to choose p(θ, j|D t ), the data posterior defining the process. As the data likelihood is independent of the parameters that do not belong to the surrogate model, we only need to define the prior of j. Let C be a Bernoulli random variable with p = 0.5. Then: while we revert to ATS, as presented in Section 3.1 (without jitter), if C = 0.

5:
x t+i ← arg max Xãs,j (x|D t ), the i-th batch point.

Hallucinating Pending Evaluations (h-ATS)
As explained in Section 2.2, approaches for parallel BO like (Desautels et al., 2014) work by augmenting the dataset with hallucinations of the pending evaluations. Evaluation i is then selected by having a(x) make use of the hallucinated surrogate model N (µ, where t is the time of the last completed evaluation and h k denotes a hallucination of f (x k ). As the h k 's are approximations resulting from the last updated surrogate model, these approaches might be over-confident on the collected observations (Kathuria et al., 2016).
If we perform the sampling of the acquisition functions in a sequential manner, we can exploit the same technique in our randomized framework by hallucinating the data posterior, rather than the surrogate model (which remains independent of the pending evaluations). By doing that, we can make use of the knowledge of the locations of the pending evaluations in order to "update" the stochastic process to a s (x|D i−1 ). This means that, at iteration i, we sample: From the high-level perspective, sampling from the hallucinated data posterior means sampling models that should explain the hallucinations. In fact, the difference between p(θ|D t+i−1 ) and p(θ|D t ) lies in that the former takes the hallucinations into account in its data likelihood . The practical effect, analogously to j-ATS, is to increase the diversity across the batch points. In this case, though, no acquisition function-specific jitter prior needs to be devised; the approach is therefore more readily applicable to new acquisition functions. For what concerns hallucinations, we adapt the approximation employed by Desautels et al. (2014) to

6:
make use of the θ q samples: where the θ q s are those sampled using the data posterior onD i−1 , as in (9), but the posterior is on the last updated model (without hallucinations). We summarize a batch iteration of this hallucinated ATS (h-ATS) in Algorithm 3.

Experimental Results
This section presents the results of our empirical evaluations for the three variants of the proposed ATS algorithm. In Section 4.1 we benchmark on well-known synthetic functions from the global optimization literature and in Section 4.2 we present the results on the hyper-parameter tuning of a gradient boosting tree algorithm on a binary classification task.
Methodology and Baselines. We compare against two state-of-the-art, well-known batch Bayesian optimization algorithms: Local Penalization (LP) (González et al., 2016) and batch LCB (B-LCB) (Desautels et al., 2014), and against Parallel "classical" Thompson Sampling (P-TS) (Kandasamy et al., 2018). LP was taken from the implementation available within the open-source GPyOpt package (the GPyOpt authors, 2016). We instead reimplemented B-LCB and P-TS within our framework. While LP is a general batch BO approach (applicable to any acquisition function), B-LCB was presented and analyzed only for LCB (therefore, even though their hallucination technique might generalize, we adhere to the common experimental practice) and P-TS parallelizes the TS acqui- sition function only. Both B-LCB and LP are very competitive against many of the more complex, non-general algorithms presented in the last two years (Daxberger & Low, 2017;Gupta et al., 2018;Lyu et al., 2018;Kathuria et al., 2016). Moreover, their performance is superior to other algorithms commonly employed as baselines, such as q-KG (Wu & Frazier, 2016) and q-EI (Chevalier & Ginsbourger, 2013); we hence forgo such a comparison.
We employ two widely used acquisition functions, LCB and EI (cf. Section 2.1) for ATS and LP. For all algorithms, we employ a GP surrogate model with Matérn 5/2 kernel (Snoek et al., 2012). For LP, B-LCB and P-TS, consistently with common practice (Lyu et al., 2018), the acquisition function parameter vector θ is set to its maximum likelihood value after each non-hallucinated model update 1 . For ATS, we instead sample multiple θs from its data posterior, as explained in Section 3.1. We perform the sampling by employing the Affine Invariant MCMC Ensemble sampler provided by the emcee python library (Foreman-Mackey et al., 2013). We decided to perform all the experiments in the batch BO setting, rather than asynchronous, to keep the uncertainty (i.e., the number of pending evaluations) constant and fixed across the algorithms and time. We remark that in practice all the three employed algorithms can be easily adapted to work in the asynchronous setting. In all cases, before starting the BO algorithms, the surrogate models are initialized with 5 random evaluations. Table 1 summarizes the characteristics of the synthetic functions we benchmark on. These functions are notoriously hard to optimize due to the presence of multiple local or global minima (Eggholder, Cosines, Hartmann), or deep valley-like regions (Branin, Rosenbrock). Therefore, they are well-known in the global optimization community, and have been previously employed as benchmarks in the context of batch BO (Kathuria et al., 2016;González et al., 2016;Gupta et al., 2018;Kandasamy et al., 2018).

Synthetic Functions
Parallelizing a Sequential Acquisition Function. Before benchmarking against the parallel BO baselines, we illustrate the effectiveness of ATS in parallelizing an arbitrary acquisition function. Figure 1 compares ATS to sequential BO, using LCB or EI, on three qualitatively different synthetic functions, plotting the smallest function value found until batch iteration t. Mean values and one standard error over at least 10 repetitions are reported. We see that ATS consistently outperforms the sequential baseline and achieves close to an ideal speedup, equal to the batch size, on the high-dimensional and multi-modal Hartmann function. The speedups on Eggholder and Branin (3.1× for M = 5 and 5× for M = 10, respectively) further highlight the benefits of parallel BO.  that the relative performance of acquisition functions depends on the optimization problem at hand (Hoffman et al., 2011). The plots report mean values of regret (distance from the function minimum, clipped at 1e−6) over at least 10 repetitions of the experiments; the ATS variant with the best performance on the given task is highlighted with a thicker line. In all plots, plain ATS eventually improves on P-TS, highlighting the benefits of focusing on the probability of maximizing an acquisition function rather than the model posterior. We separate the benchmark functions into two categories: Figure 3 presents results for valley-like functions. B-LCB seems to excel on low-dimensional, relatively easy to optimize functions like Branin. The ATS variants do not show significantly different behavior, with one exception. On the Rosenbrock function, as the optimization proceeds, plain ATS gets below the current minimum found by all the other parallel algorithms: this suggests that the more explorative functions are not fully capable of descending the deep valley region; pure exploitation eventually pays off.

ATS
On the strongly multi-modal functions shown in Figure 2 (Cosines and Eggholder exhibit sinusoidal oscillations), instead, differences become more noticeable and the ATS variants perform particularly well. The simplest variant of parallel behavior. our algorithm, plain ATS, is slower than j-ATS and h-ATS in the first iterations. By closely looking within the first batches, we indeed noticed a lack of diversity between the points. In fact, the posterior is particularly uninformative at the beginning of the optimization loop, so over-exploiting it, as ATS does, might be myopic. This is precisely the intuition that led us to the addition of jitter in the formulation of j-ATS in Section 3.2. j-ATS on average is the variant showing the best performance and clearly beats (Hart-2 4 6 iteration mann and Cosines), or is comparable to (Eggholder), the baselines on the three multi-modal functions, where otherwise very effective algorithms like B-LCB seem to get stuck on local minima. In both Figures, h-ATS appears to have a performance similar to that of ATS, except on the Eggholder function. There, the use of hallucinations to diversify the θ posteriors seems to be crucial to avoid stagnating in one of the many egg-holder-like minima, and yields superior performance with respect to the other methods. A full summary of the optimization results at the last iteration of the plots in Figures 2 and 3, including the standard error of the mean, can be found in Table 2.  We perform a similar comparison on the real-world problem of Hyper-Parameter Optimization (HPO). We focus on gradient boosting decision trees, as they enjoy widespread adoption in academia, industry and competitive data science due to their state-of-the-art performance in a wide variety of machine learning tasks (Ke et al., 2017). Specifically, our goal is to tune the hyper-parameters of the popular XGBoost (Chen & Guestrin, 2016), so as to minimize the logistic loss incurred on a binary classification problem. We utilized the well-known Higgs (P. Baldi and P. Sadowski and D. Whiteson, 2014) dataset, freely available within the UCI ML repository, whose task is to predict whether a given process will produce Higgs bosons  or not. As the overall number of signals in the original dataset is larger than ten millions, we uniformly subsampled ≈ 1 120 000 examples for the training, and ≈ 400 000 for the validation set to make the experiments feasible on our infrastructure. Table 3 reports the domains for the 5 hyper-parameters we tuned; we kept the maximum number of bins equal to 64. Figure 4 shows the minimum loss incurred on the fixed validation set after t batch iterations, averaged over 5 repetitions. We opted for a large batch size (M = 20) to mimic a realistic scenario where the significant time cost of HPO needs to be amortized through parallelism. Moreover, we chose LCB over EI due to its better sequential performance, as in the previous Section. The results show that ATS-based algorithms are competitive to B-LCB also on HPO, while LP shows inferior performance. The ATS variant exhibiting the best behavior is, again, j-ATS, which achieves almost ideal speedup to sequential BO. This is to be expected, as in larger batches the diversity provided by sampling θ alone might not be sufficient, and employing hallucinations might be short-sighted with respect to the future model updates.

Conclusions
We presented Acquisition Thompson Sampling (ATS), a novel algorithm for parallel Bayesian Optimization (BO) based on sampling acquisition functions from a stochastic process through their dependency on the surrogate model hyper-parameters. Moreover, we enhanced this conceptually simple algorithm by translating ideas from the batch BO literature into our randomized framework.
First (j-ATS), by adding jitter to the acquisition functions, we sample different explore-exploit trade-offs from a prior distribution embedded in the definition of the process, as opposed to obtaining these trade-offs by solving a multiobjective optimization problem (Gupta et al., 2018).
Second (h-ATS), we employ hallucinations of the pending evaluations to diversify the data posterior of the surrogate model hyper-parameters, rather than to update the surrogate model itself (Desautels et al., 2014). We obtain a versatile and multi-faceted algorithm, usable to parallelize any sequential acquisition function.
Finally, we benchmark against two state-of-the-art batch BO algorithms (González et al., 2016;Desautels et al., 2014) and against parallel Thompson Sampling (Kandasamy et al., 2018) on five synthetic functions from the global optimization literature, and on tuning a popular gradient boosting tree algorithm. It is shown that variants of ATS are competitive with the employed baselines and, in particular, out-perform them on strongly multi-modal functions.