{'fraction_non_alphanumeric': 0.04605071407752842, 'fraction_numerical': 0.014961624404935392, 'mean_word_length': 4.217979046975329, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Information-Theoretic Framework for Unifying Active Learning Problems

This paper presents an information-theoretic framework for unifying active learning problems: level set estimation (LSE), Bayesian optimization (BO), and their generalized variant. We first introduce a novel active learning criterion that subsumes an existing LSE algorithm and achieves state-of-the-art performance in LSE problems with a continuous input domain. Then, by exploiting the relationship between LSE and BO, we design a competitive information-theoretic acquisition function for BO that has interesting connections to upper confidence bound and max-value entropy search (MES). The latter connection reveals a drawback of MES which has important implications on not only MES but also on other MES-based acquisition functions. Finally, our unifying information-theoretic framework can be applied to solve a generalized problem of LSE and BO involving multiple level sets in a data-efficient manner. We empirically evaluate the performance of our proposed algorithms using synthetic benchmark functions, a real-world dataset, and in hyperparameter tuning of machine learning models.


Introduction
Level set estimation (LSE) is about determining a level set of an unknown function or, alternatively, a superlevel set of the function (i.e., a region of inputs where the function values are of at least a known threshold) given a finite budget of expensive (possibly noisy) function evaluations (Gotovos et al. 2013). It has important applications in environmental sensing/monitoring where the unknown function is a (spatial) field of some quantity of interest (e.g., pH, temperature, and solar radiation) (Galland, Réfrégier, and Germain 2004). On the other hand, Bayesian optimization (BO) has gained significant recognition in science and engineering fields (Brochu, Cora, and de Freitas 2010;Calandra et al. 2014;Krause and Ong 2011;Shahriari et al. 2015;Snoek, Larochelle, and Adams 2012) for its effectiveness in optimizing a black-box objective function (i.e., without a closedform expression/derivative) using a finite budget of expensive (possibly noisy) function evaluations. At first glance, one may straightforwardly regard BO as LSE by setting the threshold as the maximum value of the objective function, i.e., the superlevel set is reduced to a set of maximizers.
However, the unknown maximum value in BO does not satisfy the requirement of a known threshold in LSE. This poses the challenge of whether it is possible to develop a framework to unify LSE and BO. Though the work of Bogunovic et al. (2016) has developed such a unified approach called truncated variance reduction, it is demonstrated mainly on problems with a discrete input domain and requires enumerating over all inputs in a set of "unclassified points", which can be prohibitively large in practice (or infinite when the input domain is continuous and not discretized). In contrast, our work here proposes the first information-theoretic framework for unifying both LSE and BO that can empirically outperform the state-of-the-art LSE criteria and scale to real-world problems with a continuous input domain.
To shed light on the connection between LSE and BO, we propose to view BO as an active learning problem that involves actively estimating the superlevel set of the objective function with respect to an estimate of its maximum value; such a problem reduces to LSE when the maximum value is known instead. Improving the estimation of the superlevel set in turn refines the estimate of the maximum value. As the estimate approaches the true maximum value, the superlevel set becomes a set of the maximizers of the objective function. Unfortunately, existing LSE criteria cannot be directly applied to BO since they either impose a noiseless assumption (Low et al. 2012) or cannot handle an unknown threshold (Bryan et al. 2006).
A key contribution of our work here therefore lies in introducing a novel information-theoretic active learning criterion for LSE (Sec. 3) that can be exploited for designing a new acquisition function for BO with interesting connections to upper confidence bound (UCB) and max-value entropy search (MES) (Sec. 4). The latter connection reveals a drawback of MES (Remark 3), which has important implications on not only MES, but also on other MES-based acquisition functions such as those handling multiple objectives (Belakaria, Deshwal, and Doppa 2019;Suzuki et al. 2020) or fidelities .
The other main contribution of our work is to show how our proposed unifying information-theoretic framework can be applied to solve a generalized problem of LSE and BO involving multiple level sets/thresholds in a data-efficient manner. This problem, namely implicit LSE (Sec. 5), is about identifying a region of inputs whose function values differ from the (unknown) maximum value by at most a specified tolerance. It is motivated from the estimation of hotspots in environmental fields, which correspond to regions with large field measurements (Gotovos et al. 2013). In summary, the specific contributions of our work include: • A novel information-theoretic active learning criterion for LSE problems with a continuous input domain (Sec. 3), which subsumes an existing LSE criterion (Low et al. 2012) and empirically outperforms state-of-the-art LSE criteria (Bryan et al. 2006;Low et al. 2012) on synthetic benchmark functions and a real-world dataset (Sec. 6.1); • A new information-theoretic acquisition function for BO problems with interesting connections to UCB and MES; the latter connection reveals a drawback of using MES (Sec. 4). We empirically evaluate the performance of our proposed BO algorithm using several synthetic benchmark functions, a real-world dataset, and in hyperparameter tuning of a logistic regression model and a convolutional neural network for image classification with MNIST and CIFAR-10 datasets (Sec. 6.2); • Applying our unifying information-theoretic framework to solve the implicit LSE problem in a data-efficient manner (Sec. 5).

Gaussian Process (GP)
Let the unknown objective function be denoted as f : X → R over a bounded input domain X ⊂ R d . An LSE/BO algorithm repeatedly selects an input query x ∈ X for evaluating f to obtain a noisy observation y x f (x) + x of its function value f (x) corrupted by an additive Gaussian noise x ∼ N (0, σ 2 n ) with noise variance σ 2 n . Since it is expensive to evaluate f , the goal of LSE (BO) is to strategically select input queries for finding the level/superlevel set (global maximizer(s)) as rapidly as possible. To achieve this, we model f using a GP: Let {f (x )} x ∈X denote a GP, i.e., every finite subset of {f (x )} x ∈X follows a multivariate Gaussian distribution (Rasmussen and Williams 2006). Then, the GP is fully specified by its prior mean E[f (x )] and covariance for all x , x ∈ X ; the latter can be defined by, for example, the widely-used squared expo- . . , d ) and σ 2 s are its lengthscale and signal variance hyperparameters, respectively. For notational simplicity (and w.l.o.g.), the prior mean is assumed to be zero. Given a column vector y D (y x ) x ∈D of noisy observations from evaluating f at a set D of input queries selected in previous iterations, the GP posterior belief of the function value at any input query x is a Gaussian x ) with the following posterior mean µ x and variance σ 2 x : x ,x ∈D , K Dx K xD , and I is an identity matrix. Then, p(y x |y D ) = N (y x |µ x , σ 2 + σ 2 x + σ 2 n ).

Binary Entropy Search (BES) for Level Set Estimation (LSE)
An LSE algorithm repeatedly selects the next input query x ∈ X for evaluating f to maximize some active learning criterion based on the GP posterior belief of f given the observations y D obtained in previous iterations such that the a given threshold f • can be found as rapidly as possible.
In this section, we propose an information-theoretic active learning criterion for LSE which measures the information gain on the superlevel set X + f• from evaluating f at input query x to observe y x . Let γ • x denote an indicator variable of label −1 if x ∈ X + f• (i.e., superlevel set), and label 1 other- x as a class label of x and LSE as a binary classification problem that classifies whether each x ∈ X is in the superlevel set The active learning criterion can therefore be measured as the mutual information I(y x ; γ • X |y D , f • ) which cannot be evaluated tractably with a continuous X . So, we simplify it to the information gain on class label γ • x from evaluating f at input query x to observe y x : and the posterior entropy H(p(γ • x |y D∪{x} , f • )) of γ • x given y x is defined in a similar manner. Since γ • x is binary, our active learning criterion α BES (x, y D ) is named binary entropy search (BES). Note that BES (2) can be interpreted as the expected reduction in the uncertainty (entropy) of γ • x from evaluating f at input query x to observe y x . Though replacing γ • X with γ • x appears to be a simplification, BES demonstrates state-of-the-art performance in our experiments (Sec. 6.1). Such a simplication is also commonly adopted by existing acquisition functions for BO (e.g., (Suzuki et al. 2020;Wang and Jegelka 2017)). BES (2) can be evaluated as follows: and Ψ denotes the c.d.f. of the standard Gaussian distribution. Its derivation is shown in Appendix A. BES (3) can thus be optimized w.r.t. input query x via stochastic gradient ascent. Fig. 1a shows LSE with the threshold f • = 0 being viewed as a binary classification problem that classifies whether each x ∈ [0, 10] is in X + f• or X − f• and the level set w.r.t. f • = 0 is likely to be found on the decision boundary  (i.e., white regions on the x-axis). Fig. 1b shows large values of BES on the decision boundary that is likely to contain the level set, which is desirable. Fig. 1c shows BES using about 10 observations to explore and find roughly the level set w.r.t. f • = 0. Then, BES exploits by distributing its observations on the decision boundary (i.e., level set).
Remark 1 (Special case of BES) When the observation is noiseless (i.e., y x = f (x) or σ 2 n = 0), γ • x is fully determined by the values of f • and y x = f (x). Then, p(γ • x |y D∪{x} , f • ) is either 0 or 1 and the posterior entropy term in (2) thus becomes 0. So, BES reduces to the prior entropy term in (2) and the resulting active learning algorithm: is called entropy maximization (EM), as proposed by Low et al. (2012). EM can therefore be viewed as a special case of BES due to its noiseless observations. In other words, BES subsumes EM.

BES for Maximum Value Prediction (BES-MP) in Bayesian Optimization (BO)
A BO algorithm repeatedly selects the next input query x ∈ X for evaluating f to maximize some acquisition function based on the GP posterior belief of f given the observations y D obtained in previous iterations such that the maximizer(s) of f can be found as rapidly as possible.
Given an estimate f of the maximum value of f , the su- the threshold f can be regarded as a set of potential maximizers. So, BO can be viewed as an active learning problem that involves actively estimating the superlevel set X + f , which corresponds to an LSE problem. Therefore, we exploit our proposed BES criterion for LSE (Sec. 3) to design a new acquisition function for BO, specifically, the information gain on class label γ x (i.e., indicator variable of label −1 if x ∈ X + f , and label 1 otherwise) from evaluating f at input query x to observe y x : I(y x ; γ x |y D , f ) which can be optimized via stochastic gradient ascent by replacing γ • x and f • with γ x and f in (3), respectively. However, since the maximum value of f is unknown, we estimate it with a set F of samples of the maximum value of functions drawn from the GP posterior belief (1). These functions are drawn by applying the random Fourier feature approximation to GP (Rahimi and Recht 2008), which is widely used in existing information-theoretic acquisition functions (Hernández-Lobato, Hoffman, and Ghahramani 2014; Hoffman and Ghahramani 2015; Wang and Jegelka 2017). Then, we propose the acquisition function called BES for maximum value prediction (BES-MP) by averaging our BES criterion (for LSE) over the set F of maximum value samples: (4) At first glance, it may not seem straightforward to justify averaging BES over F in (4). To do so, we have proven in Appendix B that the average of BES over F (4) is in fact the mutual information between y x and the jointly distributed where we overload the notation f to denote a discrete uniform random variable on the support F whose distribution approximates that of the unknown maximum value of f . In the rest of this section, we will investigate the connections between BES-MP and existing acquisition functions: UCB (Srinivas et al. 2010) and MES (Wang and Jegelka 2017). Our result below reveals that UCB can, interesting, be derived from BES-MP by choosing a deterministic estimate of the maximum value of f , as proven in Appendix C: Theorem 1 (Connection to UCB) Define acquisition function of UCB as α UCB (x, y D ) µ x + βσ x (β > 0) and x UCB argmax x∈X α UCB (x, y D ). If observation y x is noiseless and the estimate of maximum value of f is chosen deterministically: f = α UCB (x UCB , y D ), then BES-MP selects the same input queries as that selected by UCB.
For noisy observation y x , though both BES-MP and MES employ a set F of samples of the maximum value of f , BES-MP differs significantly from MES in both its interpretation and model of noisy observation, as explained in the two remarks below: Remark 2 (Interpretation as information gain) BES-MP (5) can be interpreted as information gain on both the class label γ x and the threshold f ∈ F inducing the superlevel set X + f (of potential maximizers) from evaluating f at input query x to observe y x . In contrast, MES measures the information gain on maximum value from evaluating f at input query x to observe y x . BES-MP (4) is also closely related to BES (2), thus allowing our unifying information-theoretic framework for BO and LSE to be established.
Remark 3 (Model of noisy observation y x ) Another key distinction between BES-MP and MES lies in how they model p(y x |y D , f , γ x = 1). MES assumes that given f ≥ f (x), the observation y x at an input query x must be at most f (Wang and Jegelka 2017), which leads to an (upper-tail) truncated Gaussian distribution of p(y x |y D , f , γ x = 1) and its closed-form expression. However, due to noise x , y x = f (x) + x can be larger than f even though f ≥ f (x), as shown in Fig. 2. This issue can be interpreted as MES assuming to observe the noiseless f (x) when in fact, only the noisy y x is observed, which implies that MES overestimates the information gain on the maximum value from observing a noisy y x . This overestimation is significant when the noise variance σ 2 n is large relative to the posterior variance σ 2 x (1) of f (x). Such an issue also plagues the other MESbased acquisition functions such as those handling multiple objectives (Belakaria, Deshwal, and Doppa 2019;Suzuki et al. 2020) or fidelities ). On the other hand, BES-MP models p(y x |y D , f , γ x ) accurately, which may suggest an improvement to these other MES-based acquisition functions to be considered for future work.

Implicit Level Set Estimation (LSE)
Implicit LSE is about finding the superlevel set w.r.t. an unknown threshold that differs from the maximum value of f by a specified tolerance. It is motivated from the estimation of hotspots (i.e., superlevel sets) in environmental fields, which are regions of locations (i.e., inputs) whose field measurements (i.e., function values) are of at least a threshold. Since such measurements may vary throughout the year, it is desirable to define the threshold based on the (unknown) maximum value of the environmental field, which explains the term of implicit level set. For example, farmers are interested to identify the regions of their farms with high (or low) phosphorus level. Recall that LSE aims to find the superlevel set w.r.t. a known threshold while BO aims to find the maximizer(s) of the objective function, i.e., the superlevel set w.r.t. the unknown maximum value. Therefore, our LSE and BO algorithms cannot be directly applied to solve the implicit LSE problem.
A variant of an implicit LSE problem with a discrete input domain has been introduced in (Gotovos et al. 2013) where the threshold is expressed as a percentage of max x∈X f (x). However, in this paper, we prefer our above definition as it accounts meaningfully for negative function values. Note that existing works only consider problems with a discrete input domain (Gotovos et al. 2013) while our work here addresses problems with a continuous input domain such as those in our experiments.
Let α ≥ 0 be the specified tolerance. The threshold in implicit LSE is then max x∈X f (x) − α which is not known due to the unknown maximum value: max x∈X f (x). So, the implicit LSE problem is about finding the superlevel set w.r.t. max x∈X f (x) − α. It is a generalized variant of BO and LSE as it reduces to BO when α = 0 and to LSE when the maximum value of f is known.
Following the design of BES-MP in Sec. 4, one may be tempted to solve the implicit LSE problem by averaging BES over the set F α {f − α|f ∈ F } where F is a set of samples of the maximum value of f defined in Sec. 4 previously; f α ∈ F α is then an estimate of the unknown threshold in implicit LSE. Define the superlevel set x denote an indicator variable of label −1 if x ∈ X + fα , and label 1 otherwise. Similar to (4), the active learning criterion of BES-MP for implicit LSE can be written as Like (5), (6) can also be expressed as I(y x ; (γ α x , f α )|y D ) which can be interpreted as the information gain on both the class label γ α x and the threshold f α ∈ F α inducing the superlevel set X + fα from evaluating f at input query x to observe y x . 2 We can optimize (6) in the same manner as (4).
Unfortunately, the above BES-MP only actively estimates the decision boundaries between X + fα and X − fα for f α ∈ F α . Since these decision boundaries can be far from the maximizer(s) (e.g., when α is large), it is unlikely that BES-MP queries at the maximizer(s), hence yielding poor estimates f of the maximum value. For example, Fig. 3a shows that BES-MP has only 1 input query near to the maximizer of f . The poor estimates f entail poor estimates f α = f − α (i.e., dashed blue lines in Fig. 3a) and hence the poor performance of BES-MP in implicit LSE.
To improve the performance of BES-MP in implicit LSE, we consider a generalization of LSE to the k-level set estimation (k-LSE) problem (i.e., with multiple thresholds). It is an active learning problem that involves actively estimating the k level sets where the threshold of the i-th level set is represented by denote a vector of thresholds in ascending order, i.e., b i < b j if i < j. The k-LSE is equivalent to a (k + 1)-class classification problem that classifies each x ∈ X into k + 1 classes. Let γ k x ∈ {0, 1, . . . , k} denote the class label of an input x such that it is of label Similar to the design of BES, we propose an active learning criterion for k-LSE called BES k that measures the information gain on class label γ k x from evaluating f at input query x to observe y x : ) which can be expressed in a form that can be optimized via stochastic gradient ascent (Appendix D).
Implicit LSE can be viewed as a k-LSE problem such that the vector of thresholds is unknown (due to the unknown maximum value of f ). So, we can exploit our BES k criterion for k-LSE to design an active learning criterion for implicit LSE called BES 2 -MP (i.e., k = 2) by averaging BES k over a set B of estimates b = (f − α, f ) for f ∈ F : α BES 2 -MP (x, y D ) (1/|B|) b∈B I y x ; γ k x |y D , b . Similar to BES-MP (6), BES 2 -MP can also be expressed as I(y x ; (γ k x , b)|y D ) which can be interpreted as the information gain on both the class label γ k x and the threshold vector b inducing the 2 level sets from evaluating f at input query x to observe y x . 3 Fig. 3b shows that BES 2 -MP uses several input queries to determine the maximum value of f but BES-MP (Fig. 3a) does not. As a result, BES 2 -MP can estimate f − α (i.e., f α ) more accurately than BES-MP, which can be observed from Fig. 3 by comparing the dashed blue lines representing f α samples with the solid red line representing the ground truth threshold.

Remark 4 (A unifying framework)
We introduce a unifying framework for LSE, BO, and implicit LSE problems by interpreting our proposed active learning criteria or acquisition function as information gain on the class label and the threshold vector b of length k from evaluating f at input query x to observe y x . By setting k = 1, our unifying framework encompasses BES for LSE when the threshold is known (Sec. 3) and BES-MP for BO when the threshold is unknown (Sec. 4). By setting k = 2, our unifying framework encompasses BES 2 -MP for implicit LSE when the threshold vector is unknown.

Experiments and Discussion
This section empirically evaluates the performance of our proposed LSE (Sec. 6.1), BO (Sec. 6.2), and implicit LSE (Sec. 6.3) algorithms against that of state-of-the-art methods using synthetic benchmark functions, a real-world dataset, and in hyperparameter tuning of machine learning models. The code is available at https://github.com/qphong/bes-mp.

Level Set Estimation (LSE)
In this subsection, we empirically compare the performance of BES against that of the state-of-the-art EM (Low et al. 2012) (Remark 1) and straddle (STRDL) heuristic (Bryan et al. 2006) in the LSE problem. The methods of Bogunovic et al. (2016) and Gotovos et al. (2013) are demonstrated mainly on problems with a discrete input domain and hence not directly applicable to our experiments with a continuous input domain. Furthermore, STRDL is empirically shown to achieve comparable performance to these methods. So, STRDL is chosen as a direct competitor with BES while other methods (Bogunovic et al. 2016;Gotovos et al. 2013) are not empirically compared here. Since LSE is a binary classification problem (see Sec. 3) in a continuous domain X , we use the log loss as the performance metric: where X is a set of 7000 uniformly sampled inputs from X and c • x is an indicator variable of label −1 if x ∈ X + f• , and label 1 otherwise. Each experiment is repeated 30 times to account for the randomness in the observation and the optimization. Results of the log 10 of the average of the log loss are presented.
As EM assumes noiseless observations (Remark 1), our experiments are performed with observations of both small (σ 2 n = 0.0001) and large (σ 2 n = 0.09) noise variances. The GP hyperparameters are learned using maximum likelihood estimation (MLE) (Rasmussen and Williams 2006). Regarding the synthetic functions, the function values are normalized and shifted to ensure a zero prior mean.
Results for the synthetic benchmark objective functions 4 are shown in Figs. 4a to 4h. We can observe that (a) BES outperforms the other active learning criteria for both noise variance values, (b) EM outperforms STRDL when the noise variance is small (σ 2 n = 0.0001), as shown in Figs. 4a, 4e, and 4g, and (c) the performance of EM deteriorates when the noise variance is large (σ 2 n = 0.09) as it is outperformed by STRDL, as shown in Figs. 4b, 4d, and 4f. The last observation can be explained by the assumption of EM about noiseless observations (Remark 1). Fig. 4i shows the results for an LSE problem on an estimated real-world phosphorus field (Webster and Oliver 2007). The noise variance is σ 2 n = 0.025 which is learned from the dataset using MLE. It can be observed that BES outperforms EM and STRDL significantly, while EM outperforms STRDL. The standard deviation (SD) of the log loss is shown in Table 1 in Appendix E.

Bayesian Optimization (BO)
This subsection evaluates the empirical performance of BES-MP against that of the existing acquisition functions: PES, MES, UCB, and EI in optimizing synthetic benchmark functions like Michaelwicz, Hartmann-3d, and Goldstein (the negative values of functions are used), and an estimated environmental field from the phosphorus dataset (see Sec 6.1). The noise variance in the experiments with the synthetic benchmark functions is 0.01. The GP hyperparameters are learned using MLE and |F | is set to 5.
We also use BO to tune the hyperparameters of 2 machine learning models. Firstly, we train a logistic regression model on the MNIST dataset which consists of 28 × 28 grayscale images of 10 handwritten digits. The hyperparameters include the L2 regularization weight (in [10 −6 , 1]), the batch size (in [20, 500]), and the learning rate (in [10 −3 , 1]). So, the input dimension of BO is 3. The objective function is the validation accuracy on a validation set of 14K images. Secondly, we train a CNN on the CIFAR-10 dataset which consists of 50K 32×32 color images in 10 classes. The CNN includes a convolutional layer followed by a dense layer. The hyperparameters include the batch size (in [32,512]), the learning rate (in [10 −6 , 10 −2 ]) and the learning rate decay (in [10 −7 , 10 −3 ]) of the RMSprop optimization method, the convolutional filter size (in [128,256]), and the number of hidden neurons in the dense layer (in [64,256]). So, the input dimension of BO is 5. The objective function is the validation accuracy on a validation set of 10K images. We normalize the inputs in these experiments.
Following the work of Bogunovic et al. (2016), the performance metric is the regret of the best input query so far, i.e., (max x∈X f (x)) − (max x∈D f (x)). The regret is averaged over 10 random runs to account for the randomness in the stochastic optimization and the noisy observation. Fig. 5 shows that BES-MP outperforms the other acquisition functions in most of the experiments. In the other plots, BES-MP demonstrates a comparable performance to that of EI or PES. On the other hand, the performance of MES is not stable: for example, it does not perform well in Figs. 5a, 5b, and 5d. This can be explained by Remark 3. The SD of the regret is shown in Table 2 in Appendix E.

Implicit Level Set Estimation (LSE)
This subsection empirically illustrates the advantage of BES 2 -MP over BES-MP in implicit LSE problems which include several synthetic benchmark functions and an estimated phosphorus field (see Sec. 6.1). The tolerance α is specified as 0.2. The noise variance σ 2 n in the observations of the synthetic functions is 0.0001. The GP hyperparameters are optimized using MLE. The number |F | of maximum value samples is 5. Similar to Sec. 6.1, the performance metric is the log loss. Unlike (7), since the threshold is unknown, it is marginalized out in the log loss expression:  where f is marginalized: x is an indicator variable of label −1 if x ∈ X + fα , and label 1 otherwise. Each experiment is repeated 30 times. Results of the log 10 of the average of the log loss are presented. We also reduce these implicit LSE problems to LSE problems by providing the threshold (i.e., max x∈X f (x) − 0.2) to the active learning criteria for LSE: BES, EM, and STRDL, and plotting their log losses. As the threshold is known, these methods serve as baselines that should outperform implicit LSE algorithms, i.e., BES 2 -MP and BES-MP. Fig. 6 shows that BES 2 -MP outperforms BES-MP in all experiments, as expected from our discussion in Sec. 5. Besides, BES-MP does not converge in Figs. 6a and 6d as BES-MP does not gather observations to learn about the maximum value of f (Sec. 5). Regarding the baselines with known thresholds (i.e., active learning criteria for LSE: BES, EM, and STRDL), BES achieves the best performance. However, BES 2 -MP outperforms EM in Figs. 6a and 6e likely due to noisy observations. Surprisingly, even with known thresholds, STRDL is still outperformed by our BES 2 -MP and BES-MP in several experiments. It is different from the work of Gotovos et al. (2013) where baselines with known thresholds are empirically shown to outperform all methods with unknown thresholds. The SD of the log loss is shown in Table 3 in Appendix E.

Conclusion
This paper describes an information-theoretic framework for unifying the LSE, BO, and implicit LSE problems. We propose the first active learning criteria based on mutual information for LSE and implicit LSE problems, which yield the state-of-the-art empirical performance in estimating the level set of synthetic benchmark functions and an environmental field with a continuous input domain. By exploiting the relationship between LSE and BO, we design an information-theoretic acquisition function and study its connections to UCB and MES. It highlights a critical issue in modeling the noisy observation among the MESbased acquisition functions, which implies their overestimation of the information gain on the maximum value from the noisy observation. Our proposed acquisition function achieves a competitive performance in comparison with existing acquisition functions for BO in optimizing synthetic benchmark functions, an environmental field, and in hyperparameter tuning of logistic regression model and CNN. We will consider generalizing our framework to nonmyopic BO (Kharkovskii, Ling, and Low 2020;Ling, Low, and Jaillet 2016), batch BO (Daxberger and Low 2017), highdimensional BO (Hoang, Hoang, and Low 2018), and multifidelity BO (Zhang, Dai, and Low 2019) settings.
From our perspective, the societal benefits of the proposed framework outweigh its negative impact. Our LSE and implicit LSE algorithms can be used for developing methods to monitor/locate hotspots (i.e., regions where environmental field measurements exceed a threshold) in an environmental field (e.g., over lakes and farms), which has potential applications in agriculture, aquaculture, and pollution control. While some people believe that this development can have a negative impact by reducing the salary of the related jobs, the long-term benefits are more significant. For example, high-yield and low-cost agriculture can help to sustain the growing population and reduce the food price, which benefits the whole society.
BO is well-known for a wide range of applications such as automated machine learning. With the comparison between our proposed BES-MP and other information-theoretic acquisition functions, other researchers can have a better understanding of BES-MP to employ/enhance it in their own research. Furthermore, our comparison can help engineers to understand and improve existing systems implemented with MES through the clarification of its drawback, for example, by correcting the approximation in Remark 3 if the observation noise is noticeable.