Scalable3-BO: Big Data meets HPC - A scalable asynchronous parallel
  high-dimensional Bayesian optimization framework on supercomputers
  Bayesian optimization (BO) is a flexible and powerful framework that is
suitable for computationally expensive simulation-based applications and
guarantees statistical convergence to the global optimum. While remaining as
one of the most popular optimization methods, its capability is hindered by the
size of data, the dimensionality of the considered problem, and the nature of
sequential optimization. These scalability issues are intertwined with each
other and must be tackled simultaneously. In this work, we propose the
Scalable$^3$-BO framework, which employs sparse GP as the underlying surrogate
model to scope with Big Data and is equipped with a random embedding to
efficiently optimize high-dimensional problems with low effective
dimensionality. The Scalable$^3$-BO framework is further leveraged with
asynchronous parallelization feature, which fully exploits the computational
resource on HPC within a computational budget. As a result, the proposed
Scalable$^3$-BO framework is scalable in three independent perspectives: with
respect to data size, dimensionality, and computational resource on HPC. The
goal of this work is to push the frontiers of BO beyond its well-known
scalability issues and minimize the wall-clock waiting time for optimizing
high-dimensional computationally expensive applications. We demonstrate the
capability of Scalable$^3$-BO with 1 million data points, 10,000-dimensional
problems, with 20 concurrent workers in an HPC environment.
