GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box
  Optimization Algorithms
  Black-box optimization is essential for tuning complex machine learning
algorithms which are easier to experiment with than to understand. In this
paper, we show that a simple ensemble of black-box optimization algorithms can
outperform any single one of them. However, searching for such an optimal
ensemble requires a large number of experiments. We propose a
Multi-GPU-optimized framework to accelerate a brute force search for the
optimal ensemble of black-box optimization algorithms by running many
experiments in parallel. The lightweight optimizations are performed by CPU
while expensive model training and evaluations are assigned to GPUs. We
evaluate 15 optimizers by training 2.7 million models and running 541,440
optimizations. On a DGX-1, the search time is reduced from more than 10 days on
two 20-core CPUs to less than 24 hours on 8-GPUs. With the optimal ensemble
found by GPU-accelerated exhaustive search, we won the 2nd place of NeurIPS
2020 black-box optimization challenge.
