Hyper-Parameter Optimization: A Review of Algorithms and Applications
  Since deep neural networks were developed, they have made huge contributions
to everyday lives. Machine learning provides more rational advice than humans
are capable of in almost every aspect of daily life. However, despite this
achievement, the design and training of neural networks are still challenging
and unpredictable procedures. To lower the technical thresholds for common
users, automated hyper-parameter optimization (HPO) has become a popular topic
in both academic and industrial areas. This paper provides a review of the most
essential topics on HPO. The first section introduces the key hyper-parameters
related to model training and structure, and discusses their importance and
methods to define the value range. Then, the research focuses on major
optimization algorithms and their applicability, covering their efficiency and
accuracy especially for deep learning networks. This study next reviews major
services and toolkits for HPO, comparing their support for state-of-the-art
searching algorithms, feasibility with major deep learning frameworks, and
extensibility for new modules designed by users. The paper concludes with
problems that exist when HPO is applied to deep learning, a comparison between
optimization algorithms, and prominent approaches for model evaluation with
limited computational resources.
