Understanding the effect of hyperparameter optimization on machine
  learning models for structure design problems
  To relieve the computational cost of design evaluations using expensive
finite element simulations, surrogate models have been widely applied in
computer-aided engineering design. Machine learning algorithms (MLAs) have been
implemented as surrogate models due to their capability of learning the complex
interrelations between the design variables and the response from big datasets.
Typically, an MLA regression model contains model parameters and
hyperparameters. The model parameters are obtained by fitting the training
data. Hyperparameters, which govern the model structures and the training
processes, are assigned by users before training. There is a lack of systematic
studies on the effect of hyperparameters on the accuracy and robustness of the
surrogate model. In this work, we proposed to establish a hyperparameter
optimization (HOpt) framework to deepen our understanding of the effect. Four
frequently used MLAs, namely Gaussian Process Regression (GPR), Support Vector
Machine (SVM), Random Forest Regression (RFR), and Artificial Neural Network
(ANN), are tested on four benchmark examples. For each MLA model, the model
accuracy and robustness before and after the HOpt are compared. The results
show that HOpt can generally improve the performance of the MLA models in
general. HOpt leads to few improvements in the MLAs accuracy and robustness for
complex problems, which are featured by high-dimensional mixed-variable design
space. The HOpt is recommended for the design problems with intermediate
complexity. We also investigated the additional computational costs incurred by
HOpt. The training cost is closely related to the MLA architecture. After HOpt,
the training cost of ANN and RFR is increased more than that of the GPR and
SVM. To sum up, this study benefits the selection of HOpt method for the
different types of design problems based on their complexity.
