{'fraction_non_alphanumeric': 0.03320382643899572, 'fraction_numerical': 0.011100590943223743, 'mean_word_length': 4.901734104046243, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Understanding the effect of hyperparameter optimization on machine learning models for structure design problems

To relieve the computational cost of design evaluations using expensive finite element simulations, surrogate models have been widely applied in computer-aided engineering design. Machine learning algorithms (MLAs) have been implemented as surrogate models due to their capability of learning the complex interrelations between the design variables and the response from big datasets. Typically, an MLA regression model contains model parameters and hyperparameters. The model parameters are obtained by fitting the training data. Hyperparameters, which govern the model structures and the training processes, are assigned by users before training. There is a lack of systematic studies on the effect of hyperparameters on the accuracy and robustness of the surrogate model. In this work, we proposed to establish a hyperparameter optimization (HOpt) framework to deepen our understanding of the effect. Four frequently used MLAs, namely Gaussian Process Regression (GPR), Support Vector Machine (SVM), Random Forest Regression (RFR), and Artificial Neural Network (ANN), are tested on four benchmark examples. For each MLA model, the model accuracy and robustness before and after the HOpt are compared. The results show that HOpt can generally improve the performance of the MLA models in general. HOpt leads to few improvements in the MLAs accuracy and robustness for complex problems, which are featured by high-dimensional mixed-variable design space. The HOpt is recommended for the design problems with intermediate complexity. We also investigated the additional computational costs incurred by HOpt. The training cost is closely related to the MLA architecture. After HOpt, the training cost of ANN and RFR is increased more than that of the GPR and SVM. To sum up, this study benefits the selection of HOpt method for the different types of design problems based on their complexity.


Introduction
In the development of complex structures, such as a vehicle or aircraft, a large number of full-scale numerical simulations are often needed. As the supplement and partial substitute of these expensive simulations, surrogate models have been widely used in engineering design and optimization to reduce the computational cost [1][2][3]. A surrogate model is established from the design or simulation datasets through regression, to approximate the real model. It provides an efficient way of predicting the responses of new design alternatives without running additional simulations [4]. Surrogate-based methods have been applied successfully in engineering practices, for example, vehicle crashworthiness design [5], crane bridge optimization [6], transportation facility design [7] and so on. To be adapted to various engineering problems, which are characterized by a different number of design variables, degree of nonlinearities, and loading rates and so on, several surrogate models ha ve been proposed, such as polynomial response surface model, radial basis function and Kriging model [5]. However, it is difficult, if not possible, to use one surrogate modeling method to provide accurate predictions for all types of problems [8]. Therefore, much effort was made on the selection of a suitable algorithm for a specific engineering problem [8][9][10], and the ensemble and aggregation of multiple surrogate models [11][12][13][14][15] for a higher modeling accuracy.
To comply with the requirement of surrogate modeling with higher accuracy, MLAs have been applied to implement surrogate models for a wide range of problems due to their powerful learning ability and high flexibility [8,10,[16][17][18][19][20]. The traditional surrogate methods contain only model parameters that can be fitted by data. For example, the constants of the linear regression can be determined by the least square error method; polynomial response surface parameters can be determined by gradient-based methods.
Unlike the traditional surrogate modeling approaches, a typical MLA contains not only model parameters but also hyperparameters [21][22][23], which have to be assigned before model training to control the training process and model structures. They have a great impact on model flexibility, a ccuracy, and robustness [22][23][24][25].
Up to date, no systematic studies have been conducted regarding the effect of hyperparameters on the MLA-based surrogate modeling in engineering design. Most of the existing studies [26], which applying MLAs as surrogate models, either directly use the default values provided by the MLA package [27], or determine their values based on experience [28] or trial-and-error [29]. Although these surrogate models were trained with good accuracy under pre-determined loss function(s), the potential of the MLAs has not been fully exploited and the way that hyperparameters affect the surrogate modeling performance has not been clarified. To analyze the influence of hyperparameters on the prediction performance, several studies have been performed by parametric analysis in various areas, including the biomechanical analysis [30], material design [31] and mineral exploration [32]. In these studies, however, only a small portion of hyperparameters were tuned from a specific dataset, and the final values of hyperparameters were determined subjectively. Also, the effect of MLA hyperparameters has not been investigated in the area of structural design. Lacking this knowledge, it would be very difficult to fully understand and improve the surrogate modeling of MLAs in the structural design practice. Furthermore, the potential of MLAs cannot be fully exploited solely by traditional parametric studies [33]. The HOpt in structural design is therefore needed. HOpt, which adapts the optimization methods in hyperparameters tuning by taking the hyperparameters as design variables, has been considered as an effective technique to search optimal va lues of selected hyperparameters. With the help of HOpt, the great improvement of MLAs prediction accuracy has been confirmed on many public datasets [23][24][25]34]. However, no studies have been reported on HOpt in structure engineering.
In this work, by developing a multi-objective HOpt framework, the effort is made to analyze the surrogate modeling performances of four frequently used MLAs, namely, Gaussian Process Regression (GPR), Support Vector Machine (SVM), Random Forest Regression (RFR), and Artificial Neural Network (ANN). These MLAs are selected as they are widely applied in structural engineering problems, as summarized in Table 1 [13,28,29,32,. After a discussion of their accuracy, training cost, and model robustness before and after the HOpt, the hyperparameters tuning of two superior MLAs are also discussed.
The remaining parts of this work are organized as follows. Sections 2 introduces the basic algorithms of the four MLAs under the context of surrogate modeling. The HOpt framework is described in detail in Section 3 and then it is used to formulate the HOpt for the four MLAs in Section 4. Four engineering benchmark structures are selected and introduced in Section 5, where the simulation datasets are generated.
The optimization processes are completed, and the results are presented and analyzed in Section 6 to investigate the hyperparameters effect on modeling accuracy and training cost. A further discussion in Section 7 reveals the effect of HOpt on the surrogate model robustness. Besides, parametric studies on the hyperparameters of two superior MLAs, GPR and ANN, are conducted and the results are summarized and documented in the supplement file.

Gaussian Process Regression (GPR)
GPR is a non-parametric regression algorithm through Gaussian processes (GP) and takes the function as a sample in GP and can be expressed as, where, 2 N(0, ) ε σ ∼ is a random noisy variable. The prior distribution of ( ) f x is assumed to be a zeromean Gaussian distribution (in Equation 2) to simplify the modeling process since the mean value can be fitted easily. Hence, GPR only models the residual errors by The predicted * ( ) f X is also a Gaussian distribution, where K is kernel functions, such as the linear, polynomial, exponential, Gaussian and Laplacian kernels [32,60,61]. Kernels define the basic element of GPR. Thus, kernel functions and their parameters in Equation 16 are tunable.

Support Vector Machine (SVM)
Unlike GPR, SVM seeks to fit the response linearly. However, a complex structure often exhibits nonlinear responses. Therefore, SVM tries to map design variables to a new space with higher (h) dimensions (Figure 1), where a linear regression can be realized in Equation 7.
where μ(x) is the mapping function: The training objective compromises model complexity and accuracy by Equation 8. The complexity [62,63] is controlled by the first term, namely, 1 2 T ω ω . The second term manages the error by the ω ω ξ ξ (8) subject to: where C is the weight of error penalization. i ξ and * i ξ are two slack variables, introduced by Cortes and Vapnik et al. [64], to take points out of the ε -bounds back to constraints. The Lagrange method is used by introducing constraints into the objective through Lagrange multiplier α and α * [62,64] and solving a dual problem by the quadratic programming procedure with the solution in Equation 10, where the dot product, ( , ) ( ), ( ) x , defines the kernel function and the kernel functions and related parameters are tunable.

Random Forest Regression (RFR)
Random forest for regression was developed by growing a large number of tree-like data structures (termed as "trees") [65] with numerical labels [66]. The nodes on the tree split the dataset into partitions based on a splitting criterion [65]. The RFR is also a non-parametric method to use the unweighted average of all trees' outputs a s the overall output in Equation 11. Different from other methods, it can take the categorical variables directly rather than transfer them to numerical values.
where R N is the pre-defined number of trees; ( ) d D x is the output of the dth tree.
The bagging technique [66] is used in training, where a subset is sampled from the training dataset with replacement. This subset is used to train a tree and then repeat this process to generate a group of trees.
This increases the diversity of trees and improve RFR robustness [32] with the generalization error reduced [66]. The structure related parameters, e.g. the number of trees, are tuned as summarized in Table 2.

Artificial Neural Network (ANN)
The ANN is inspired by the bio-neural system with interconnected neurons. In general, an ANN contains one input layer, multiple hidden layers, and one output layer. They are fully connected by neurons in each layer. For an ANN with a single hidden layer in Figure 2(a), the sum of weighted (wij) design variables (xi) is taken as the input of jth hidden neuron. An activation function is defined to calculate the output of each neuron by Equation 12, where bj is a bias. The outputs of all h n hidden neurons are weighted as the model output in Equation 13. The weights and biases, as model parameters, are trained by data to predict new designs' responses. Several common activation functions, e.g., tanh, sigmoid, relu, and softrelu [33,67] are shown in Figure 2 By searching weights and biases, ANN training aims to find a local minimum of a pre-defined measure, using the gradient method. The optimizers, such as sgd and adagrad, [33,67], control the searching process. The optimizers, activation functions, number of hidden neurons are tunable.

The framework of multi-objective hyperparameters optimization
The framework of a multi-objective HOpt was established based on the characteristics of MLAs and design problems [24,25]. The sequential model-based optimization (SMBO) [21,68]   Using the constructed RFR surrogate model, the optimal hyperparameters are predicted and evaluated by training their MLA. The tuning process is terminated when the termination criterion, e.g., the maximum number of iterations (NT), is reached. Otherwise, this optimal design will be added to the hyperparameters dataset and then update the RFR surrogate model for the next HOpt iteration. Finally, a Pareto front can be generated.
In Figure 3, the core part is the optimum acquisition with details introduced as follows. Based on the accuracy measure responses, a number of effective search criteria can be generated as objectives to achieve the trade-off between the exploration and exploitation, for example, the individual or combination of expected improvement (EI), GP upper confidence bound, the maximum possibility of improvement, minimum conditional entropy and the lower confidence bound (LCB) [24,25,69]. In this study, the LCB is used, and it is defined as, where x hp is the vector of hyperparameters and the ˆ( ) x µ hp and ˆ( ) ).
Using LCBs as objectives, the multi-objective HOpt will be implemented by using a design space shrinking procedure recursively. Within a single iteration, a group of designs (hyperparameters combinations) will be sampled randomly in the design space of this iteration and their response is predicted by the RFR surrogate model. The best alternative will be selected and a smaller subspace will be generated around this point. The construction of the new subspace needs to consider both exploration and exploitation. A small subspace improves convergence speed but may lose the optimum. Similarly, a larger subspace may seize the optimum but causes a low convergence speed. In this study, the (i+1)th subspace is defined as represent the hyperparameters space of the ith iteration and its sizes and * _ hp i X is the optimum of ith iteration. ∩ a nd p r are the intersection operation of two subspaces and a proportion to define the expanded space size centered on the best design of the current iteration. In this smaller subspace, a new round of sampling and evaluation process will be implemented until the termination criterion is reached. The total number of the space shrinking iteration is limited to it n .
After the iterations are terminated, an optimum design can be obtained.
To fully explore the design space and improve robustness, this optimization process will be repeated re n times. Then, a group of designs is generated by taking the Pareto front from each of the re n optimization processes, where each process is implemented with it n iterations for space shrinking as described above. The overall Pareto front will be obtained from this group of designs and used to determine the hyperparameter values of corresponding MLA models.

Implementation of hyperparameters optimization
Based on available datasets, the hyperparameters can be optimized by the above framework. Before this, the hyperparameters domain, measures, and formulation should be implemented.

Design synthesis and post-processing
The dataset used for MLA training can often be generated by the design of experiment (DoE). Many algorithms are available to create DoE, e.g. Pseudo-Monte Carlo Sampling (PMCS), Latin Hypercube Sampling (LHS), and Orthogonal Array Sampling (OAS) [70]. In this study, the uniform LHS is used since it could fill the design space uniformly to fully explore the design space. It is also noted that design variables may have different orders of magnitudes. This would outweigh features with a larger value over these with smaller value [73]. Therefore, the min-max normalization is used to scale the values of design variables into the same range [0, 1] by where, for a specific variable, the NV is the normalized value, v is the variable value; max v and min v a re the maximum and minimum value of this variable, respectively.

Hyperparameters domains
For each MLA, hyperparameters are identified for optimization considering their significant impact on model accuracy. Their initial values and ranges are determined based on the experience and previous studies as presented in Table 2. For GPR, the kernels and related parameters are tuned as the bold where, Besides kernel parameters, the ε and the weight factor C in Equations 8 and 9 are also tuned for SVM.
A smaller ε will add more items to the latter term in Equation 8 and larger C will overweight error terms, which may cause the highly complex objective function. These may reach a highly accurate SVM model, but take the risk of overfitting since the relative underweighting of the former term in Equation 8 may cause a too complex model and so versa. The HOpt aims to gain the trade-off between model complexity and accuracy.
In RFR, the number of regression trees is a key factor for a ccuracy, but it is also critical to the computational cost. With more trees included, a higher computational cost will be caused. Meanwhile, the tree-related parameters a re also optimized, i.e. the number of randomly selected features for each split (NF), minimum terminal node size (Min TS) and maximum numbers of terminal nodes (Max TN). updates weights in the form of where t w and t w ∇ are the weight matric and gradient of the current epoch, respectively, and 1 t w − ∆ is the step size of last epoch. ζ and ξ are the learning rate and momentum, respectively. A large learning rate accelerates the convergence speed but may lose the optimum while the too small value causes the high convergence cost. For other optimizers, adaptive learning rates are also used to speed up the learning process [67]. In this study, ANN training epochs were limited to 2,000, which is proven to be enough for training convergence by an additional test.

Measures of accuracy
During the surrogate modeling, a number of statistical indices, such as the root mean square error (RMSE), maximum absolute error (MXAE), mean absolute error, correlation, and decision coefficient, are frequently used as measures of accuracy [20,74,75]. In this study, the RMSE (Equation 18) and MXAE (Equation 19) are used as two HOpt objective measures [74] as the global and local accuracy measures, respectively.
is the error of the predicted response relative to its real value of the ith design (x i ); N is the dataset size. Also, the training computational time (T) of MLAs is evaluated. This helps understand the effect of HOpt on the cost and make a decision on the selection of a suitable MLA in terms of computational cost.

Benchmark problems for the study of MLAs
In this study, four representative engineering structures are taken as examples to demonstrate the effect of hyperparameters by the HOpt. In these four problems, the variation of geometry (bar, sheet, and block), loading (static vs. dynamic) and boundary conditions (fully constrained and contact), as well as deformation modes (small vs. large deformation) are all considered. Hence, the methods and results associated with these case studies can be easily extended to a wide range of structures.

Structures subject to static loading
Under the static loading condition, two typical structures, i.e. a ten-bar planer truss (TbPT) [33] and a torque arm (TqA) [76][77][78][79], are introduced in Figure  To design the TqA structure, geometric constraints must be considered. The constant thickness of this structure is 3 mm and the distance between two circle centers is 420 mm. The design variables and their ranges are listed in Table 3. The left circle is fully constrained and loads are added to the right circle as shown in Figure 4(b). To avoid structure failure, the stress is limited under 800 MPa and the total mass is used as the objective.

Structures subject to dynamic loading
Under the dynamic loading conditions, two vehicular components are studied, i.e. the thin-walled Sshaped beam (ShB) discussed in our previous studies [80,81] and thin-walled octagonal multi-cell tube (OMcT) reported in Bai et al [82]. Their geometry parameterization and FE models are illustrated in Figure   5. As critical energy-absorbing parts on the passenger car, these structures can sustain large plastic deformation and dissipate a large amount of kinetic energy due to impact.
In Figure 5(a), the shape of ShB is fully described by 7 design variables and its total length is 1,000 mm. Figure 5(a) also shows the FE model of ShB, which is subjected to the frontal impact at 10 m/s. The specific energy absorption (SEA) is set as the design objective in Equation 21, where M is the structural mass; F(x) and d are the impact force-displacement history and total deflection, respectively.
is the averaged crushing force and max F is the peak crushing force. The detailed material parameters, loading, and boundary conditions can be seen in [80,81]. These two models were validated in these studies and the details are not repeated here. In the current study, the impact speed of OMcT is increased to 50 km/h to represent the loading condition defined in NCAP standard (National Highway Traffic Safety Administration: https://www.nhtsa.gov/laws-regulations). The four models were simulated by the FE simulation using implicit (TbPT and TqA) and explicit (ShB and OMcT) solvers. For each structure, 1000 design cases are generated for MLA learning.

Pareto front of hyperparameters
The Pareto fronts are generated through the SMBO as shown the supplement file. Although the optimal hyperparameters may vary for different datasets, the values of hyperparameters in Pareto front would be close for the da tasets with similar sample sizes and design space dimensions [24,83]. In other words, the optimal hyperparameter values obtained from this study can be applied to the other similar design problems without significant change or used as the basis for optimization [23]. For each Pareto front, one hyperparameters group is selected randomly as summarized in Table 4, where corresponding mean loss values of five cross-validations are also included.

Effect of hyperparameters optimization
By using the values in Table 4, sixteen models are trained with 5-folds cross-validation. The distributions of RMSE a nd MXAE among 5-folds cross-validation are presented by boxplots in Figure 6 and 7, respectively. The results before (trained using initial values in Table 2 Table 4 and Figure 8, a fter HOpt, the ANN trained by TbPT is reduced slightly, which is caused by the larger learning rate, which accelerates the training convergence. however, the other structural cases presented lower convergence speed although they have larger learning rates, which are In summary, compared with GPR and SVM, the training time of ANN and RFR is more sensitive to hyperparameter values. HOpt improved the ANN performance most significantly, which is followed by the GPR. Therefore, it is recommended to conduct HOpt for ANN if enough computational sources are available. Otherwise, the GPR is recommended for applying the HOpt. Besides, in the HOpt for SVM, small ε-bounds or large error penalization C should be avoided, because they will cause a too complex training objective function and then unexpected singularities and a high convergence cost.

Discussion
In this section, an analysis is carried out on the relationship between the performance of HOpt method to improve the MLA modeling performance and characteristics of the design problems, as summarized in Table 5. The characteristics include the structure type, the types and number of design variables, load type, and the design domain continuity, which performances are mainly RMSE, MXAE and computational time.
Previous studies have demonstrated that the dimensionality and nonlinearity of design problems, which partially depends on the structure and load types, have a strong impact on the surrogate model a ccuracy and robustness [19,84]. In addition to structure and load types, the dimensionality of the design space, continuity of the feasible domain, and the design variable type also influence the complexity of the problem. Due to the design constraints, the feasible domains of TbPT and TqA are broken into discontinuous subdomains. The nonlinear boundaries of discontinuous feasible domains pose an additional level of challenge to MLA model training. Furthermore, mixed-variable problems are more challenging for MLA models.
In Table 5 Table 5, the cases with no improvement are denoted with red and the increased training time is denoted with blue. With all the information on problem complexity and MLA performances, we discuss the effect of HOpt in the following three aspects.
• Effect of HOpt on MLA model accuracy HOpt may not always improve the model accuracy for problems with characteristics similar to the TqA. Compared with the other three structures, the HOpt takes less and unstable improvement on the TqA.
The SVM and ANN performance for TqA are even deteriorated slightly after HOpt. This was caused the mixed-variable design space and we also notice that this case study has the highest dimensionality in all cases. More optimization cycles, as described in Section 3, may be helpful to gain more improvement.
Also, the discontinuous design domain induced singular points on the boundaries, which will cause difficulties in training surrogate models. For example, it may cause the singularity in the covariance matrix of GPR as shown in Equation 6.  [85,86]. To summarize based on the results of present study, HOpt may lead to deteriorated accuracy and robustness for design problems with the relatively high complexity, but it can improve the modeling performance with less design variables. However, further reduction of the dimension will eliminate the need of HOpt. We recommend GPR as a good start for complex problems due to its low training cost and relatively high accuracy. It is also noticed that HOpt leads to insignificant improvement in simple design problems.

Conclusions
In this study, the effects of hyperparameters on MLA surrogate models, namely GPR, SVM, RFR, and

B.1 Hyperparameters effect on GPR
The only hyperparameters in GPR are kernel-related parameters, that is, the four kernel functions in Equation 16. The GPR performance with different Sigma (σ) is plotted in Figure B.1 (a) and (b) for radial basis and Laplacian kernel, respectively. The low σ (< 2) level takes a critical effect. The best GPR performance σ are 1 and 0 for radial basis and laplacian kernels, respectively.  Table 4 shows that the optimal degrees would be 2 or 3, so low degree levels in Figure B.2(a) is amplified to see the details in Figure B.2(b). The 2nd or 3rd order polynomial also presented a high model accuracy. The influence of scale and offset was increased with the degree, which is indicated by the larger standard errors. Furthermore, for scale and offset, their absolute values should be larger than 1 since this will significantly reduce the RMSE error as shown in Figure B.3. If their absolute values are larger than 1, it generally works well to generate an accurate GPR. The hyperbolic tangent kernel also contains the parameters of scale and offset. Figure B.4 shows their influence on GPR performance, which indicates a similar trend in four cases.
The scale and offset are suggested to be around 1 and 10, respectively, for an accurate GPR. In summary, the RMSE accuracy of sixteen models trained with the best kernel hyperparameters values is presented in Figure B.5. The polynomial kernel shows the best performance on GPR accuracy. Meanwhile, the radial basis kernel demonstrates a large scatter and less unstable performance.

B.2 Hyperparameters effect on ANN
To explore the hyperparameters effect of ANN, a basic one-hidden-layer ANN is constructed with the following hyperparameter values based on the optimization: batch size=100, learning rate=0.1, hidden neurons=20, optimizer=sgd, activation function = tanh, momentum=0.9. Also, only the ShB dataset is used considering the high computational cost of ANN training and the similar trend of hyperparameters effect for each dataset. All models in this part are trained with 10,000 epochs.
The influence of batch size and the learning rate is studied and plotted with log scale contour in FigureB.6.
Larger learning rates and smaller batch sizes speed up the convergence, but too large learning rates could lead to unstable training and may lose the optimum. However, too small mini-batch size could increase the communication cost, i.e. the computational cost to synchronize the shared variables (gradient or model parameters, etc.) between different mini-batches, although it could increase the convergence speed [1].
Base on Figure B.6, a learning rate of 0.1 is determined as the optimum. Under this condition, the larger mini-batch size is preferred only if the same accuracy can be achieved to reduce communication costs [1]. Therefore, the mini-batch size is set as 100 for the current dataset size. Considering the training dataset size (i.e. 1,000) in this part, 20 hidden neurons are used since it results in slightly better accuracy than the 5. In Figure B.7(b), among four optimizers, adagrad results in the highest accuracy, stability, and convergence speed. Although the sgd shows a similar accuracy, it is limited by the low convergence speed caused by its complicated training process [2]. Hence, adagrad would be a good choice.
In Figure B.7(c), relu shows low RMSE accuracy due to its linear behavior in Figure 2. After the transmission through one hidden layer with the linear activation function, the output of ANN will be still linear, which is not suitable for the problems with a non-linear response, such as ShB. Except for relu, the other three activation functions are all nonlinear and generated similar accuracy levels. The tanh reaches a quick and stable convergence due to its large value variation in the range of [0 1]. Therefore, tanh is used for the nonlinear response prediction.
As discussed, a multi-layered ANN possesses a higher ability to learn high nonlinear responses. To explore the multi-layers effect on ANN training, another four multi-layered models are built with optimal hyperparameters identified earlier. As shown in Figure B.7(d), 7-20-1 presents the best performance. The model 7-10-5-1 suggests a good but slightly lower accuracy since its number of weights (i.e. 125) is also less than a quarter of the training dataset size. The other three ANN structures greatly increase the number of weights and lead to unstable convergence process and deteriorated accuracy, which indicates insufficient training. This means compared with increasing the number of layers, the number of weights is more important to fully train an accurate ANN. Supplemental Material Section C: