{'fraction_non_alphanumeric': 0.03537772687041183, 'fraction_numerical': 0.010543031186546571, 'mean_word_length': 4.753332334880935, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exponentially Weighted l_2 Regularization Strategy in Constructing Reinforced Second-order Fuzzy Rule-based Model

In the conventional Takagi-Sugeno-Kang (TSK)-type fuzzy models, constant or linear functions are usually utilized as the consequent parts of the fuzzy rules, but they cannot effectively describe the behavior within local regions defined by the antecedent parts. In this article, a theoretical and practical design methodology is developed to address this problem. First, the information granulation (Fuzzy C-Means) method is applied to capture the structure in the data and split the input space into subspaces, as well as form the antecedent parts. Second, the quadratic polynomials (QPs) are employed as the consequent parts. Compared with constant and linear functions, QPs can describe the input-output behavior within the local regions (subspaces) by refining the relationship between input and output variables. However, although QP can improve the approximation ability of the model, it could lead to the deterioration of the prediction ability of the model (e.g., overfitting). To handle this issue, we introduce an exponential weight approach inspired by the weight function theory encountered in harmonic analysis. More specifically, we adopt the exponential functions as the targeted penalty terms, which are equipped with l2 regularization (l2) (i.e., exponential weighted l2, ewl_2) to match the proposed reinforced second-order fuzzy rule-based model (RSFRM) properly. The advantage of el 2 compared to ordinary l2 lies in separately identifying and penalizing different types of polynomial terms in the coefficient estimation, and its results not only alleviate the overfitting and prevent the deterioration of generalization ability but also effectively release the prediction potential of the model.


Introduction
Fuzzy rule-based model (also known as fuzzy system or fuzzy model) is an advanced modeling architecture based on fuzzy logic, if-then rules, and fuzzy reasoning [1]. A fuzzy rule-based model exploits rules as a means of knowledge representation to formalize the knowledge existed in the model. Moreover, due to its modular architecture, well-developed design methodologies and practices, as well as its advantages in interpretability, it has been used in a wide spectrum of realms such as fuzzy control, pattern analysis, fuzzy decision, time series prediction, robotics, etc. [2][3][4][5][6].
There are two well-known fuzzy rule-based models in dealing with regression issues. The first is the Mamdani fuzzy model, in which the condition and the conclusion parts are represented by fuzzy sets [7]. The second type is the Takagi-Sugeno-Kang (TSK) fuzzy model [8], where the premise is represented by a fuzzy set, and the conclusion is a function. In the case of using the standard pre-defuzzified operation, the Mamdani model can be regarded as a TSK model with constant conclusions. There are lots of different variants of fuzzy rule-based models. In early 2001, Oh and Pedrycz proposed a hybrid neuro-fuzzy model of TSK-type model. The model first uses the clustering method to determine the initial values of the apices of the membership functions employed in the model, and then optimizes and adjusts the positions of the apices of the membership functions through evolutionary algorithm and improved complex algorithm [9]. Besides, the TSK model is also commonly used in classification problems. The TSK model is functionally equivalent to the corresponding radial basis neural network (RBFNN) under certain minor constraints [10]. A concise clustering-based fuzzy model is constructed by combining clustering method and RBFNN, where clustering is used as a learning approach to determine the arguments of the receptive field [11][12][13]. The output of the receptive field is directly used as the value of the matching degree (firing strength) of the fuzzy model. The final output of the fuzzy model needs to be the weighted average sum of the output of each rule in the model, in which the weighted average can guarantee the semantic integrity of the partition function (the sum of the matching degree is one). These hybrid approaches focus on identifying the parameters of the RBFNN (such as the width and center of the Gaussian Function) without any structural modification. After that, a modified method of replacing the nodes in the activation layer with clusters has been proposed, which directly treats the partition grade from the partition function as the output of the activation layer [14,15,22].
In [16], a novel design method of fuzzy model based on clustering technique is proposed, which enhances the performance of the model by identifying and refining the rules with maximum errors. Kim et al. redesigned the conventional neural network structure based on fuzzy clustering and constructed a context layer by using the output space [17]. The experimental results show that the fuzzy model with context layer can improve the quality of the model. In [18], a TSK-type fuzzy classifier constituted of multiple zero-order (weights between the hidden layer and the output layer are constant) single hidden layer feedforward neural networks (SLFNs) is proposed, which aims to decompose complex problems into several simple problems, reduce memory requirements and computational overhead, as well as maintain classification performance.
Generally, in the conventional TSK-type fuzzy models, constant or linear functions are served as the consequent parts of fuzzy rules (i.e., zero-order TSK type or first-order TSK type models). Also, in some extended TSK-type fuzzy models, the conclusion part is also simplified as much as possible (e.g., the zero-order SLFN is as the consequent part of the rule) [18]. However, some high-order polynomials (e.g., quadratic polynomials) are rarely considered. The reason is that although high-order polynomials can improve the approximation ability of the model, they can also easily lead to the deterioration of the prediction ability of the model (e.g., overfitting). To cope with this issue and boost the generalization ability of the model, we propose a reinforced second-order fuzzy rule-based model, which is implemented with the help of fuzzy clustering (viz., FCM) partition and quadratic polynomial (QP) as well as exponential weighted 2 regularization. To be precise, we are looking for the suitable weight function in harmonic analysis theory. In harmonic analysis, there are three weight function classes corresponding to Hardy operator [19], Hardy-Littlewood maximal (HLM) operator [20] and one-sided HLM operator [21] in weighted 2 -norm, which is the continuous version of weighted 2 -norm. The rule is that the larger operator matches the smaller weight function class. We will adopt the exponential weights, which matches one-sided HLM, as the target penalty terms. The essential reason is that exponential function (such as ) is rapidly increasing, which can control the high-order terms in polynomials effectively.
The essential features and novelty of our work can be enumerated as follows. First, a novel reinforced second-order fuzzy rule-based model is proposed based on FCM partition and quadratic polynomial (QP) as well as exponential weighted 2 regularization ( -2 ) to deal with regression problems. Second, inspired by the weight function theory in harmonic analysis, an exponential weighted 2 regularization is designed to mitigate overfitting elaborately. Different from ordinary 2 , -2 can identify and penalize the terms of different types of polynomials in coefficient estimation. Third, QP and -2 are used in collaboration, and its result can effectively improve the prediction accuracy and stability of the model.
In the sequel, the main contribution of this study is that the effective strategy through exponential weighted 2 regularization ( -2 ) combined with FCM partition and QP leads to better prediction accuracy as well as sound interpretability in constructing fuzzy rule-based model based on a series of experimental comparative studies.
The paper is arranged in the following sections. First, Section 2 presents the architecture of the reinforced second-order fuzzy rule-based model. The learning mechanism with exponential weighted 2 regularization is discussed in Section 3. The design steps of RSFRM are reported in Section 4. The effectiveness of RSFRM is verified in Section 5. Section 6 concludes and outlines future work.

Architecture of reinforced second-order fuzzy rule-based model 2.1. Takagi-Sugeno-Kang fuzzy rule-based model
Takagi-Sugeno-Kang (TSK) fuzzy model (also known as TS or Sugeno fuzzy model) is a classic fuzzy rule-based model (refer to Fig. 1(a)), which aim to devise a systematic fashion to generating fuzzy rules from a finite set of input-out data pairs [8,10]. A representative form of fuzzy rule in TSK fuzzy model is: 껐Ꭹ is and … is antecedent part where , 2 ,…, represent the linguistic fuzzy terms in the antecedent part, and ( ) denotes the crisp function in the consequent part, =[ , 2 ,…, ] T stands for the input vector. Generally, ( ) is the zero-order or first-order polynomial of the input variables. TSK fuzzy model can be expressed as an equivalent feedforward neural network, namely the fuzzy neural network (FNN), as displayed in Fig. 1(b) [10,17,18]. In Fig. 1(b), the layer stands for the input layer, " " node in the 2 layer denotes the linguistic fuzzy term denoted by membership function, the output of " " node in the layer represents matching degree ( ) and which can be calculate as: 2 2 (2) where denotes the -norm operator (e.g., minimum and product are commonly used here). indicates the linguistic fuzzy term of the input variable to the rule. The " " node in the layer represents the normalization operation. The node of layer 5 yield local output (model) by multiplying the consequent part and their associated antecedent part. The overall output of FNN is a combination of all the local models: where denotes the th output of the model, c represents the number of fuzzy rules, ( ) indicates the normalized matching degree of the th data to the th rule.

Architecture of the proposed second-order fuzzy rule-based model
The essence of the fuzzy rule-based model is embodied in the idea of "divide and conquer". That is, the antecedent parts of the fuzzy rule define the local regions (models), and the consequent parts describe the behavior of the region through different components. Therefore, when exploiting the model to some local regions, these regions need to be guardedly delimited. An information granulation (Fuzzy C-Means, FCM) method is utilized to analyze and reveal data distribution on the input space and form local regions in this study.
The local regions of the input space are defined by some linguistic fuzzy terms (fuzzy sets) generated by FCM clustering algorithms. Given a finite set input variables , , the loss function of FCM is as follows: where indicates the number of clusters, as well as the number of rules of the proposed model. stands for the prototype. represents the Euclidean Metric between the data and the prototype. denotes the fuzzification coefficient is used to control the specificity of the underlying linguistic information (the typical value of is 2).
represents the membership grade of the pattern in the cluster (fuzzy set). Moreover, needs to satisfy the following restrictions: By minimizing the loss function, we obtain and : The values of and are updated by the FCM algorithm through iteration (6) and (7). The proposed model implements a modular fuzzy architecture through three parts (antecedent, consequent, and inference), as displayed in Fig. 2(a). The antecedent and consequent parts are related to the constitution of fuzzy rules and their subsequent analysis. The inference part involves the aggregation of multiple fuzzy rules. Unlike the conventional TSK-type fuzzy model (refer to Fig.  1(b)), the structure of the proposed model is more compact and concise. The second and third layers in Fig. 1(b) are compressed into one layer (refer to Fig. 2(b)) (that is, the output of the partition grade is directly used as the matching degree). In addition, since the partition grade needs to satisfy the constraint conditions (Eq. (5)), the fourth layer in Fig. 1 (b) is omitted. Fig. 2 illustrates the modular architecture and the Equivalent FNN architecture of the reinforced second-order fuzzy rule-based model (RSFRM).
In the conventional TSK-type fuzzy model, zero-order or first-order polynomials (viz., constant or linear function) are generally selected as the consequents of fuzzy rules, and high-order polynomials (e.g., quadratic polynomials) are rarely considered. In fact, quadratic polynomials ( ( ) ) have a sound description and interpretation of the local regions (models) divided by the antecedents, but they are prone to overfitting.
where , , represent the coefficients of polynomial, denote the amount of input variables.
In order to cope with this problem and enhance the prediction accuracy of RSFRM, exponential weighted 2 regularization and quadratic polynomial are used in collaboration. The output of RSFRM can be rewritten as follows: where stands for the th output of the model, c stands for the number of fuzzy rules, represents the partition grade.

Learning mechanism with exponential weighted regularization 3.1. Least square error estimation for the proposed model
The proposed RSFRM uses the Least Square Error (LSE) estimation to calculate the coefficients of the consequent part (polynomial) of the fuzzy rule. The objective function of LSE is to minimize the sum of the squared error of the observed (original) output and the model output, as in Eq. (10): where denotes the number of data patterns, and represent the real and model output, respectively. ( ) denotes the consequent part of the rule, indicates the partition grade of the pattern to the prototype. By taking the derivative of objective function (10), the general expression of the coefficient vector can be obtained: where the specific forms of A 2) , and G are as follows (illustrated by the example of quadratic polynomial): ( 2)

Exponential weighted regularization
Theoretically, as the complexity of the model increases, the ability of the model to approximate the data becomes better, and the predictive performance of the model increases. However, the prediction performance of the model actually decreases as the model complexity increases excessively (i.e., overfitting). For the proposed fuzzy model, its complexity can be considered from two aspects. (a) The number of fuzzy rules. The fuzzy rules are mainly determined by the partition of the input space, which is a pivotal indicator to assess the complexity of the neuro-fuzzy model [23][24]. Also, the number of fuzzy rules is hyperparameter and needs to be specified in advance. (b) The type of consequent part of the fuzzy rules. The consequent part describes the behavior of the local region through different components (e.g., crisp functions) [10]. Although the more complex components can describe the local region in detail, the risk of overfitting faced by the model is higher. The consequent part of the rules of the proposed model is constructed by polynomial functions. Under the premise that the order and variables of the polynomial are determined, only its coefficients can affect the performance and stability of the model. The simpler the coefficient, the more stable the model becomes. In the simplest case, all coefficients are directly set to zero, and the output of the model is always zero. The model is simple, but it is meaningless because it cannot be effectively predicted.
As mentioned before, in the proposed model, LSE is used to minimize the defined objective function (i.e., Eq. (10)) to compute the coefficients of the consequent parts of the fuzzy rules. In the process of coefficient estimation, multiple factors including overtraining could lead to the estimation of unstable coefficients with large deviations between the coefficients, resulting in a poor generalization ability of the model. Generally, we consider using 2 regularization ( 2 ) to improve the weakness of LSE [25][26][27]. 2 is a technique for analyzing and alleviating overfitting of multivariate regression. This approach helps reduce the deviation between the coefficients and prevents the generalization ability from deteriorating. By adding the 2 regularization parameter (penalty term) to the objective function applied in the current fuzzy model, the 2 regularization can be easily applied, as shown below where denotes the number of data, and indicate the real and model output, respectively. stands for the penalty term, stands for the number of fuzzy rules, denotes the number of input variables, and are the coefficients of the quadratic polynomial. We can obtain the extended coefficient vector expression by taking the partial derivative of (14). Compared with the general estimation method, the regularized estimation method adds a positive number on the diagonal of the design matrix ( ) to make the matrix nonsingular.
However, the ordinary 2 penalizes all coefficients uniformly [26][27], that is, adding the same penalty factor to each entry of the diagonal of the design matrix. In this way, it may be unfair to treat all entries on the diagonal of the design matrix in the same way. It does not consider the different impacts of different types of polynomial terms on the performance of the fuzzy model.
As shown in Fig. 3, the diagonal of the design matrix contains three types of matrix entries of different orders, such as zero-order entries, quadratic entries, and biquadratic entries, composed of different polynomial terms and their corresponding matching degrees. Obviously, these three types of entries have different effects on the coefficient estimation as well as the performance of the model. In the consequent parts of the fuzzy rules, the higher-order terms of the polynomials can describe the behavior within the local region defined by the antecedent parts of fuzzy rules in more detail, which help the model to exhibit a stronger approximation ability, but exposes the model to a higher risk of overfitting. Therefore, in the process of regularization, it is unreasonable to treat the terms of different types (orders) in polynomial as equivalent, and use the same penalty terms to punish uniformly the coefficients of terms of different types. The unified penalty limits the ability of consequent parts of fuzzy rules composed of polynomial terms of different orders to properly describe (represent) the behavior within local regions. Inspired by the weight function theory of harmonic analysis, we propose exponential weighted 2 regularization ( -2 ) to achieve a more precise "penalty" (or "targeted penalty"). More specifically, different penalty terms are selected according to polynomial terms of different types, so that the coefficients of polynomial terms of different types can be adjusted (punished) more fairly and reasonably. Ordinary 2 can be regarded as a critical (special) form of -2 , in which all the penalty terms of -2 are equal. Compared with the ordinary 2 , -2 can more accurately slow down the excessive description of behavior with local regions by polynomials with high-order terms through different penalty terms, which can promote the consequent parts (denoted by polynomials) can more reasonably represent the behavior within local regions during the training phase and effectively predict the unknown behavior within local regions during the test phase. In other words, -2 can provide opportunities to improve generalization ability by establishing a effective trade-off between the training and testing of the model, and reasonable release the representation (description) ability of high-order polynomials within local regions, which not only alleviates the overfitting of the model caused by overtraining and other factors, but also effectively boosts the prediction performance of the model. 9 / 22 The objective function considering -2 can be augmented as: Here, , , , , , and (   2   ) have the same meaning as in (14).
Eq. (16) can be represented in the form of matrix, as follows: where 2 is a diagonal matrix containing , 2 , three types of entries, ( 2 2) 2. We take the derivative of 2 : Let Eq. (16) be equal to , the analytic solution to the polynomial coefficients is expressed as:

Design procedure of the reinforced second-order fuzzy rule-based model
Overall, the design procedure of the reinforced second-order fuzzy rule-based model includes the following steps:

[Step 1] Form training and test datasets.
Divide the given dataset into two blocks, namely the training and test datasets. The training data is used to establish the model, and the remaining data is utilized to estimate the predictive performance of the model. Information granulation (e.g., Fuzzy C-Means) technique is utilized to analyze and reveal the data distribution of the input space to form the local regions. At the same time, the matching degree (which is denoted by partition grade) of each fuzzy rule in the corresponding local region are calculated by using (6) and (7).

[Step 3] Estimate the coefficients of the consequent part in each fuzzy rule.
Coefficients of the consequent part in each fuzzy rule are computed by LSE estimation using (19). Here, LSE and -2 are used together. Before using, we need to set the parameters of -2 in advance.
[Step 4] Compute the predicted output of the proposed RSFRM by using (9).

Experimental studies 5.1. Experimental setup
We use training error and test error to represent the performance of each model (including control and comparative models) on training data and test data. The performance of the model is mainly quantified as Root Mean Square Error (RMSE) and Mean Square Error (MSE), as in Eq. (20): The basic comparative models of the experiment mainly include the zero-order fuzzy rule-based model (zero-order FRM), first-order FRM, second-order FRM and second-order FRM with 2 . The order indicates the types of the conclusion parts of the fuzzy rules such as constants, linear functions and quadratic polynomials. All experiments are based on -fold cross-validation, and each dataset is split into chunks, of which one of chunks is used for testing, and the rest is used for training. This process is repeated times. is set to 5 in this section. The experimental parameter setting is summarized in Table 1, which is determined via -fold cross-validation method. The reason for choosing these specific values is that we need to consider the possibility of studying the performance of RSFRM in a variety of scenarios. subject to 껐 껐 2 껐 10 -8 , 10 -6 , 10 -4 , 10 -2 , 10 -1 , 10 0 , 10 1 , 10 2 , 10 3

Specific evaluations of the proposed model 5.2.1. Synthetic data
We consider a nonlinear function containing two variables, whose mathematical description is as follows: ( ) ( 2 ) ͳ9( ͳ 5+e 2 ( ( 2 ͳ6) 2 ) ( 7 )) (2 ) where the domain of definition of input ( 2 ) is [0, 1]. Fig. 4(a) visualizes the set of solutions of (21). Randomly select 500 pairs of input-output samples from the solution set as training and test datasets, where Fig. 4(b) displays the spatial distribution of the selected data.
(a) Solution set of Synthetic data.
(b) 500 pairs sampled data for experiment.  Table 4 shows the performance (which is denoted by training/ test error) of the control (proposed) and the comparative models including the zero/ linear/second-order fuzzy rule model (FRM) as well as the second-order FRM with regularization. The performance values are reported as average and its associated standard deviation (STD). The items in bold denote the best performance. We utilize MSE/2 as the performance index. Obviously, compared with zero-order and first-order FRM, second-order FRM has better approximation and prediction capabilities. For the three second-order FRMs with 2 (which use different penalty terms), their training error is the same as that of the second-order FRM without 2 , but their test error is slightly worse. This means that using ordinary 2 does not improve the performance of the model. Although the proposed model shares the same training error as the second-order FRM and its variants with 2 , its generalization ability is improved by using -2 .   Fig. 5 visualizes the partition of input space and the antecedent parts of fuzzy rules formed by FCM. The input space is divided by ten local regions (subspaces), and those values are served as matching degrees to estimate coefficients of the consequent parts of the fuzzy rule. Fig. 6 displays the comparison between the target (actual) output and the model output. Fig. 6(a) visualized 500 actual input-output points, while in Fig. 6(b), the blue tetrahedron stands for the model's training output, and the red cube denotes the model's predicted output.

Automobile Miles Per Gallon data (MPG)
Automobile MPG data includes 7 input variables and 392 input-output data pattern pairs. The output is the fuel consumption of the automobile represented by miles per gallon. Table 3 reports the performance (viz., training and test errors) of the control and the comparative models. RMSE is used as the performance index. The value of the performance is described in terms of mean and its related standard deviations (STD). Obviously, compared with zero-order FRM and linear-order FRM, second-order FRM has a tiny training error. However, compared with the fitting advantage of second-order FRM in training data, its generalization ability (test error) is severely deteriorated. Although the 2 regularization ( 2 ) technique can alleviate the overfitting problem of the second-order FRM, and it does not give full play the prediction performance of the quadratic function (QP) well. Compared with ordinary 2 , -2 can separately adjust (punish) different types of polynomial terms, so it can improve the predictive ability of the model by reasonably release the ability of QP to describe the behavior within local region. The optimal predictive performance of the proposed RSFRM is 2.702±0.178, which is significantly improved than that of other comparative models.
We use the logarithm of the sum of the squared coefficients (LSSC) of the consequent part of fuzzy rules to look into the effect of regularization techniques ( 2 and -2 ) on the change of coefficients (the deviation among coefficients).

ܵܵܵ (
( 2 ) 2 ( ) 2 ) (22) Fig. 7 shows the effect of regularization techniques on the coefficients and performance of the model. Without regularization, the training error is close to zero, and thereby the approximation ability reaches the ideal result. But there is a large interval between the generalization ability and approximation ability of this model. When ordinary 2 is used, the LSSC of the model decreases with the increase of the penalty term ( ), and the training error of the model increases with the increase of the penalty term (without 2 , the penalty term can be regarded as zero). Regularization can strengthen the generalization ability of the model, but there is no definite relationship between the penalty term and test error. Compared with the three models using ordinary 2 , the LSSC and training errors of the proposed model using -2 are close to the one with the minimum test error, but the generalization ability (test error) of the proposed model is superior to that of this comparative model. This is because -2 penalizes different polynomial terms separately, which can sound balance the relationship between the model's approximation ability and prediction ability through fine shrinking of the deviation among coefficients. In other words, -2 can better boost the prediction performance of the model when compared with ordinary 2 .    Fig. 8 shows the effect of the complexity (denoted by the number of fuzzy rules) on the performance of the model. In the case of 2 is not used, the training error of the model decreases sharply with the increase of the complexity, but its test error deteriorates seriously. When using ordinary 2 , a suitable (e.g., 10 -1 ) can slow down the overfitting of the model and improve the generalization ability of the model. In contrast, an excessively large (e.g., 10 1 ) may cause the training error and test error of the model to deteriorate simultaneously. The proposed model with -2 can respectively punish coefficients of polynomial terms of different types, which not only reduces the overfitting of the model but also releases the representation ability of quadratic function within the local region. Compared with other comparative models, the proposed model with -2 exhibits better prediction ability and stability under different complexity situations. Fig. 9 displays the error comparison between the model output and the target output, where the blue line indicates that the error between the two outputs is zero. Compared with the three comparative models, the output positions of RSFRM are closer to the blue line (baseline). Table 4 provides a comparison with existing models in the literature.

Computer Activity data (CA)
We consider the CA dataset, which is a collection of computer system activity metrics. The dataset contains 21 input variables and 8192 pairs of input-output data patterns. The output is the fraction of time that CPUs run in user mode. Table 5 offers the performance of the control and the comparative models. MSE/2 is utilized as the performance index. The performance value is expressed in terms of mean and STD, and boldface indicates the best performance. Compared with zero-order FRM and first-order FRM, second-order FRM has better approximation ability but has worse test error (i.e., overfitting). A suitable penalty term can alleviate the overfitting of the model and improve the generalization ability of the model. In Table 5, the best prediction performance of RSFRM is 3.733 ± 0.248, which is significantly improved when compared with the three second-order FRMs with 2 .  The effect of regularization techniques ( 2 and -2 ) on the coefficients and performance of the model is displayed in Fig. 10. In Fig. 10(a), the LSSC decreases as the penalty term ( ) increases (without 2 , can be treated as zero). The growth of does not always cause an increase in the training error of the model. When = 10 -8 , the training error of the model is slightly lower than that without regularization, which makes the generalization performance worse (the test error is more lager). This is because the further training (overtraining) of the model makes its overfitting further serious. The proposed model with -2 can improve generalization ability by establishing a reasonable and effective trade-off between the training and testing of the model, and further exerting the representation ability of quadratic polynomials within local models. Fig. 11 visualizes the impact of the complexity (represented by the number of fuzzy rules) on model performance. The black solid line with the circle represents the change of the proposed model (which uses -2 ) with the increasing of the complexity. When the penalty term ( ) is small (e.g., 10 -8 , 10 -1 ), the training error of the model decreases as the model complexity increases, and the larger penalty term (e.g., 10 1 ) will increase the training error of the model. For small or large penalty terms (e.g., 10 -8 , 10 1 ), the test error of the corresponding model becomes worse as the model complexity increases, and the appropriate can improve the generalization ability of the model. Compared with 2 , -2 , which contains three different penalty terms, can better balance the relationship between the training performance and test performance of the model. As a result, -2 exhibits more excellent and stable prediction performance under different complexity. Table 6 offers a comparative summary of the proposed model compared to other models.
(b) Test error. Fig. 11. Effects of complexity (number of rules) on the performance of models (CA).

Overall evaluations of the proposed model
In order to further evaluate and analyze the prediction effect of the proposed model, the proposed model is experimented by using a series of machine learning datasets (http://keel.es/). Table 7 offers the pertinent details of the 23 datasets used in the experiments in this section, such as the abbreviation of the dataset, the number of dimensionalities, and the scale of the data. In order to make a fair comparison, we used the method of statistical analysis. The Friedman test is widely known as the nonparametric alternative of repeated measures analysis of variance, which is used to check repeated observations on the same subject and evaluate whether the measured average grade (rank) is remarkably different from the mean grade [36]. Table 8 shows the prediction performance (represented by test error) comparison between the comparative models including the second-order fuzzy rule-based model (FRM) and its three variants with different 2 penalty terms and the proposed model with -2 . The numbers in parentheses indicate the rank of each model on the same dataset. When the significance level ( ) is set to 0.05, we can calculate the statistic Ꭹ Ꭹ according to the average rank (Avg. rank) shown in Table 8. Since Ꭹ Ꭹ = 18.50 is greater than the critical value F (4, 88) = 2.48, the Friedman test rejects the null hypothesis. For further comparison, we utilize the Bonferroni-Dunn test as the ad-hoc test to check whether the proposed model is statistically significantly superior to the other models. The Bonferroni-Dunn test is applicable to all comparative models are compared with the control (proposed) model without comparing between them. When the difference between the average ranks of two models (one is the control model and the other is the comparative model) is greater than at least the critical difference (CD), there is a significant difference in performance (that is, one model is significantly superior to the other) [33]. Under this experimental condition, the CD is 1.16. According to the results obtained in Table 9, the differences between the control model and the four comparison models are all greater than (or equal to) CD. We can conclude that the exponential weighted 2 regularization ( -2 ) can significantly improve the generalization ability of the model compared with ordinary 2 and without 2 . In addition, this study applied three state-of-art regression models such as MOKBL±MOMs [24], FRULER [35] and METSK-HD e [34] to compare the overall performance of the proposed model, including prediction performance and complexity. We use the number of fuzzy rules in the model to quantify the complexity of the model. It is generally believed that the fewer the number of rules, the stronger the interpretability of the model [23]. Table 10 shows the prediction ability (which is denoted by test error) and the number of rules of the models constructed on the various datasets of the proposed and the comparative models. MSE/2 is utilized as the performance index. The entries in boldface mean the smallest test error (or the number of rules) among the entire models. We use the Friedman test to make a fair comparison of the model's prediction performance and complexity, respectively. The last row of the table represents the average rank of the test errors and number of rules of each model on all datasets. is rejected. We use Bonferroni-Dunn test for further analysis and comparison. If the difference between the average ranks of the two models is at least greater than one CD, then there is a clear difference between the two models. Under this experimental condition, CD is 0.91. In Table 11, the difference in the average rank between all the comparative models and the control model is greater than CD. It can be argued that the proposed RSFRM is superior to the three state-of-art models in terms of prediction ability.    Table 11 shows the difference in the average rank of the number of rules between the proposed RSFRM and the three comparative models. From the Bonferroni-Dunn test (CD is 0.91), we can say that the proposed RSFRM is lower to METSK-HD e and MOKBL±MOMs in terms of complexity (low complexity means high interpretability). Through the experimental results reported in Table 10, the complexity of the proposed RSFRM is the similar as that of FRULER (win: 12 datasets, loss: 11 datasets).

Concluding remarks
In this article, we have presented and investigated the reinforced second-order fuzzy rule-based model (RSFRM), which is realized with the aid of Fuzzy C-Means (FCM) partition and quadratic polynomial (QP) as well as exponential weighted 2 regularization ( -2 ). FCM is utilized to analyze data distribution and divide the input space into local regions, as well as form the antecedent parts of fuzzy rules. The consequent part of the fuzzy rule is constructed by the quadratic polynomial. Compared with constant and linear function, QP can better describe the behavior within the local region by refining the relationship between input and output.
-2 is designed to alleviate the overfitting problem that QP may cause. Different from the ordinary 2 , -2 can separately identify and penalize the terms of different types of polynomial terms in the coefficient estimation, and its results can not only alleviate the overfitting and prevent the deterioration of generalization ability but also effectively release the prediction potential of the model.
The proposed RSFRM was comprehensively evaluated and statistically analyzed by using 23 machine learning datasets with different complexity. Through the experimental results, we proved that the prediction ability of RSFRM is better than that of its related comparative models. We also compared RSFRM with three start-of-art models, and the results show that RSFRM boosts the prediction accuracy of the model without sacrificing the interpretability of the model (that is, without increasing the complexity of the model).
The proposed RSFRM does not consider the influence of feature selection and dimensionality reduction on the performance of the model. Therefore, future work includes selecting appropriate feature selection or dimensionality reduction techniques to further strengthen the performance of the model.