{'fraction_non_alphanumeric': 0.03209814301319217, 'fraction_numerical': 0.020082346021342744, 'mean_word_length': 4.716618635926993, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Liu Estimator in the Multinomial Logistic Regression Model

This paper considers the Liu estimator in the multinomial logistic regression model. We propose some different estimators of the biasing parameter. The mean square error (MSE) is considered as the performance criterion. In order to compare the performance of the estimators, we performed a Monte Carlo simulation study. According to the results of the simulation study, we found that increasing the correlation between the independent variables and the number of regressors has a negative effect on the MSE. However, when the sample size increases the MSE decreases even when the correlation between the independent variables is large. Based on the minimum MSE criterion some useful estimators for estimating the biasing parameter d are recommended for the practitioners.


Introduction
In linear regression, the multicollinearity problem occurs when the explanatory variables are intercorrelated. When this problem occurs in multiple linear regression, it is not convenient to estimate the parameters by using the ordinary least squared (OLS) method due to the large variance and the instability of the OLS method. Similarly, in binary and multinomial logistic regression models the maximum likelihood estimation (MLE) method has these drawbacks. Thus, the inferences based on the regression analysis may not be trustworthy. Hoerl and Kennard (1970) suggested the ridge regression to solve this problem in the multiple linear regression model. In literature, many researchers have proposed different ridge estimators. To mention a few, Kibria (2003), Khalaf and Shukur (2005), Alkhamisi et al. (2006), Alkhamisi and Shukur (2007), Muniz and Kibria (2009), Månsson et al. (2010), Muniz et al. (2012). In ridge regression, a small positive number k is added to the diagonal entries of the correlation matrix ′ to decreases the variance so that one can obtain stable estimated coefficients. However, there is a disadvantage of the ridge estimator such that the estimated parameters are non-linear functions of the ridge parameter and that the small values of not be large enough to overcome multicollinearity. Liu (1993) proposed the Liu estimator which has the advantage of being a linear function of the shrinkage parameter . This estimator has the advantages of ridge estimator and Stein estimator Stein (1956).
Logistic regression, being a kind of generalized linear model, is one of the most widely used statistical analysis in many fields. Moreover, there is a growing interest in the problem of multicollinearity for the logistic regression model. We recommend the following studies: In Schaefer et al. (1984), a logistic ridge estimator was proposed for this problem. Månsson and Shukur (2011) and Kibria et al. (2012) proposed many ridge estimators to be used in logistic ridge regression. Månsson et al. (2012) proposed to generalize the Liu estimator to the binary logistic regression model. Huang (2012) also proposed a two-parameter estimator in logistic regression. Inan and Erdogan (2013) proposed the Liu-type logistic estimators by generalizing the Liu-type estimator (Liu, 2003) to the binary logistic regression model. Asar and Genç (2016) proposed some new shrinkage estimators for the Liu-type logistic estimator.
However, multicollinearity has not been studied in the multinomial logistic regression models so far. One exception is the paper of Månsson et al. (2018) in which the authors proposed to use ridge regression when there is a multicollinearity problem in the multinomial logistic regression. They conducted a Monte Carlo simulation study to evaluate the performance of the existing ridge estimators and the MLE using the mean squared error (MSE) criterion.
Another exception is that Abonazel and Farghali (2019) proposed a two-parameter estimator including both ridge and Liu estimators in the multinomial logistic regression.
The purpose of this paper is to generalize the Liu estimator (Liu, 1993) to the multinomial logistic regression and propose some methods to estimate the shrinkage parameter used in multinomial logistic Liu estimator (MLLE) to combat multicollinearity in the multinomial logistic regression model. Liu estimator with the proposed shrinkage estimators is assumed to perform much better than the maximum likelihood estimation when the explanatory variables are correlated. We design a Monte Carlo simulation to evaluate the performance of the estimators. Mean squared error (MSE) is used to compare the estimators as a performance criterion.
This paper is organized as follows: In Section 2, we present the model, the methodology, and the proposed shrinkage estimators. The details of the Monte Carlo experiment are given in Section 3. We provide the results and discussions of the simulation in Section 4. An application of real data is demonstrated in Section 5. Finally, a summary and conclusion are presented.

Maximum likelihood estimator
When the dependent variable consists of ( > 2) different categories, the multinomial logistic regression model developed by Luce (1959) is one of the most popular statistical methods. The multinomial logistic model can be specified as follows where is the lth row of which is a data matrix with explanatory variables and is a vector of coefficients. Most commonly, the method of maximum likelihood is used when estimating by maximizing the following log-likelihood function

(2.2)
In order to maximize the Equation (2.2), we should obtain the derivative with respect to and equate the resulting function zero and then solve for . We obtain the following subsequent equation:

. (2.3)
Since these equations are nonlinear in , the iteratively re-weighted least squares (IRLS) method is generally used as follows (Saleh and Kibria, 2013): where and where is the estimated values of using and is the lth element of . After some algebra, the Equation (2.4) becomes where with and . Månsson et al. (2018) obtained the asymptotic covariance and scalar mean squared error (MSE) of MLE respectively as follows: (2.6) and

(2.7)
where is the ith eigenvalue of .
However, some of the 's becomes close to zero such that the variance of becomes inflated when there is multicollinearity between the explanatory variables. Due to this problem, becomes unstable and has large standard errors.

Multinomial logistic Liu estimator
Since the variance of MLE is inflated, we propose to use the logistic Liu estimator (Månsson et al., 2012) to decrease the variance in the multinomial logistic model. Although Abonazel and Farghali (2019)  (2.8) where . It is easy to obtain the asymptotic covariance and bias of MLL as follows: (2.9)

and
(2.10) ( ) where . Therefore, we can obtain the matrix mean squared error (MMSE) and scalar MSE of MLL respectively as follows: (2.11) and (2.12) where is the square of the ith component of such that is the matrix whose columns are eigenvectors of the matrix and such that 's are the eigenvalues of . Månsson et al. (2012) showed that there is always a value of between zero and one so that .

Estimators of the biasing parameter d
To estimate the parameter , we differentiate the MSE function (2.12) with respect to and obtain the following function: . (2.13) where . If we solve the Equation (2.13) for each parameter , we obtain the following individual parameters: Following Månsson et al. (2012), we propose to choose the biasing parameter as follows: , , .
Since is not known in reality, we estimate it by .

Design of the simulation
In this section, we present the details and the results of the Monte Carlo simulation which is conducted to evaluate the performances of the estimators MLL and MLE. We refer to the following papers to design the simulation Lee and Silvapulle (1988)  The coefficient vector is chosen due to Newhouse and Oman (1971) such that which is a commonly used restriction, for example, see Kibria (2003). We generate the observations of the dependent variable using

(3.2)
where and is the lth row of the data matrix .
The simulation is repeated 2000 times. To compute the simulated MSEs of the estimators, the following equation is used respectively:

(3.3)
where is MLE or MLL in the simulation and R is the number of replications. The convergence tolerance is taken to be .

Results of the simulation
The simulated MSE values of the estimators are reported in Tables 1-4. According to the tables, we can make the following conclusions: • It is observed that increasing the degree of correlation affects the estimators negatively, namely, their MSE values increases. Especially, the MSE of MLE is inflated when the degree of correlation is increased. The least affected estimator is MLL with / from this situation.
• Moreover, we observe that increasing the sample size makes a positive effect on the estimators. MSE of MLE decreases rapidly when the sample size increases.
• It can also be observed from tables that if we increase the number of explanatory variables, the estimators are affected negatively, especially, MSE of MLE is inflated.
• If we compare the estimators of the biasing parameter , we see that / has the best performance. Actually, MLL with / has the best performance in all situations.
• Furthermore, we also observe that when the degree of correlation is 0.9 and the sample size is 500 or 1000, the performance of the estimators becomes close to each other.
When p=4, MLL with 0 is always better than MLL with 1 .
• However, if we increase , we observe that increasing the degree of correlation or the sample size makes MLL with 1 better than MLL with 0 . For example, when p = 20, although the performance of MLL with 0 is better than 1 for small sample sizes and = 0.9, the performance of MLL 1 becomes better than MLL with 0 for large sample size (n = 1000) and = 0.999.
• Overall, MLL with / is the best estimator among the others. Therefore, we recommend using MLL in multicollinear situations in the multinomial logistic regression models.

A Real Data Application
In this section of the study, the data compiled for three different plant species from the leaf data set created by Silva (2013) was analyzed. Each plant's scientific name, the number of leaf specimens by species, and the texture features of leaves analyze are given in Table 5. The correlation matrix of the texture features of leaves is given in Table 6.
When the correlation matrix is examined, it is seen that there are high correlations between texture features. The square root of the ratio between the maximum and min eigenvalue of the correlation matrix for explanatory variables is called condition number (CN).

=@A
Since there are two weight matrices in the multinomial model, we compute the condition numbers of each matrix as CN1=384.845 and CN2=598.147. The values of CN between 10 and 30 indicate the presence of multicollinearity and when a value is larger than 30, the multicollinearity is regarded as strong (Kim, 2019). Therefore, we conclude that there is a strong multicollinearity problem with leaf data.   Table 7. According to Table 7 it is readily observed that the multinomial Liu estimator has a lower MSE value for each level.

Conclusion
This paper generalizes the well-known Liu estimator to the multinomial logistic regression model. We obtain the scalar mean squared error of the estimators MLE and MLL. Since there is biasing parameter in MLL, we propose some methods based on the study of Månsson et al. (2012) to estimate this parameter. A Monte Carlo simulation is designed to investigate the performance of MLL and MLE in the presence of different degrees of multicollinearity.
Moreover, a real data example is presented to illustrate the findings of the paper. According to both the results of simulation and real application, we conclude that MLL has a better performance than MLE in multicollinear situations. Therefore, we recommend MLL to the researchers.