Nonlinear Estimation Procedures - Function Minimization Algorithms

Now that we have discussed different regression models, and the loss functions that can be used to estimate them, the only "mystery" that is left is how to minimize the loss functions (to find the best fitting set of parameters), and how to estimate the standard errors of the parameter estimates. Nonlinear Estimation uses one very efficient general algorithm (quasi-Newton) that approximates the second-order derivatives of the loss function to guide the search for the minimum (i.e., for the best parameter estimates, given the respective loss function).

For nonlinear least-squares regression (i.e., nonlinear regression functions, and the least-squares loss function), Nonlinear Estimation includes a dedicated algorithm that is very efficient and robust, and is the recommended estimation method when analyzing large data sets and using the least-squares loss function): The Levenberg-Marquardt algorithm (Levenberg, 1944; Marquardt, 1963; see also Moré, 1977).

In addition, Nonlinear Estimation offers several more general function minimization algorithms that follow different search strategies (which do not depend on the evaluation of derivatives). These strategies are sometimes more effective for estimating loss functions with local minima; therefore, these methods are often particularly useful to find appropriate start values for the estimation via the quasi-Newton method.

In all cases, STATISTICA can compute (by request) the standard errors of the parameter estimates. These standard errors are based on the second-order partial derivatives for the parameters, which are computed via finite difference approximation.


In the following paragraphs we will first discuss some general issues involved in unconstrained optimization, and then briefly review the methods used in this module. For more detailed discussions of these procedures you can refer to Brent (1973), Gill and Murray (1974), Peressini, Sullivan, and Uhl (1988), and Wilde and Beightler (1967). For specific algorithms, see Dennis and Schnabel (1983), Eason and Fenton (1974), Fletcher (1969), Fletcher and Powell (1963), Fletcher and Reeves (1964), Hooke and Jeeves (1961), Jacoby, Kowalik, and Pizzo (1972), and Nelder and Mead (1964).


Penalty Functions, Constraining Parameters. All estimation procedures in Nonlinear Estimation are unconstrained in nature. That means that the program will move parameters around without any regard for whether or not permissible values result. For example, in the course of logit regression we may get estimated values that are equal to 0.0, in which case the logarithm cannot be computed (since the log of 0 is undefined). When this happens, the program will assign a penalty to the loss function, that is, a very large value. As a result, the various estimation procedures usually move away from the regions that produce those functions. However, in some circumstances, the estimation will "get stuck," and as a result, you would see a very large value of the loss function. This could happen, if, for example, the regression equation involves taking the logarithm of an independent variable which has a value of zero for some cases (in which case the logarithm cannot be computed).

If you want to constrain a procedure in Nonlinear Estimation, then this constraint must be specified in the loss function as a penalty function (assessment). By doing this, you can control what permissible values of the parameters to be estimated can be manipulated by the program. For example, if two parameters (a and b) are to be constrained to be greater than or equal to zero, then one must assess a large penalty to these parameters if this condition is not met. Below is an example of a user-specified regression and loss function, including a penalty assessment designed to "penalize" the parameters a and/or b if either one is not greater than or equal to zero:

Loss function: L = (obs - pred)2 + (a<0)*100000 + (b<0)*100000



Levenberg-Marquardt Algorithm (Nonlinear Least Squares). Levenberg (1944) and Marquardt (1963) proposed an efficient method for estimating the parameters of nonlinear regression models, when using the least-squares loss function. Using the least squares loss function, the second-order partial derivatives do not have to be computed (or approximated) in order to find the least-squares parameter estimates; instead, the algorithm will in each iteration solve a set of linear equations to compute the gradient, which computationally is relatively (compared to other optimization techniques) easy and fast. Details of the derivation and specific steps defining the Levenberg-Marquardt algorithm are discussed in Moré (1977).  In particular when analyzing large data sets, this estimation procedure (and the least-squares loss function) is the recommended method for computing parameter estimates.  

The Levenberg-Marquardt (LM) method is an improvement/modification of the Gauss-Newton method for obtaining the minimizing solution of the nonlinear least squares problem. Consider the nonlinear model fitting y = ¦ (q,c) with the given data Xi and Yi, i = 1,...,m where Xi is of dimension k and θ is of dimension N. The LM method seeks θ* the solution of θ (locally) minimizing:

g(q)= åim=1(Yi-¦ (q,Xi ))2

The LM finds the solution applying the routine:

q j+1=q j-(J'J+lD)-1J'(Y-¦(q,Xi))

iteratively, where:


is the m x 1 vector containing Y1,..., Ym


is the m x k  matrix containing X1,..., Xm


is the  m x n Jacobian matrix for ¦(q,c) with respect to  q


is a  n x n  diagonal matrix to adjust scale factors

The automatic scaling method explained in More (1977) is used in STATISTICA. You are asked to input your (nonlinear) model function ¦(q,c), initial parameter values (default 0.1), and the data for Y and X.



dependent variable Y, and independent variables X


model function ¦(q,c)

Initial values

the value of θ in the first iteration


Minimized value of g(θ*)

Coefficient of determination R-square

Estimator θ, i.e. θ*

Covariance matrix of θ*, g(θ*)(J'J)-1/(m-n)

Correlation matrix of θ*

Confidence Interval of θ* at variable significant level

Residuals and Predicted values

ANOVA table

History of iteration for the least square values and the parameters




Rosenbrock Pattern Search. Where all other methods fail, the Rosenbrock pattern search method often succeeds. This method rotates the parameter space and aligns one axis with a ridge (this method is also called the method of rotating coordinates); all other axes remain orthogonal to this axis. If the loss function is unimodal and has detectable ridges pointing towards the minimum of the function, then this method will proceed with sure-footed accuracy towards the minimum of the function. However, note that this search algorithm may terminate early when there are several constraint boundaries (resulting in the penalty value; see above) that intersect, leading to a discontinuity in the ridges.


The Hessian matrix (and asymptotic standard errors for the parameters) are computed separately via finite difference approximation by selecting the Asymptotic standard errors check box from the Advanced tab of the Model Estimation dialog. This procedure yields very precise asymptotic standard errors for all estimation methods.