Least squares: Wikis

Note: Many of our articles have direct quotes from sources you can cite, within the Wikipedia article! This article doesn't yet, but we're working on it! See more info or our list of citable articles.

Top rankings for Least squares

3rd	Top numerical analysis topics

Did you know ...

least-squares spectral analysis is a method for estimating a frequency spectrum, based on a least squares fit between data and trigonometric functions?

More interesting facts on Least squares

Include this on your site/blog:

Categories: Applied mathematics > Mathematical optimization > Statistical methods > Regression analysis > Single equation methods (econometrics) > Mathematical and quantitative methods (economics)

Encyclopedia

Updated live from Wikipedia, last check: June 03, 2012 21:51 UTC (39 seconds ago)

From Wikipedia, the free encyclopedia

The method of least squares is a standard approach to the approximate solution of overdetermined systems, i.e. sets of equations in which there are more equations than unknowns. "Least squares" means that the overall solution minimizes the sum of the squares of the errors made in solving every single equation.

The most important application is in data fitting. The best fit in the least-squares sense minimizes the sum of squared residuals, a residual being the difference between an observed value and the value provided by a model.

Least squares problems fall into two categories, linear least squares and nonlinear least squares, depending on whether or not the residuals are linear in all unknowns. The linear least-squares problem occurs in statistical regression analysis; it has a closed form solution. The non-linear problem has no closed solution and is usually solved by iterative refinement; at each iteration the system is approximated by a linear one, thus the core calculation is similar in both cases.

The least-squares method was first described by Carl Friedrich Gauss around 1794.^[1] Least squares corresponds to the maximum likelihood criterion if the experimental errors have a normal distribution and can also be derived as a method of moments estimator.

The following discussion is mostly presented in terms of linear functions but the use of least-squares is valid and practical for more general families of functions. For example, the Fourier series approximation of degree n is optimal in the least-squares sense, amongst all approximations in terms of trigonometric polynomials of degree n. Also, by iteratively applying local quadratic approximation to the likelihood (through the Fisher information), the least-squares method may be used to fit a generalized linear model.

The result of fitting a set of data points with a quadratic function.

1 History
- 1.1 Context
- 1.2 The method itself
2 Problem statement
3 Solving the least squares problem
4 Least squares, regression analysis and statistics
5 Weighted least squares
- 5.1 Principal components
- 5.2 LASSO method
6 See also
7 Notes
8 References
9 External links

History

Context

The method of least squares grew out of the fields of astronomy and geodesy as scientists and mathematicians sought to provide solutions to the challenges of navigating the Earth's oceans during the Age of Exploration. The accurate description of the behavior of celestial bodies was key to enabling ships to sail in open seas where before sailors had relied on land sightings to determine the positions of their ships.

The method was the culmination of several advances that took place during the course of the eighteenth century^[2]:

The combination of different observations taken under the same conditions contrary to simply trying one's best to observe and record a single observation accurately. This approach was notably used by Tobias Mayer while studying the librations of the moon.
The combination of different observations as being the best estimate of the true value; errors decrease with aggregation rather than increase, perhaps first expressed by Roger Cotes.
The combination of different observations taken under different conditions as notably performed by Roger Joseph Boscovich in his work on the shape of the earth and Pierre-Simon Laplace in his work in explaining the differences in motion of Jupiter and Saturn.
The development of a criterion that can be evaluated to determine when the solution with the minimum error has been achieved, developed by Laplace in his Method of Situation.

The method itself

Carl Friedrich Gauss

Carl Friedrich Gauss is credited with developing the fundamentals of the basis for least-squares analysis in 1795 at the age of eighteen. Legendre was the first to publish the method, however.

An early demonstration of the strength of Gauss's method came when it was used to predict the future location of the newly discovered asteroid Ceres. On January 1, 1801, the Italian astronomer Giuseppe Piazzi discovered Ceres and was able to track its path for 40 days before it was lost in the glare of the sun. Based on this data, it was desired to determine the location of Ceres after it emerged from behind the sun without solving the complicated Kepler's nonlinear equations of planetary motion. The only predictions that successfully allowed Hungarian astronomer Franz Xaver von Zach to relocate Ceres were those performed by the 24-year-old Gauss using least-squares analysis.

Gauss did not publish the method until 1809, when it appeared in volume two of his work on celestial mechanics, Theoria Motus Corporum Coelestium in sectionibus conicis solem ambientium. In 1829, Gauss was able to state that the least-squares approach to regression analysis is optimal in the sense that in a linear model where the errors have a mean of zero, are uncorrelated, and have equal variances, the best linear unbiased estimator of the coefficients is the least-squares estimator. This result is known as the Gauss–Markov theorem.

The idea of least-squares analysis was also independently formulated by the Frenchman Adrien-Marie Legendre in 1805 and the American Robert Adrain in 1808.

Problem statement

The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of n points (data pairs) $(x_i,y_i)\!$ , i = 1, ..., n, where $x_i\!$ is an independent variable and $y_i\!$ is a dependent variable whose value is found by observation. The model function has the form $f (x,β)$ , where the m adjustable parameters are held in the vector $\boldsymbol \beta$ . The parameter values for which the model "best" fits the data need be found. The least squares method finds its optimum when the sum, S, of squared residuals

$S=\sum_{i=1}^{n}{r_i}^2$

is a minimum. A residual is defined as the difference between the value of the dependent variable and the model value

r i = y i - f (x i,β).

An example of a model is that of the straight line. Denoting the intercept as $β 0$ and the slope as $β 1$ , the model function is given by $f(x,\boldsymbol \beta)=\beta_0+\beta_1 x$ . See linear least squares for a fully worked out example of this model.

A data point may consist of more than one independent variable. For an example, when fitting a plane to a set of height measurements, the plane is a function of two independent variables, x and z, say. In the most general case there may be one or more independent variables and one or more dependent variables at each data point.

Solving the least squares problem

The minimum of the sum of squares is found by setting the gradient to zero. Since the model contains m parameters there are m gradient equations.

$\frac{\partial S}{\partial \beta_j}=2\sum_i r_i\frac{\partial r_i}{\partial \beta_j}=0,\ j=1,\ldots,m$

and since $r_i=y_i-f(x_i,\boldsymbol \beta)\,$ the gradient equations become

$-2\sum_i \frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j} r_i=0,\ j=1,\ldots,m$

The gradient equations apply to all least squares problems. Each particular problem requires particular expressions for the model and its partial derivatives.

Linear least squares

Main article: Linear least squares

A regression model is a linear one when the model comprises a linear combination of the parameters, i.e.

$f(x_i, \beta) = \sum_{j = 1}^{m} \beta_j \phi_j(x_{i})$

where the coefficients, $φ j$ , are functions of $x i$ .

Letting

$X_{ij}= \frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j}= \phi_j(x_{i}) . \,$

we can then see that in that case the least square estimate (or estimator, in the context of a random sample), $\boldsymbol \beta$ is given by

$\boldsymbol{\hat\beta} =( X ^TX)^{-1}X^{T}\boldsymbol y$

For a derivation of this estimate see Linear least squares.

Non-linear least squares

Main article: Non-linear least squares

There is no closed-form solution to a non-linear least squares problem. Instead, numerical algorithms are used to find the value of the parameters $β$ which minimize the objective. Most algorithms involve choosing initial values for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation.

${\beta_j}^{k+1}={\beta_j}^k+\Delta \beta_j$

k is an iteration number and the vector of increments, $\Delta \beta_j\,$ is known as the shift vector. In some commonly used algorithms, at each iteration the model may be linearized by approximation to a first-order Taylor series expansion about $\boldsymbol \beta^k\!$

$\begin{align} f(x_i,\boldsymbol \beta) & = f^k(x_i,\boldsymbol \beta) +\sum_j \frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j} \left(\beta_j-{\beta_j}^k \right) \ & = f^k(x_i,\boldsymbol \beta) +\sum_j J_{ij} \Delta\beta_j. \end{align}$

The Jacobian, J, is a function of constants, the independent variable and the parameters, so it changes from one iteration to the next. The residuals are given by

$r_i=y_i- f^k(x_i,\boldsymbol \beta)- \sum_{j=1}^{m} J_{ij}\Delta\beta_j=\Delta y_i- \sum_{j=1}^{m} J_{ij}\Delta\beta_j$ .

To minimize the sum of squares of $r i$ , the gradient equation is set to zero and solved for $\Delta \beta_j\!$

$-2\sum_{i=1}^{n}J_{ij} \left( \Delta y_i-\sum_{j=1}^{m} J_{ij}\Delta \beta_j \right)=0$

which, on rearrangement, become m simultaneous linear equations, the normal equations.

$\sum_{i=1}^{n}\sum_{k=1}^{m} J_{ij}J_{ik}\Delta \beta_k=\sum_{i=1}^{n} J_{ij}\Delta y_i \qquad (j=1,\ldots,m)\,$

The normal equations are written in matrix notation as

$\mathbf{\left(J^TJ\right)\Delta \boldsymbol \beta=J^T\Delta y}.\,$

These are the defining equations of the Gauss–Newton algorithm.

Differences between linear and non-linear least squares

The model function, f, in LLSQ (linear least squares) is a linear combination of parameters of the form $f = X_{i1}\beta_1 + X_{i2}\beta_2 +\cdots$ The model may represent a straight line, a parabola or any other polynomial-type function. In NLLSQ (non-linear least squares) the parameters appear as functions, such as $β 2, e β x$ and so forth. If the derivatives $\partial f /\partial \beta_j$ are either constant or depend only on the values of the independent variable, the model is linear in the parameters. Otherwise the model is non-linear.
Many solution algorithms for NLLSQ require initial values for the parameters, LLSQ does not.
Many solution algorithms for NLLSQ require that the Jacobian be calculated. Analytical expressions for the partial derivatives can be complicated. If analytical expressions are impossible to obtain the partial derivatives must be calculated by numerical approximation.
In NLLSQ non-convergence (failure of the algorithm to find a minimum) is a common phenomenon whereas the LLSQ is globally concave so non-convergence is not an issue.
NLLSQ is usually an iterative process. The iterative process has to be terminated when a convergence criterion is satisfied. LLSQ solutions can be computed using direct methods, although problems with large numbers of parameters are typically solved with iterative methods, such as the Gauss–Seidel method.
In LLSQ the solution is unique, but in NLLSQ there may be multiple minima in the sum of squares.
Under the condition that the errors are uncorrelated with the predictor variables, LLSQ yields unbiased estimates, but even under that condition NLLSQ estimates are generally biased.

These differences must be considered whenever the solution to a non-linear least squares problem is being sought.

Least squares, regression analysis and statistics

The methods of least squares and regression analysis are conceptually different. However, the method of least squares is often used to generate estimators and other statistics in regression analysis.

Consider a simple example drawn from physics. A spring should obey Hooke's law which states that the extension of a spring is proportional to the force, F, applied to it.

$f(F_i,k)=kF_i\!$

constitutes the model, where F is the independent variable. To estimate the force constant, k, a series of n measurements with different forces will produce a set of data, $(F_i, y_i), i=1,n\!$ , where y_i is a measured spring extension. Each experimental observation will contain some error. If we denote this error $\varepsilon$ , we may specify an empirical model for our observations,

$y_i = kF_i + \varepsilon_i. \,$

There are many methods we might use to estimate the unknown parameter k. Noting that the n equations in the m variables in our data comprise an overdetermined system with one unknown and n equations, we may choose to estimate k using least squares. The sum of squares to be minimized is

$S = \sum_{i=1}^{n} \left(y_i - kF_i\right)^2.$

The least squares estimate of the force constant, k, is given by

$\hat k=\frac{\sum_i F_i y_i}{\sum_i {F_i}^2}.$

Here it is assumed that application of the force causes the spring to expand and, having derived the force constant by least squares fitting, the extension can be predicted from Hooke's law.

In regression analysis the researcher specifies an empirical model. For example, a very common model is the straight line model which is used to test if there is a linear relationship between dependent and independent variable. If a linear relationship is found to exist, the variables are said to be correlated. However, correlation does not prove causation, as both variables may be correlated with other, hidden, variables, or the dependent variable may "reverse" cause the independent variables, or the variables may be otherwise spuriously correlated. For example, suppose there is a correlation between deaths by drowning and the volume of ice cream sales at a particular beach. Yet, both the number of people going swimming and the volume of ice cream sales increase as the weather gets hotter, and presumably the number of deaths by drowning is correlated with the number of people going swimming. Perhaps an increase in swimmers causes both the other variables to increase.

In order to make statistical tests on the results it is necessary to make assumptions about the nature of the experimental errors. A common (but not necessary) assumption is that the errors belong to a Normal distribution. The central limit theorem supports the idea that this is a good assumption in many cases.

The Gauss–Markov theorem. In a linear model in which the errors have expectation zero conditional on the independent variables, are uncorrelated and have equal variances, the best linear unbiased estimator of any linear combination of the observations, is its least-squares estimator. "Best" means that the least squares estimators of the parameters have minimum variance. The assumption of equal variance is valid when the errors all belong to the same distribution.
In a linear model, if the errors belong to a Normal distribution the least squares estimators are also the maximum likelihood estimators.

However, if the errors are not normally distributed, a central limit theorem often nonetheless implies that the parameter estimates will be approximately normally distributed so long as the sample is reasonably large. For this reason, given the important property that the error is mean independent of the independent variables, the distribution of the error term is not an important issue in regression analysis. Specifically, it is not typically important whether the error term follows a normal distribution.

In a least squares calculation with unit weights, or in linear regression, the variance on the jth parameter, denoted $\text{var}(\hat{\beta}_j)$ , is usually estimated with

$\text{var}(\hat{\beta}_j)= \sigma^2\left( \left[X^TX\right]^{-1}\right)_{jj} \approx \frac{S}{n-m}\left( \left[X^TX\right]^{-1}\right)_{jj},$

where the true residual variance σ² is replaced by an estimate based on the minimised value of the sum of squares objective function S.

Confidence limits can be found if the probability distribution of the parameters is known, or an asymptotic approximation is made, or assumed. Likewise statistical tests on the residuals can be made if the probability distribution of the residuals is known or assumed. The probability distribution of any linear combination of the dependent variables can be derived if the probability distribution of experimental errors is known or assumed. Inference is particularly straightforward if the errors are assumed to follow a normal distribution, which implies that the parameter estimates and residuals will also be normally distributed conditional on the values of the independent variables.

Weighted least squares

See also: Weighted mean

The expressions given above are based on the implicit assumption that the errors are uncorrelated with each other and with the independent variables and have equal variance. The Gauss–Markov theorem shows that, when this is so, $\hat\boldsymbol\beta$ is a best linear unbiased estimator (BLUE). If, however, the measurements are uncorrelated but have different uncertainties, a modified approach might be adopted. Aitken showed that when a weighted sum of squared residuals is minimized, $\hat\boldsymbol\beta$ is BLUE if each weight is equal to the reciprocal of the variance of the measurement.

$S = \sum_{i=1}^{n} W_{ii}{r_i}^2,\qquad W_{ii}=\frac{1}{{\sigma_i}^2}$

The gradient equations for this sum of squares are

$-2\sum_i W_{ii}\frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j} r_i=0,\qquad j=1,\ldots,n$

which, in a linear least squares system give the modified normal equations

$\sum_{i=1}^{n}\sum_{k=1}^{m} X_{ij}W_{ii}X_{ik}\hat \beta_k=\sum_{i=1}^{n} X_{ij}W_{ii}y_i, \qquad j=1,\ldots,m\,$

$\mathbf{\left(X^TWX\right)\hat \boldsymbol \beta=X^TWy}.$

When the observational errors are uncorrelated the weight matrix, W, is diagonal. If the errors are correlated, the resulting estimator is BLUE if the weight matrix is equal to the inverse of the variance-covariance matrix of the observations.

When the errors are uncorrelated, it is convenient to simplify the calculations to factor the weight matrix as $w_{ii}=\sqrt W_{ii}$ . The normal equations can then be written as

$\mathbf{\left(X'^TX'\right)\hat \boldsymbol \beta=X'^Ty'}\,$

where

$\mathbf{X'}=\mathbf{wX}, \mathbf{y'}=\mathbf{wy}.\,$

For non-linear least squares systems a similar argument shows that the normal equations should be modified as follows.

$\mathbf{\left(J^TWJ\right)\boldsymbol \Delta\beta=J^TW \boldsymbol\Delta y}.\,$

Note that for empirical tests, the appropriate W is not known for sure and must be estimated. For this Feasible Generalized Least Squares (FGLS) techniques may be used.

Principal components

The first principal component about the mean of a set of points is equivalent to the linear least squares solution. One of the most computationally efficient ways to solve a linear least squares problem is to use the EM technique to compute the first principal component about the mean of the data. This algorithm can be trivially modified to compute a weighted least squares solution as well.

LASSO method

In some contexts a regularized version of the least squares solution may be preferable. The LASSO algorithm, for example, finds a least-squares solution with the constraint that $| β | 1$ , the L¹-norm of the parameter vector, is no greater than a given value. Equivalently, it may solve an unconstrained minimization of the least-squares penalty with $α | β | 1$ added, where $α$ is a constant (this is the Lagrangian form of the constrained problem.) This problem may be solved using quadratic programming or more general convex optimization methods, as well as by specific algorithms such as the least angle regression algorithm. The L¹-regularized formulation is useful in some contexts due to its tendency to prefer solutions with fewer nonzero parameter values, effectively reducing the number of variables upon which the given solution is dependent ^[3]. For this reason, the LASSO and its variants are fundamental to the field of compressed sensing.

Notes

^ Bretscher, Otto (1995). Linear Algebra With Applications, 3rd ed.. Upper Saddle River NJ: Prentice Hall.
^ Stigler, Stephen M. (1986). The History of Statistics: The Measurement of Uncertainty Before 1900. Cambridge, MA: Belknap Press of Harvard University Press.
^ Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Royal. Statist. Soc B., Vol. 58, No. 1, pages 267–288

References

Å. Björck, Numerical Methods for Least Squares Problems, SIAM, 1996 [1].
C.R. Rao, H. Toutenburg, A. Fieger, C. Heumann, T. Nittner and S. Scheid, Linear Models: Least Squares and Alternatives, Springer Series in Statistics, 1999.
T. Kariya, H. Kurata, Generalized Least Squares, Wiley, 2004.
J. Wolberg, Data Analysis Using the Method of Least Squares: Extracting the Most Information from Experiments, Springer, 2005.

External links

Derivation of quadratic least squares
Power Point Statistics Book -- Excellent slides providing an introductory regression example (University of Texas at Arlington)

Least squares and regression analysis

Computational statistics

Least squares · Linear least squares · Non-linear least squares

Regression analysis

Ordinary least squares · Generalized least squares · Weighted least squares · Iteratively reweighted least squares · Partial least squares · Total least squares · Ridge regression

Regression as a
statistical model

Correlation	Pearson product-moment correlation · Rank correlation (Spearman's rho, Kendall's tau) · Partial correlation · Confounding variable

Linear regression	Simple linear regression · Ordinary least squares · General linear model · Analysis of variance · Analysis of covariance

Predictor structure	Polynomial regression · Growth curves · Segmented regression · Regression discontinuity · Local regression

Non-standard	Nonlinear regression · Nonparametric · Semiparametric · Robust · Quantile · Isotonic

Non-normal errors	Generalized linear model · Binomial · Poisson · Logistic

Model exploration

Mallows' Cp · Stepwise regression · Model selection

Background

Mean and predicted response -Gauss–Markov theorem - Errors and residuals - Goodness of fit - Studentized residual - Minimum mean-square error

Design of experiments

Response surface methodology - Optimal design - Bayesian design

Numerical approximation

Numerical analysis - Approximation theory - Numerical integration - Gaussian quadrature - Orthogonal polynomials - Chebyshev polynomials - Chebyshev nodes

Applications

Curve fitting - Calibration curve - Numerical smoothing and differentiation - Moving least squares

Category (Regression Analysis) - Statistics Category - Statistics Portal - Statistics outline - Statistics topics

Study guide

Up to date as of January 14, 2010

From Wikiversity

We use the method of Least squares when we have a series of measures (x_i, y_i) with i = 1, 2, ..., n (i.e., we measured a set of values we called y, and each of these depended on the value of a variable we called x), and we know that the measured points (x_i, y_i), when drawn on a plane, should (in theory) form a straight line.

Because of measurement errors, the points measured, most probably, will not form a straight line, but they will be aproximately aligned. We are interested in finding the straight line which is the most similar to the points measured. So we want to find a function $f (x) = a x + b$ (that is, a straight line) such that $f(x_i)\approx y_i.$ We only need to find the appropiate a and b and we will have found our function.

How do we express most similar mathematically?

The criterion we will follow to find those a and b is to minimize the error $f (x i) - y i$ . If we define it this way, this error is the vertical distance between each measured point and the straight line we are seeking. To minimize the total error, we will try to minimize the sum of all errors:

$E = \sum_{i=1}^n (f(x_i) - y_i)$ .

But this formula for the global error has a problem: If we have two points, the value $y i$ of the first being far below the line (so with a big positive error), and the other one, $y j$ , being far above the line (so with a big negative error), when we sum both errors they will cancel out mutually, giving a total error of $E = 0$ . Evidently, that is not what we are seeking. The solution to this problem is to change our formula for the total error and try this one:

$E = \sum_{i=1}^n (f(x_i) - y_i)^2$ .

We will not sum the individual errors, but their squares. This way, all of them will be positive, and to minimize the sum all of them will have to tend to 0.

Why don't we use the absolute value of the errors (

| f (x i) - y i |

)? Because, as we will see, if we use the square of the errors there exists an exact formula to find the a and b which minimize the error (and with the absolute values there is not such formula).

This is why the method is called "least squares", because it tries to find the line which produces the least squares of the individual errors.

Note that we are assuming that the x values are exact, and all the errors are in the y values. In most cases this may be not true, and we could have errors in the y and in the x values. There are other methods which consider this, but the least squares method is the most simple and the most used.

Which are a and b, then?

The values for a and b that minimize the total error E are:

$a = \frac{n\sum_{i=1}^n (x_iy_i) - \sum_{i=1}^n x_i \sum_{i=1}^n y_i}{n\sum_{i=1}^n x_i^2 - (\sum_{i=1}^n x_i)^2}$

and

$b = \frac{\sum_{i=1}^n x_i^2 \sum_{i=1}^n y_i - \sum_{i=1}^n x_i \sum_{i=1}^n (x_iy_i)}{n\sum_{i=1}^n x_i^2 - (\sum_{i=1}^n x_i)^2}$

The most convenient way to calculate these values is to tabulate your data in columns, namely $x_i\;$ , $y_i\;$ , $x_i^2$ and $x_iy_i\;$ . After calculating the products for each pair $(x_i,y_i)\;$ , we sum each column, obtaining this way $\sum_{i=1}^n x_i\;$ , $\sum_{i=1}^n y_i\;$ , $\sum_{i=1}^n x_i^2\;$ and $\sum_{i=1}^n (x_iy_i)\;$ . With these values calculated we only have to substitute them in the formulas for a and b and we are done.

Of course, all the sums and products can be done automatically using a spreadsheet such as Excel. The steps are explained here with an example.

How to find a and b

To find a and b we have to minimize the two-variables function E discussed above. To minimize N-variables functions, as you shall see when you see partial derivatives, you have to find the point(s) where all of its partial derivatives become 0.

Considering the form of E, this gives a set of two linear equations of the two variables a and b. The system is easily solved using Crammer's rule (or equivalently, since the matrix of coefficients is 2x2, inverting it, which is trivial). And the solution found is the expresions given above for a and b.

Category: Mathematics

The Full Wiki

Search:

More info on Least squares

Related topics