In mathematical statistics and information theory, the Fisher information (sometimes simply called information^{[1]}) is the variance of the score. Its role in the asymptotic theory of maximumlikelihood estimation was emphasized by the statistician R.A. Fisher (following some initial results by F. Y. Edgeworth). In Bayesian statistics, the Fisher information is used in the calculation of the Jeffreys prior.
An important use of the Fisher information matrix in statistical analyses is its contribution to the calculation of the covariance matrices of estimates of parameters fitted by maximium likelihood. It can also be used in the formulation of test statistics, such as the Wald test.
Contents 
The Fisher information was discussed by several early statisticians, notably F. Y. Edgeworth.^{[2]} For example, Savage^{[3]} says: "In it [Fisher information], he [Fisher] was to some extent anticipated (Egeworth 19089 esp. 502, 5078, 662, 6778,825 and references he [Edgeworth] cites including Pearson and Filon 1898 [. . .])." There are a number of early historical sources^{[4]} and a number of reviews of this early work.^{[5]}^{[6]}^{[7]}
The Fisher information is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ upon which the likelihood function of θ, L(θ) = f(X;θ), depends. The likelihood function is the joint probability of the data, the Xs, conditional on the value of θ, as a function of θ. Since the expectation of the score is zero, the variance is simply the second moment of the score, the derivative of the log of the likelihood function with respect to θ. Hence the Fisher information can be written as:
which implies . The Fisher information is thus the expectation with respect to of the squared score. A random variable carrying high Fisher information implies that the absolute value of the score is often high.
The Fisher information is not a function of a particular observation, as the random variable X has been averaged out. The concept of information is useful when comparing two methods of observing a given random process.
If lnf(x;θ) is twice differentiable with respect to θ, and if the regularity condition
holds, then the Fisher information may also be written as^{[8]}
Thus Fisher information is the negative of the expectation under of the second derivative of the log of f with respect to θ. Information may thus be seen to be a measure of the "sharpness" of the support curve near the maximum likelihood estimate of θ. A "blunt" support curve (one with a shallow maximum) would have a low negative expected second derivative, and thus low information; while a sharp one would have a high negative expected second derivative and thus high information.
Information is additive, in that the information yielded by two independent experiments is the sum of the information from each experiment separately:
This result follows from the elementary fact that if random variables are independent, the variance of their sum is the sum of their variances. Hence the information in a random sample of size n is n times that in a sample of size 1 (if observations are independent).
The information provided by a sufficient statistic is the same as that of the sample X. This may be seen by using Neyman's factorization criterion for a sufficient statistic. If T(X) is sufficient for θ, then
for some functions g and h. See sufficient statistic for a more detailed explanation. The equality of information then follows from the following fact:
which follows from the definition of Fisher information, and the independence of h(X) from θ. More generally, if T = t(X) is a statistic, then
with equality if and only if T is a sufficient statistic.
The CramérRao inequality states that the inverse of the Fisher information is a lower bound on the variance of any unbiased estimator of θ.
Van Trees (1968) and Frieden (2004) provide the following method of deriving the CramérRao bound, a result which describes use of the Fisher information, informally:
Consider an unbiased estimator . Mathematically, we write
The likelihood function f(X;θ) describes the probability that we observe a given sample x given a known value of θ. If f is sharply peaked, it is easy to intuit the "correct" value of θ given the data, and hence the data contains a lot of information about the parameter. If the likelihood f is flat and spreadout, then it would take many, many samples of X to estimate the actual "true" value of θ. Therefore, we would intuit that the data contain much less information about the parameter.
Now, we differentiate the unbiasedness condition above to get
We now make use of two facts. The first is that the likelihood f is just the probability of the data given the parameter. Since it is a probability, it must be normalized, implying that
Second, we know from basic calculus that
Using these two facts in the above let us write
Factoring the integrand gives
If we square the equation, the CauchySchwarz inequality lets us write
The rightmost factor is defined to be the Fisher Information
The leftmost factor is the expected meansquared error of the estimator θ, since
Notice that the inequality tells us that, fundamentally,
In other words, the precision to which we can estimate θ is fundamentally limited by the Fisher Information of likelihood function.
A Bernoulli trial is a random variable with two possible outcomes, "success" and "failure", with "success" having a probability of θ. The outcome can be thought of as determined by a coin toss, with the probability of obtaining a "head" being θ and the probability of obtaining a "tail" being 1 − θ.
The Fisher information contained in n independent Bernoulli trials may be calculated as follows. In the following, A represents the number of successes, B the number of failures, and n = A + B is the total number of trials.
(1) defines Fisher information. (2) invokes the fact that the information in a sufficient statistic is the same as that of the sample itself. (3) expands the log term and drops a constant. (4) and (5) differentiate with respect to θ. (6) replaces A and B with their expectations. (7) is algebra.
The end result, namely,
is the reciprocal of the variance of the mean number of successes in n Bernoulli trials, as expected (see last sentence of the preceding section).
When there are N parameters, so that θ is a Nx1 vector then the Fisher information takes the form of an NxN matrix, the Fisher Information Matrix (FIM), with typical element:
The FIM is a NxN positive semidefinite symmetric matrix, defining a Riemannian metric on the Ndimensional parameter space, thus connecting Fisher information to differential geometry. In that context, this metric is known as the Fisher information metric, and the topic is called information geometry.
If the following regularity condition is met:
then the Fisher Information Matrix may also be written as:
We say that two parameters θ_{i} and θ_{j} are orthogonal if the element of the ith row and jth column of the Fisher Information Matrix is zero. Orthogonal parameters are easy to deal with in the sense that their maximum likelihood estimates are independent and can be calculated separately. When dealing with research problems, it is very common for the researcher to invest some time searching for an orthogonal parametrization of the densities involved in the problem.
The FIM for a Nvariate multivariate normal distribution has a special form. Let and let Σ(θ) be the covariance matrix. Then the typical element , 0 ≤ m, n < N, of the FIM for is:
where denotes the transpose of a vector, tr(..) denotes the trace of a square matrix, and:
Note that a special, but very common case is the one where Σ(θ) = Σ, a constant. Then
In this case the Fisher information matrix may be identified with the coefficient matrix of the normal equations of least squares estimation theory.
The Fisher information depends on the parametrization of the problem. If θ and η are two scalar parametrizations of an estimation problem, such that θ = h(η) and h is a differentiable function, then
where and are the Fisher information measures of η and θ, respectively.^{[9]}
In the vector case, suppose and are kvectors which parametrize an estimation problem, and suppose that where is continuously differentiable. Then,^{[10]}
where the (i,j)th element of the Jacobian matrix is defined by
In information geometry, this is seen as a change of coordinates on a Riemannian manifold, and the intrinsic properties of curvature are unchanged under different parametrization.
Fisher information is widely used in optimal experimental design. Because of the reciprocity of estimatorvariance and Fisher information, minimizing the variance corresponds to maximizing the information.
When the linear (or linearized) statistical model has several parameters, the mean of the parameterestimator is a vector and its variance is a matrix. The inverse matrix of the variancematrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Using statistical theory, statisticians compress the informationmatrix using realvalued summary statistics; being realvalued functions, these "information criteria" can be maximized.
Traditionally, statisticians have evaluated estimators and designs by considering some summary statistic of the covariance matrix (of a meanunbiased estimator), usually with positive real values (like the determinant or matrix trace). Working with positive realnumbers brings several advantages: If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone). For several parameters, the covariancematrices and informationmatrices are elements of the convex cone of nonnegativedefinite symmetric matrices in a partially ordered vector space, under the Loewner (Löwner) order. This cone is closed under matrixmatrix addition, under matrixinversion, and under the multiplication of positive realnumbers and matrices. An exposition of matrix theory and the Loewnerorder appears in Pukelsheim.
The traditional optimalitycriteria are the informationmatrix's invariants; algebraically, the traditional optimalitycriteria are functionals of the eigenvalues of the (Fisher) information matrix: see optimal design.
In Bayesian statistics, the Fisher information is used to calculate the Jeffreys prior, which is a standard, noninformative prior for continuous distribution parameters.^{[11]}
Other measures employed in information theory:
