Top topics
Wikis
A statistical hypothesis test is a method of making statistical decisions using experimental data. In statistics, a result is called statistically significant if it is unlikely to have occurred by chance. The phrase "test of significance" was coined by Ronald Fisher: "Critical tests of this kind may be called tests of significance, and when such tests are available we may discover whether a second sample is or is not significantly different from the first."[1]
Hypothesis testing is sometimes called confirmatory data analysis, in contrast to exploratory data analysis. In frequency probability, these decisions are almost always made using null-hypothesis tests; that is, ones that answer the question Assuming that the null hypothesis is true, what is the probability of observing a value for the test statistic that is at least as extreme as the value that was actually observed?[2] One use of hypothesis testing is deciding whether experimental results contain enough information to cast doubt on conventional wisdom.
Statistical hypothesis testing is a key technique of frequentist statistical inference, and is widely used, but also much criticized. The main direct alternative to statistical hypothesis testing is Bayesian inference. However, other approaches to reaching a decision based on data are available via decision theory and optimal decisions.
The critical region of a hypothesis test is the set of all outcomes which, if they occur, will lead us to decide that there is a difference. That is, cause the null hypothesis to be rejected in favor of the alternative hypothesis. The critical region is usually denoted by C.
Contents |
A statistical test procedure is comparable to a trial. A defendant stands trial and is considered innocent as long as his guilt is not proven. The prosecutor tries to prove the guilt of the defendant. Only when there is enough charging evidence the defendant is condemned.
In the start of the procedure there are two hypotheses H0: "the defendant is innocent", and H1: "the defendant is guilty". The first one is called null hypothesis, and is for the time being accepted. The second one is called alternative (hypothesis). It is the hypothesis one tries to prove.
In good law practice one doesn't want to condemn an innocent defendant. That's why the hypothesis of innocence is only rejected when an error is very unlikely. Such an error is called error of the first kind (i.e. the condemnation of an innocent person), and the occurrence of this error is controlled to be seldom. As a consequence of this asymmetric behaviour, the error of the second kind (setting free a guilty person), is often rather large.
A person is tested for clairvoyance. He is 25 times shown the backside of a randomly chosen play card and asked which suit it is. The number of hits is called X.
As we try to prove his clairvoyance, for the time being the null hypothesis is the person is not clairvoyant. The alternative is of course: the person is (more or less) clairvoyant
If the null hypothesis is valid, the only thing the test person can do is guessing. For every card the probability of guessing right is 1/4. If the alternative is valid the test person will predict the suit right with probability greater than 1/4. We will call the probability of guessing right p. The hypotheses then are:
and
When the test person correctly predict all 25 cards, we will consider him clairvoyant and hence reject the null hypothesis. And also with 24 or 23 hits. With 5 or 6 hits on the other hand, there is no cause considering him so. But what about 12 hits, or with 17? What is the critical number c of hits, from where on it is hard to believe, they are just coincidental?
How do we determine the critical value c? It is obvous that with the choice c=25 (i.e. we only accept clairvoyance when all cards are predicted right) we're more critical than with c=10. In the first case almost no one will be recognised to be clairvoyant, in the second case some more.
In practice one decides how critical one will be, i.e. how often one accepts an error of the first kind.
With c = 25 the probability of such an error is:
hence very small. It is the probability of randomly guessing right all 25 times.
Less critical, with c = 10, gives:
a much greater probability.
Before the test is actually performed, the probability of an error of the first kind is determined, typically values
between 1 % and 5 %. Depending on this - we choose a value of 1% - the critical value c is calculated in such a way that:
From all the numbers c, with this property, we choose the smallest, in order to minimize the probability of an error of the second kind. In this example we find: c = 12.
As an example, consider determining whether a suitcase contains some radioactive material. Placed under a Geiger counter, it produces 10 counts per minute. The null hypothesis is that no radioactive material is in the suitcase and that all measured counts are due to ambient radioactivity typical of the surrounding air and harmless objects. We can then calculate how likely it is that we would observe 10 counts per minute if the null hypothesis were true. If the null hypothesis predicts (say) on average 9 counts per minute and a standard deviation of 1 count per minute, then we say that the suitcase is compatible with the null hypothesis (this does not guarantee that there is no radioactive material, just that we don't have enough evidence to suggest there is). On the other hand, if the null hypothesis predicts 3 counts per minute and a standard deviation of 1 count per minute, then the suitcase is not compatible with the null hypothesis, and there are likely other factors responsible to produce the measurements.
The test described here is more fully the null-hypothesis statistical significance test. The null hypothesis represents what we would believe by default, before seeing any evidence. Statistical significance is a possible finding of the test, declared when the observed sample is unlikely to have occurred by chance if the null hypothesis were true. The name of the test describes its formulation and its possible outcome. One characteristic of the test is its crisp decision: to reject or not reject the null hypothesis. A calculated value is compared to a threshold, which is determined from the tolerable risk of error.
Hypothesis testing is defined by the following general procedure:
It is important to note the philosophical difference between accepting the null hypothesis and simply failing to reject it. The "fail to reject" terminology highlights the fact that the null hypothesis is assumed to be true from the start of the test; if there is a lack of evidence against it, it simply continues to be assumed true. The phrase "accept the null hypothesis" may suggest it has been proved simply because it has not been disproved, a logical fallacy known as the argument from ignorance. Unless a test with particularly high power is used, the idea of "accepting" the null hypothesis may be dangerous. Nonetheless the terminology is prevalent throughout statistics, where its meaning is well understood.
The following definitions are mainly based on the exposition in the book by Lehmann and Romano:[3]
The direct interpretation is that if the p-value is less than the required significance level, then we say the null hypothesis is rejected at the given level of significance. Criticism on this interpretation can be found in the corresponding section.
In the table below, the symbols used are defined at the bottom of the table. Many other tests can be found in other articles.
| Name | Formula | Assumptions or notes | |||
|---|---|---|---|---|---|
| One-sample z-test |  |
(Normal population or n > 30)
and σ known. (z is the distance from the mean in relation to the standard deviation of the mean). For non-normal distributions it is possible to calculate a minimum proportion of a population that falls within k standard deviations for any k (see: Chebyshev's inequality). |
|||
| Two-sample z-test |  |
Normal population and independent observations and σ1 and σ2 are known | |||
| One-sample t-test | 
|
(Normal population or n > 30)
and σ unknown. For non-normal populations, n should be large enough to ensure both that dist. of mean is close to normal and that s is a good estimate of σ. |
|||
| Paired t-test | 
|
(Normal population of differences or n > 30) and σ unknown | |||
| Two-sample pooled t-test, equal variances* |  |
(Normal populations or n1 + n2 > 40) and independent observations and σ1 = σ2 and σ1 and σ2 unknown | |||
| Two-sample unpooled t-test, unequal variances* |  |
(Normal populations or n1 + n2 > 40) and independent observations and σ1 ≠ σ2 and σ1 and σ2 unknown | |||
| One-proportion z-test |  |
n .p0 > 10 and n (1 − p0) > 10 and it is a SRS (Simple Random Sample). | |||
| Two-proportion z-test, pooled | 
|
n1 p1 > 5 and n1(1 − p1) > 5 and n2 p2 > 5 and n2(1 − p2) > 5 and independent observations | |||
| Two-proportion z-test, unpooled |  |
n1 p1 > 5 and n1(1 − p1) > 5 and n2 p2 > 5 and n2(1 − p2) > 5 and independent observations | |||
| One-sample chi-square test |  |
One of the following
• All expected counts are at least 5 • All expected counts are > 1 and no more that 20% of expected counts are less than 5 |
|||
| *Two-sample F test for equality of variances |  |
Arrange so 
> 
and reject H0 for F >
F(α / 2,n1 − 1,n2
− 1)[7] |
|||
Definitions of symbols:
In general, the subscript 0 indicates a value taken from the null hypothesis, H0, which should be used as much as possible in constructing its test statistic. |
|||||
Hypothesis testing is largely the product of Ronald Fisher, Jerzy Neyman, Karl Pearson and (son) Egon Pearson. Fisher was an agricultural statistician who emphasized rigorous experimental design and methods to extract a result from few samples assuming Gaussian distributions. Neyman (who teamed with the younger Pearson) emphasized mathematical rigor and methods to obtain more results from many samples and a wider range of distributions. Modern hypothesis testing is an (extended) hybrid of the Fisher vs Neyman/Pearson formulation, methods and terminology developed in the early 20th century.
The following example is summarized from Fisher.[8] Fisher thoroughly explained his method in a proposed experiment to test a Lady's claimed ability to determine the means of tea preparation by taste. The article is less than 10 pages in length and is notable for its simplicity and completeness regarding terminology, calculations and design of the experiment. The example is loosely based on an event in Fisher's life. The Lady proved him wrong.[9]
If and only if the 8 trials produced 8 successes was Fisher willing to reject the null hypothesis – effectively acknowledging the Lady's ability with > 98% confidence (but without quantifying her ability). Fisher later discussed the benefits of more trials and repeated tests.
Statistical hypothesis testing plays an important role in the whole of statistics and in statistical inference. For example, Lehmann (1992) in a review of the fundamental paper by Neyman and Pearson (1933) says: "Nevertheless, despite their shortcomings, the new paradigm formulated in the 1933 paper, and the many developments carried out within its framework continue to play a central role in both the theory and practice of statistics and can be expected to do so in the foreseeable future".
Some statisticians have commented that pure "significance testing" has what is actually a rather strange goal of detecting the existence of a "real" difference between two populations. In practice a difference can almost always be found given a large enough sample. The typically more relevant goal of science is a determination of causal effect size. The amount and nature of the difference, in other words, is what should be studied.[10] Many researchers also feel that hypothesis testing is something of a misnomer. In practice a single statistical test in a single study never "proves" anything.[11]
Rejection of the null hypothesis at some effect size has no bearing on the practical significance of the observed effect size. A statistically significant finding may not be relevant in practice due to other, larger effects of more concern, whilst a true effect of practical significance may not appear statistically significant if the test lacks the power to detect it. Appropriate specification of both the hypothesis and the test of said hypothesis is therefore important to provide inference of practical utility.
Little criticism of the technique appears in introductory statistics texts. Criticism is of the application, or of the interpretation, rather than of the method.
Criticism of null-hypothesis significance testing is available in other articles (for example "Statistical significance") and their references. Attacks and defenses of the null-hypothesis significance test are collected in Harlow et al..[12]
The original purposes of Fisher's formulation, as a tool for the experimenter, was to plan the experiment and to easily assess the information content of the small sample. There is little criticism, Bayesian in nature, of the formulation in its original context.
In other contexts, complaints focus on flawed interpretations of the results and over-dependence/emphasis on one test.
Numerous attacks on the formulation have failed to supplant it as a criterion for publication in scholarly journals. The most persistent attacks originated from the field of Psychology. After review, the American Psychological Association did not explicitly deprecate the use of null-hypothesis significance testing, but adopted enhanced publication guidelines which implicitly reduced the relative importance of such testing.
The International Committee of Medical Journal Editors recognizes an obligation to publish negative (not statistically significant) studies under some circumstances.
The applicability of the null-hypothesis testing to the publication of observational (as contrasted to experimental) studies is doubtful.
Philosophical criticism to hypothesis testing includes consideration of borderline cases. Any process that produces a crisp decision from uncertainty is subject to claims of unfairness near the decision threshold. (Consider close election results.) The premature death of a laboratory rat during testing can impact doctoral theses and academic tenure decisions.
"... surely, God loves the .06 nearly as much as the .05"[13]
The statistical significance required for publication has no mathematical basis, but is based on long tradition.
"It is usual and convenient for experimenters to take 5% as a standard level of significance, in the sense that they are prepared to ignore all results which fail to reach this standard, and, by this means, to eliminate from further discussion the greater part of the fluctuations which chance causes have introduced into their experimental results."[8]
Ambivalence attacks all forms of decision making. A mathematical decision-making process is attractive because it is objective and transparent. It is repulsive because it allows authority to avoid taking personal responsibility for decisions.
Pedagogic criticism of the null-hypothesis testing includes the counter-intuitive formulation, the terminology and confusion about the interpretation of results.
"Despite the stranglehold that hypothesis testing has on experimental psychology, I find it difficult to imagine a less insightful means of transiting from data to conclusions."[14]
Students find it difficult to understand the formulation of statistical null-hypothesis testing. In rhetoric, examples often support an argument, but a mathematical proof "is a logical argument, not an empirical one". A single counterexample results in the rejection of a conjecture. Karl Popper defined science by its vulnerability to dis-proof by data. Null-hypothesis testing shares the mathematical and scientific perspective rather than the more familiar rhetorical one. Students expect hypothesis testing to be a statistical tool for illumination of the research hypothesis by the sample; it is not. The test asks indirectly whether the sample can illuminate the research hypothesis.
Students also find the terminology confusing. While Fisher disagreed with Neyman and Pearson about the theory of testing, their terminologies have been blended. The blend is not seamless or standardized. While this article teaches a pure Fisher formulation, even it mentions Neyman and Pearson terminology (Type II error and the alternative hypothesis). The typical introductory statistics text is less consistent. The Sage Dictionary of Statistics would not agree with the title of this article, which it would call null-hypothesis testing.[2] "...there is no alternate hypothesis in Fisher's scheme: Indeed, he violently opposed its inclusion by Neyman and Pearson."[15] In discussing test results, "significance" often has two distinct meanings in the same sentence; One is a probability, the other is a subject-matter measurement (such as currency). The significance (meaning) of (statistical) significance is significant (important).
There is widespread and fundamental disagreement on the interpretation of test results.
"A little thought reveals a fact widely understood among statisticians: The null hypothesis, taken literally (and that's the only way you can take it in formal hypothesis testing), is almost always false in the real world.... If it is false, even to a tiny degree, it must be the case that a large enough sample will produce a significant result and lead to its rejection. So if the null hypothesis is always false, what's the big deal about rejecting it?"[15] (The above criticism only applies to point hypothesis tests. If one were testing, for example, whether a parameter is greater than zero, it would not apply.)
"How has the virtually barren technique of hypothesis testing come to assume such importance in the process by which we arrive at our conclusions from our data?"[14]
Null-hypothesis testing just answers the question of "how well the findings fit the possibility that chance factors alone might be responsible."[2]
Null-hypothesis significance testing does not determine the truth or falseness of claims. It determines whether confidence in a claim based solely on a sample-based estimate exceeds a threshold. It is a research quality assurance test, widely used as one requirement for publication of experimental research with statistical results. It is uniformly agreed that statistical significance is not the only consideration in assessing the importance of research results. Rejecting the null hypothesis is not a sufficient condition for publication.
"Statistical significance does not necessarily imply practical significance!"[16]
Practical criticism of hypothesis testing includes the sobering observation that published test results are often contradicted. Mathematical models support the conjecture that most published medical research test results are flawed. Null-hypothesis testing has not achieved the goal of a low error probability in medical journals.[17][18]
Many authors have expressed a strong skepticism, sometimes labeled as postmodernism, about the general unreliability of statistical hypothesis testing to explain many social and medical phenomena. For example, modern statistics do not reliably link exposures of carcinogens to spatial-temporal patterns of cancer incidence. There is not a strong convention in statistical hypothesis testing to consider alternate units of scale. With temporal data the units chosen for temporal aggregation (hour, day, week, year, decade) can completely alter the trends and cycles. With spatial data, the units chosen for analysis (the modifiable areal unit problem) can alter or reverse relationships between variables. If the issue of analysis scale is ignored in a hypothesis test then skepticism about the results is justified.
Hypothesis testing is controversial when the alternative hypothesis is suspected to be true at the outset of the experiment, making the null hypothesis the reverse of what the experimenter actually believes; it is put forward as a straw man only to allow the data to contradict it. Many statisticians have pointed out that rejecting the null hypothesis says nothing or very little about the likelihood that the null is true. Under traditional null hypothesis testing, the null is rejected when the conditional probability P(Data as or more extreme than observed | Null) is very small, say 0.05. However, some say researchers are really interested in the probability P(Null | Data as actually observed) which cannot be inferred from a p-value: some like to present these as inverses of each other but the events "Data as or more extreme than observed" and "Data as actually observed" are very different. In some cases , P(Null | Data) approaches 1 while P(Data as or more extreme than observed | Null) approaches 0, in other words, we can reject the null when it's virtually certain to be true. For this and other reasons, Gerd Gigerenzer has called null hypothesis testing "mindless statistics"[19] while Jacob Cohen described it as a ritual conducted to convince ourselves that we have the evidence needed to confirm our theories.[20]
Bayesian statisticians normally reject the idea of null hypothesis testing, instead using various techniques in Bayesian inference. Given a prior probability distribution for one or more parameters, sample evidence can be used to generate an updated posterior distribution. In this framework, but not in the null hypothesis testing framework, it is meaningful to make statements of the general form "the probability that the true value of the parameter is greater than 0 is p". According to Bayes' theorem, we have:
thus P(Null | Data) may approach 1 while P(Data | Null) approaches 0 only when P(Null)/P(Data) approaches infinity, i.e. (for instance) when the a priori probability of the null hypothesis, P(Null), is also approaching 1, while P(Data) approaches 0: then P(Data | Null) is low because we have extremely unlikely data, but the Null hypothesis is extremely likely to be true.
A specific criticism of statistical hypothesis testing, proposed by Bruno de Finetti is that it is incoherent, meaning that if a bookie set gambling odds by statistical hypothesis testing, a gambler could construct a Dutch book, ensuring that the bookie always lost money. In financial terms, the bookie could be arbitraged. This argument was used to justify Bayesian probability instead, as it is coherent.
In 2002, a group of psychologists launched a new journal dedicated to experimental studies in psychology which support the null hypothesis. The Journal of Articles in Support of the Null Hypothesis (JASNH) was founded to address a scientific publishing bias against such articles. According to the editors,
The "File Drawer problem" is a problem that exists due to the fact that academics tend not to publish results that indicate the null hypothesis could not be rejected. This does not mean that the relationship they were looking for did not exist, but it means they couldn't prove it. Even though these papers can often be interesting, they tend to end up unpublished, in "file drawers."
Ioannidis has inventoried factors that should alert readers to risks of publication bias.[18]
Jones and Tukey suggested a modest improvement in the original null-hypothesis formulation to formalize handling of one-tail tests. Fisher ignored the 8-failure case (equally improbable as the 8-success case) in the example test involving tea, which altered the claimed significance by a factor of 2[21].
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
Search
= 
= sample mean of differences
= x/n = sample 
