Significance Threshold

A significance threshold is the agreed line you set before an experiment to decide how improbable a result must be, under the 'no difference' assumption, to count as significant.

A significance threshold is the number actually compared against the p-value; the most common is 5%, meaning you accept that, at worst, one time in twenty you will declare a difference real when it is not. Choosing this threshold is a decision about risk, not a law of nature: where the cost of a mistake is high (a medical or financial decision) the threshold is made stricter. The common error is moving the threshold after seeing the result so the data 'passes'; the threshold must be locked before the experiment. Another key point is multiple testing: if you test twenty things at once, on average one crosses a 5% threshold by chance, so with more tests the threshold must be tightened.