Statistical Significance

Statistical significance means the difference you see in the data is large enough that it most likely did not come from chance — not that the difference is necessarily important or large.

Statistical significance answers only one question: 'if there were no real difference, how likely was seeing data like this?' If that likelihood is low, the result is called significant. The biggest misunderstanding is equating significance with importance; on a very large sample even a tiny, worthless difference becomes significant, and conversely on a small sample an important difference may not. Another trap is peeking at the result repeatedly and stopping the moment it turns significant, which multiplies the chance of a false result. Significance must always be read beside the effect size and the confidence interval, so you know both that the difference is real and how large it is.