Confidence Level

A confidence level is the percentage stating what share of the computed intervals would contain the true value if a measurement were repeated many times; the most common is 95%.

A confidence level is the flip side of a significance threshold: a 5% threshold means a 95% confidence level. Its exact meaning is subtle and often misstated; the 95% does not refer to one particular interval but to the method: if the experiment were repeated a hundred times, about 95 of the hundred intervals would contain the true value. Raising the confidence level is not free: 99% confidence needs a wider interval and a larger sample, so there is a constant trade-off between 'being surer' and 'being more precise'. The confidence level should be chosen to match the cost of a mistake, not always set to the highest.