Confidence Interval

A confidence interval is a range of values that, instead of a single number, shows the likely range of a metric's true value at a given confidence level.

A confidence interval is the antidote to reading one number as certain truth: it says 'a 5% conversion rate' really means 'somewhere between 4% and 6%'. The interval's width is itself information; a narrow interval means plenty of data and a precise estimate, a wide interval means a small sample and high uncertainty. Its most important use is comparison: if two versions' confidence intervals overlap heavily, you cannot confidently say one is better, even if their central numbers differ. The common error is reporting the middle number with no interval, which leads the reader to a certainty the data does not hold — especially on a small sample, this is misleading.