Sample Size

Sample size is the number of observations an experiment or analysis needs to distinguish a real effect from chance with sufficient confidence.

Sample size is the bridge between wish and statistics: the smaller the effect you are chasing, the larger the sample needed for it to rise above the noise. The relationship is non-linear; to see an effect half the size, you need roughly four times the sample, which is what makes testing subtle changes on low-traffic sites practically impossible. The common error is drawing conclusions from a small sample: with a hundred visits, the conversion rate swings several points a day and any reading of it is a deception. Sample size must be computed before the experiment from the minimum detectable effect, the confidence level and the statistical power — not after seeing the data.