When should an experiment be stopped?

At the sample size and duration computed before the run began, not at the first moment the result looks significant. Checking repeatedly and stopping at the most favourable point is itself a source of error, and because behaviour differs across days of the week the run should cover at least one full weekly cycle.

How does an experiment differ from a before-and-after comparison?

A before-and-after comparison credits the change with everything else that happened in the same window, from seasonality to campaigns to other releases. An experiment measures both groups at the same time under the same conditions, so those factors hit both groups and drop out of the result.

Experiment

Intermediate

Updated Jul 11, 2026 Shahin editorial 3 min read

PronunciationEN/ɪkˈspɛrɪmənt/

An experiment randomly splits users between two or more variants of a change so that the difference in an outcome metric can be attributed to the change itself.

Randomisation does the real work: it makes the groups comparable on everything the analyst never thought to measure, so the difference that remains is attributable to the variant. The result is read against a significance threshold, usually a p-value below 0.05, and that number states only how often data this extreme would appear if the variant had no effect at all. This is where the misreading happens: it is not the probability that the variant works, and a result above the threshold is not evidence that the two variants are equal, only that the test failed to separate them. Stopping the moment a dashboard turns significant pushes the false-positive rate far above the stated level, which is why the sample size and the run length are fixed before the experiment starts.

FAQ