How does impact analysis differ from an A/B test?

In an experiment the random split happens before release and the comparison runs on two groups at the same time under the same conditions, which gives the strongest causal claim available. Impact analysis is what remains once the change has shipped to everyone and randomisation is no longer possible, and its output is an estimate that depends on modelling assumptions rather than a direct measurement.

Impact Analysis

Advanced

Updated Jul 11, 2026 Shahin editorial 3 min read

PronunciationEN/ˈɪmpækt əˈnæləsɪs/

Impact analysis estimates how much a specific change moved a metric, by comparing what happened after it with what the metric would plausibly have done otherwise.

The hard part is the counterfactual, since the version of events without the change is never observed and has to be constructed. Three constructions are common: a randomised control group, which turns the analysis into an experiment; a comparable untouched segment measured over the same period, which is difference in differences; and a forecast built from the period before the change, which is the synthetic control family. A plain before-and-after comparison uses none of them and quietly credits the change with everything else that happened in the same week, including seasonality, campaigns and other releases. Two further traps are novelty, where a visible change lifts a metric for a few days simply because it is new, and shipping several changes at once, which makes the effect of any single one unrecoverable however the analysis is run.

FAQ