Statistical Power

Statistical power is the probability that an experiment, if a real effect exists, will actually detect it; the usual target is 80% or higher.

Significance protects against one kind of error (declaring an effect when there is none); statistical power protects against the symmetric error (missing an effect that is really there). An underpowered experiment is dangerous because it returns 'not significant' and the team wrongly thinks the change had no effect, when the experiment simply lacked the eyes to see it. Power depends on three things: sample size, the size of the effect you are chasing, and the spread of the data; this is why power must be computed before the experiment to be sure it is worth running. Running an underpowered experiment wastes traffic and time, because even if your change is genuinely good, there is a high chance the experiment misses it.