Minimum Detectable Effect

Minimum detectable effect (MDE) is the smallest change in a metric that an experiment, with a given sample size and power, can tell apart from chance.

MDE is a contract you sign before the experiment: you say 'a change smaller than this does not matter to me, so I will set up the test to see exactly this size'. Its importance is that it ties three things together — the smaller the MDE, the larger the sample needed and the longer the experiment. The expensive mistake is not setting an MDE and then, after the experiment, saying the result 'was not significant'; in truth the experiment never had the power to see that small effect, and 'no effect' is confused with 'did not see the effect'. Setting an MDE is a business decision, not a statistical one: you must ask what the smallest improvement worth the change is, and design the experiment around it.