Impact Report

An impact report is a report that tries to measure the effect of a specific action or change on a target metric and separate it from background variation.

An impact report answers 'did what we did actually make a difference', but here lies analysis's most dangerous bias: attributing every change after an action to that action. An honest impact report needs a comparison baseline — either a group that did not see the action, or the pre-action trend showing where the metric was heading without intervention. Without that baseline, an impact report is just a 'before and after' that charges seasonality and simultaneous events to the action. The difference between an impact report and plain correlation is this effort to isolate cause, though in observational data that isolation is never complete.