Bayesian vs Frequentist

Bayesian vs frequentist are two schools of statistics that look at the same data differently: the frequentist weighs the probability of the data under a fixed hypothesis, the Bayesian updates the probability of the hypothesis as data arrives.

The two schools give two different answers to one practical question. The frequentist asks 'if there were no difference, how improbable was seeing this data', and its output is a p-value and a confidence interval; this is the default lens of most classic experimentation tools. The Bayesian asks 'given this data, what is the probability that version B is better', and its output is a more direct statement, easier for a product manager to grasp, but it requires setting a prior belief that skews the result if chosen badly. Neither is absolutely 'right'; the Bayesian usually makes continuous monitoring easier and is more intuitive on small data, while the frequentist is more traditional and more established in standard reporting. What matters is not switching an experiment mid-way from one school to the other to get the desired result.