Multivariate Testing

Multivariate testing is an experiment that changes several page elements at once in different combinations, so both each element's effect and the interaction between elements can be seen.

An A/B test changes one thing; a multivariate test measures several at once and all their combinations, its value being that it tells you whether a new headline and a new image together have an effect greater than their sum. But its cost is a combinatorial explosion: three elements with two states each means eight versions, and traffic must be split across all eight, so multivariate testing is only practical for high-traffic sites. The common trap is running a multivariate test on low traffic, where no combination reaches enough volume and the result is just noise. If traffic is limited, several sequential A/B tests are safer than one multivariate test.