Holdout Group

A holdout group is a share of users deliberately kept out of a change, or a whole set of changes, for a period, so the long-term cumulative effect of those changes can be measured.

A holdout is in the same family as a control group but differs in span and purpose: a control group is for one short experiment, while a holdout is a group that sees none of the team's changes for weeks or months, to reveal how much value the team's combined work created over the long run. Its value is that small effects, each negligible alone but accumulating together, become visible only against an untouched baseline. Its cost is clear too: some users are deliberately kept from improvements, so the holdout is kept small and its span limited.