Ad Experiment Power Calculator

Setup

Existing sales — mean

Existing sales — standard deviation (SD)

Campaign spending per person

True lift (% increase in sales)

Gross margin

▴ Advanced settings

Significance level α (two-sided)

Target power

True average treatment effect (ATE)

True ROI

ATE / SD (i.e. Cohen’s d)

Required n per group for target power

Rule of thumb (16/d²)

Sample sizes

Treatment

Control

Power for testing the null of no effect (i.e. ROI = -100%).

Power with your n

Minimum detectable effect (MDE) with your n

SE of treatment effect

SE of ROI estimate

95% CI width for ROI

Power to detect ROI > 0

Even if we can detect some effect of ads, we often need a larger experiment size to get a precise ROI estimate.