A/B Test Sample Size for AI Ad Copy (several variants, two-proportion test)

How it works

Each AI variant is compared with the current ad (the control) using a two-sided test of two proportions. Sample per version n = [z₁₋α/2 √(2p̄(1−p̄)) + z₁₋β √(p₁(1−p₁) + p₂(1−p₂))]² ÷ (p₂ − p₁)², where p₁ is today’s rate, p₂ = p₁ × (1 + lift) and p̄ is their average. With several variants the Bonferroni correction divides the significance level by the number of variants so that the chance of at least one false winner stays near the level you chose. Total sample = n × (variants + 1); days = total ÷ daily traffic split evenly across versions.

⚠ Estimates for information only — not financial advice.

FAQ

Why do more variants need more traffic?

Every extra variant needs its own sample, and with the Bonferroni correction each comparison must pass a stricter test. Test a few strong AI variants rather than many weak ones.

Can I stop as soon as one variant looks better?

Stopping early when a result first looks significant inflates false winners. Decide the sample size in advance and run the test to the end, or use a method designed for sequential testing.

Checked 10 Nov 2026. Sample size for comparing two proportions (Fleiss, Levin & Paik, Statistical Methods for Rates and Proportions, 3rd ed., Wiley 2003, chapter 4, formula without continuity correction); Bonferroni adjustment for multiple comparisons.