A/B Test Sample Size Calculator

Understanding the A/B test sample size calculator — concept and principle

Before you start an A/B test you need to know how many visitors each version must see. Too few and a real improvement can be lost in random noise; too many and you waste traffic and time. This calculator gives the visitors needed per variant for a test that compares two conversion rates.

How it is calculated

The calculator uses the standard sample size formula for a two-sided two-proportion z-test:

n = (z₁₋α/₂ · √(2p̄(1 − p̄)) + z₁₋β · √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)²

• p₁ is the baseline conversion rate; p₂ = p₁ × (1 + relative lift) is the rate you want to be able to detect.
• p̄ = (p₁ + p₂) ÷ 2 is the average of the two rates.
• z₁₋α/₂ comes from the significance level α (1.96 for α = 5%), and z₁₋β from the power (0.84 for 80%).
• n is rounded up to a whole visitor; the total is 2n.

The principle

Each observed conversion rate wobbles around its true value. The size of that wobble shrinks with the square root of the sample. The formula picks n so that two things hold at once: if there is no real difference, a false alarm happens only α of the time, and if the true lift is as large as your target, the test detects it with probability equal to the power. Because the gap (p₂ − p₁) is squared in the denominator, halving the lift you want to detect roughly quadruples the required sample.

Worked example

1. p₁ = 0.04, p₂ = 0.04 × 1.15 = 0.046 (4.6%), p̄ = 0.043.
2. z₁₋α/₂ = 1.95996, z₁₋β = 0.84162.
3. First term: 1.95996 × √(2 × 0.043 × 0.957) = 0.56228. Second term: 0.84162 × √(0.04 × 0.96 + 0.046 × 0.954) = 0.24142.
4. (0.56228 + 0.24142)² = 0.64594; divided by (0.006)² = 0.000036 gives 17,942.7.
5. Rounded up: 17,943 visitors per variant, 35,886 in total.

How to read the result

Divide the total by your daily eligible traffic to estimate the test duration, and run it for whole weeks so weekday effects even out. If the duration is unrealistic, test a bolder change (larger lift) or a higher-traffic page rather than cutting the sample. Low baseline rates need much larger samples.

Common mistakes

• Entering an absolute lift (4% to 5%) when the field expects a relative one (25%).
• Stopping as soon as the result looks significant, which inflates false positives.
• Splitting traffic unevenly or across more than two variants without adjusting the sample.

Frequently asked questions

What power should I use?

80% is the common default; 90% gives more protection against missing a real effect at the cost of about a third more traffic.

Does the sample count visitors or sessions?

Use the unit you randomise on. If users are assigned to a variant, count users, not page views.

Why is my required sample so large?

Small baseline rates and small lifts both push n up sharply, because the difference you are measuring is tiny relative to the noise.

Sources: Fleiss, Levin & Paik, Statistical Methods for Rates and Proportions, 3rd ed. (Wiley, 2003) · Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020)