Hong Life CalculatorsUnderstanding the A/B test significance calculator — concept and principle
This calculator checks whether the difference in conversion rate between two versions of a page, email or feature is larger than random noise would explain. It reports the relative lift of B over A, the z statistic and a two-sided p-value from a two-proportion z-test, and whether the result is significant at the 95% level.
How it is calculated
- Conversion rates: p_A = conversions A ÷ visitors A, p_B = conversions B ÷ visitors B.
- Lift = (p_B − p_A) ÷ p_A × 100%.
- Pooled rate: p = (c_A + c_B) ÷ (n_A + n_B).
- z = (p_B − p_A) ÷ √[p(1 − p)(1/n_A + 1/n_B)].
- p-value (two-sided) = 2 × (1 − Φ(|z|)), where Φ is the standard normal CDF.
- Significant at 95% if p < 0.05.
The principle
Under the null hypothesis both versions share the same true conversion rate, so the best estimate of it is the pooled rate. The difference of two sample proportions then has a standard error of √[p(1 − p)(1/n_A + 1/n_B)], and for large samples the difference divided by that standard error follows an approximately standard normal distribution (central limit theorem). The p-value is the probability of seeing a gap at least this large in either direction if there were truly no difference.
Worked example
- Inputs: A has 500 conversions from 10,000 visitors; B has 560 from 10,000.
- p_A = 5.0%, p_B = 5.6%. Lift = 0.6 ÷ 5.0 = 12%.
- Pooled p = 1,060 ÷ 20,000 = 0.053.
- Standard error = √(0.053 × 0.947 × 0.0002) ≈ 0.0031683.
- z = 0.006 ÷ 0.0031683 ≈ 1.893752976.
- p-value = 2 × (1 − Φ(1.8938)) ≈ 0.058257806, which is above 0.05, so significant at 95%: no.
| Visitors B | p-value (two-sided) |
|---|---|
| 2,500 | 0 |
| 5,000 | 0 |
| 7,500 | 1.292322e-11 |
| 10,000 | 0.058257806 |
| 12,500 | 0.0673550875 |
| 15,000 | 0.0000011202 |
| 20,000 | 0 |
How to read the result
A 12% lift sounds large, but with these sample sizes it could plausibly arise by chance about 6% of the time. That is not proof that B is no better; it means the test has not yet collected enough evidence. A p-value is not the probability that B is better, and significance says nothing about whether the lift is big enough to matter. Look at a confidence interval for the difference and decide on a minimum worthwhile effect before the test.
Common mistakes
- Checking results daily and stopping as soon as p dips below 0.05 ("peeking"), which inflates false positives.
- Not fixing the sample size in advance with a power calculation.
- Testing many variants or metrics and reporting only the best one.
Frequently asked questions
Is 0.058 "almost significant"?
It is weak evidence. Report it honestly and, if the effect matters, run the test longer with a pre-planned sample size.
Should I use a one-sided test?
Only if you decided before the test that you would ignore a result in the other direction. Switching after seeing the data invalidates the p-value.
Why use the pooled rate?
Pooling is correct under the null hypothesis of equal rates, which is what the p-value tests. Confidence intervals usually use the unpooled standard error.
Sources: A. Agresti, Categorical Data Analysis, 3rd ed., Wiley, 2013, Section 3.1 · R. Kohavi, D. Tang and Y. Xu, Trustworthy Online Controlled Experiments, Cambridge University Press, 2020