A/B Test Significance (p-value) Calculator

Understanding the A/B test significance calculator — concept and principle

After an A/B test ends, you need to know whether the difference in conversion rates is real or could easily be chance. This calculator runs a two-sided two-proportion z-test and returns the p-value, the z-score, both conversion rates and the relative lift of B over A.

How it is calculated

• Conversion rates: p_A = conversions A ÷ visitors A, p_B = conversions B ÷ visitors B.
• Pooled rate: p = (conversions A + conversions B) ÷ (visitors A + visitors B).
• Standard error: SE = √(p(1 − p)(1/n_A + 1/n_B)).
• z = (p_B − p_A) ÷ SE, and p-value = 2 × (1 − Φ(|z|)), where Φ is the standard normal cumulative distribution.
• Relative lift = p_B ÷ p_A − 1. The result is "significant at 95%" when p < 0.05.

The principle

The test starts from the null hypothesis that A and B convert at the same rate. Under that assumption both groups share one pooled rate, and the difference between two sample rates follows an approximately normal distribution with standard error SE. The z-score says how many standard errors apart the observed rates are. The p-value is the probability of seeing a gap at least that large, in either direction, if there were truly no difference. The normal approximation works well when each group has a reasonable number of conversions and non-conversions.

Worked example

1. p_A = 200 ÷ 5,000 = 4.0%; p_B = 245 ÷ 5,000 = 4.9%.
2. Pooled p = 445 ÷ 10,000 = 0.0445.
3. SE = √(0.0445 × 0.9555 × (1/5,000 + 1/5,000)) = 0.004124.
4. z = 0.009 ÷ 0.004124 = 2.18.
5. p-value = 2 × (1 − Φ(2.18)) ≈ 0.029, shown as 0.03, below 0.05, so the result is significant at 95%. Relative lift = 4.9 ÷ 4.0 − 1 = 22.5%.

How to read the result

A small p-value means the data would be surprising if there were no real difference. It does not tell you the size of the effect or the probability that B is better. Look at the lift and its business value together with the p-value. A p-value of 0.03 at 95% confidence still leaves a meaningful chance of a false positive across many tests, so confirm important wins.

Common mistakes

• Checking results daily and stopping at the first p < 0.05.
• Reading 1 − p as "the probability B is better".
• Comparing tiny samples where the normal approximation is weak.

Frequently asked questions

What does "not significant" mean?

The data are compatible with no difference, but the test may simply have lacked enough visitors. It is not proof that A and B are equal.

Is 95% the only valid confidence level?

No. It is a convention. Decide the level before the test, based on the cost of a wrong decision.

Why use a pooled rate?

Under the null hypothesis both groups share the same rate, so pooling gives the best estimate of it for the standard error.

Sources: Agresti, Categorical Data Analysis, 3rd ed. (Wiley, 2013) · Abramowitz & Stegun, Handbook of Mathematical Functions, formula 26.2.17 (normal distribution approximation) · Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020)