A/B Test Calculator
An A/B test can look convincing long before the evidence is strong enough to support a decision. A few extra conversions in one version may be a meaningful improvement, or they may be normal variation in a small sample.
Use this calculator to compare two conversion rates. Enter the visitors and conversions for each version, choose the confidence level you decided to use, and review the difference rather than relying on the raw conversion rates alone.
How to read the result
The calculator reports the conversion rate for each variant and the difference between them in two ways. Absolute change is the percentage-point difference between Variant B and Variant A. Relative lift shows that difference relative to Variant A’s conversion rate.
The p-value comes from a two-sided two-proportion z-test. If the p-value is below the threshold for the confidence level you selected, the calculator marks the observed difference as statistically significant. That does not mean the winning version is automatically worth implementing. A small change can be statistically detectable but commercially unimportant.
The confidence interval shows a range of plausible values for the difference between B and A. A wide interval tells you that the estimate is still uncertain. If the interval crosses zero, the data is compatible with either version performing better.
When should you keep the test running?
Do not stop a test simply because one version is ahead today. Your testing window, traffic allocation and stopping rule should be decided before you start reviewing the result. Repeatedly checking a test and ending it as soon as the number looks favourable can increase the chance of a false conclusion.
This calculator also warns when one or both variants have very few conversions or non-conversions. The normal approximation used by the z-test becomes less dependable with very small counts, so more data may be needed.
What this calculator does not decide for you
Statistical significance answers a narrow question about the evidence in the observed data. It does not tell you whether the change is profitable, whether it will persist after rollout, whether the experiment was designed correctly, or whether outside changes affected the result.
Before acting on a result, check the size of the lift, revenue or lead quality, the duration of the experiment, traffic sources, device mix and any operational effects caused by the change.
Frequently Asked Questions
What confidence level should I use for an A/B test?
The calculator supports 90%, 95% and 99%. A 95% level is commonly used, but the right threshold depends on the cost of making the wrong decision and how the experiment was designed. Choose the threshold before you judge the result instead of changing it to make a result pass.
Does a 95% confidence level mean there is a 95% chance Variant B is better?
No. In this frequentist test, the confidence level does not give a direct probability that one variant is better. The test asks how compatible the observed difference is with the assumption that the underlying conversion rates are equal.
What is the difference between absolute lift and relative lift?
If Variant A converts at 5% and Variant B converts at 6%, the absolute change is 1 percentage point. The relative lift is 20% because the 1-point increase is 20% of the original 5% conversion rate.
Why can a large percentage lift still be statistically insignificant?
A large-looking lift can come from a small sample. For example, moving from one conversion to two conversions doubles the observed conversion count, but that is usually too little evidence to make a reliable decision. Sample size and the number of successes and failures matter as well as the percentage difference.
Can I use this for more than two variants?
No. This calculator compares two proportions. Tests with three or more variants need a method that accounts for the additional comparisons rather than running several independent A/B tests without adjustment.