Runs in your browser. Nothing you enter is uploaded or stored.
To find out whether your A/B test result is real, enter the visitors and conversions for each version into the free A/B Test Significance Calculator above. It runs a two-proportion z-test and shows the conversion rates, the relative lift, the p-value, a confidence interval for the difference, the statistical power and a sample ratio check that catches broken traffic splits. A second panel works out how many visitors you need per variant before you start. Paste a CSV to check up to 500 tests at once and download the results.
Last updated: 11 November 2026. Written by Saad Raza.
Key takeaways
- A p-value below 0.05 means a difference this large would show up less than 5 percent of the time if the two versions truly performed the same. It is not the probability that B is better.
- Decide the sample size before you start and stop when you reach it. Checking daily and stopping at the first significant result inflates false positives far above 5 percent.
- Small lifts need large samples. Detecting a 10 percent relative improvement on a 3 percent baseline takes about 53,000 visitors per variant at 95 percent confidence and 80 percent power.
- Always look at the confidence interval, not only the verdict. An interval that runs from almost zero to a large gain says the result is uncertain.
- If the traffic split is not what you intended, the result cannot be trusted. The sample ratio check shows this.
What significance tells you
In a split test you show version A to some visitors and version B to others and count how many convert. Chance alone will make the two rates differ a little, so you need a way to separate a real effect from luck. The two-proportion z-test does that. It pools both groups to estimate the rate you would expect if the versions were identical, measures how far apart the actual rates are in units of standard error, and turns that into a p-value.
The p-value is often misread. A result of 0.03 does not mean there is a 97 percent chance that B is better. It means that if there were no real difference, you would see a gap this big about 3 times in 100. That is good evidence, not proof, and it says nothing about how large the benefit is. The confidence interval answers the second question, giving a range of differences that fit your data.
How to use the A/B Test Significance Calculator
- Enter the number of visitors and the number of conversions for version A, the control, and version B, the variant.
- Choose your confidence level, usually 95 percent, and whether the test is two-sided (any difference) or one-sided (B better than A only). Two-sided is the safer default.
- Read the result: both conversion rates, the relative lift, the z-score, the p-value, the confidence interval and the verdict.
- Check the sample ratio line. If it is flagged, investigate the split before reading the result.
- To plan a test, open the sample size planner. Enter your baseline conversion rate and the smallest relative lift worth detecting.
- For many tests, paste a CSV with the columns
name,visitors_a,conversions_a,visitors_b,conversions_b. Up to 500 rows are calculated and can be downloaded.
Reading the output
| Output | What it means |
|---|---|
| Conversion rates | Conversions divided by visitors for each version |
| Relative lift | The change in B compared with A, as a percentage of A’s rate |
| z-score and p-value | How unusual the gap would be if both versions were truly equal |
| Confidence interval | The range of absolute differences consistent with the data |
| Power | The chance this sample would detect a difference of the size you observed |
| Sample ratio check | Whether the split between groups is close enough to what you planned |
| Verdict | B wins, A wins or no significant difference at your confidence level |
A worked example
A product page test sends 8,000 visitors to each version. Version A converts 320 of them, 4.00 percent, and version B converts 390, 4.875 percent. The relative lift is 21.9 percent. The z-score is 2.69 and the p-value is 0.0072, well below 0.05, so the verdict is that B wins. The 95 percent confidence interval for the difference runs from about 0.24 to 1.51 percentage points, which is a wide range: the true gain could be small or large. The tool reports power of about 77 percent for a difference this size, a reminder that a test of this scale would sometimes miss a real effect of that size.
Now the planning side. A shop with a 3 percent conversion rate wants to detect a 10 percent relative lift, from 3.00 to 3.30 percent. At 95 percent confidence and 80 percent power, the planner calculates 53,211 visitors per variant. If the page gets 2,000 visitors a day, split in two, that is about 53 days. Knowing this up front stops you from calling a winner after a week on 7,000 visitors.
Mistakes that produce false winners
- Peeking. Looking at the result every day and stopping when p dips under 0.05 will declare a winner far more often than 5 percent of the time. Fix the sample size or the end date first.
- Testing many things at once. Ten metrics or ten variants means one will look significant by luck. Name one primary metric in advance.
- Ending on a good week. Run tests for full weekly cycles so weekday and weekend behaviour are both included.
- Uneven traffic. A split that should be 50/50 but shows 52/48 on a large sample points to a technical fault, such as a redirect or a script that fails for one version.
- Counting visits instead of visitors. If one person can convert twice, count the number of converting visitors so the test assumptions hold.
Where to use it
Landing page headlines, button text, price displays, form length, email subject lines and ad creative all suit this test, as long as each visitor is counted once and you have enough of them. For paid campaigns, compare the winner’s economics in the ROAS and CPA calculator, since a higher conversion rate can still lose money if it comes from a more expensive audience. The conversion rate calculator gives the baseline and the interval for a single result, and the email subject line tester helps you choose which two lines to put in an email split.
Organic search has its own version of this problem. Title and meta changes affect click-through, but rankings and seasonality move at the same time, so a before-and-after comparison is weaker than a true split. The CTR benchmark calculator shows what click-through you should expect at each position, which helps you judge whether a new title is working. If you want help testing and improving the pages that bring in search traffic, see my SEO audit service.
Limits
The calculator assumes independent visitors, a fixed sample size and conversion as a yes or no outcome. It is not suited to revenue per visitor, which needs a different test, and it does not correct for running many tests or many variants. A significant result tells you the difference is unlikely to be chance, not that it will last: novelty effects fade, and audiences change. The sample size planner uses the standard normal approximation and may differ slightly from other tools. Up to 500 tests per CSV, processed in your browser; nothing is sent to a server.
Questions about the A/B Test Significance Calculator
What p-value counts as significant?
The usual cut-off is 0.05, which matches 95 percent confidence. Use a stricter level, such as 0.01, when a wrong decision is costly or hard to reverse.
How many visitors do I need for an A/B test?
It depends on your baseline rate and the smallest improvement you care about. Lower baselines and smaller lifts need more. Use the planner: 3 percent with a 10 percent relative lift needs about 53,000 per version.
Can I stop a test early if the result looks significant?
Not safely. Early stopping on a good result inflates false positives. Decide the sample size or end date in advance, or use a sequential method designed for early looks.
What does a sample ratio mismatch mean?
The traffic split differs from your plan by more than chance explains, so something is wrong with assignment or tracking. Fix the cause and rerun before you trust the result.
Should I use one-sided or two-sided?
Two-sided unless you have a firm reason to care only about an improvement. It tells you if B is worse as well as better.
If you want testing and search growth working together, see my SEO services or ask for a free audit. You can also browse the other free marketing and analytics tools.
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.
