An A/B test compares two versions of a page, email, or offer to see which performs better, but the comparison is only meaningful if enough visitors have seen each version. With too few visitors, random variation swamps any real difference, and the test either declares a winner that is not really winning or misses a genuine improvement. The A/B Test Sample Size Calculator estimates how many visitors each version needs, but understanding its inputs helps you set a test that can actually answer your question.
Three inputs drive the calculation. The baseline conversion rate is the current rate, such as 10 percent of visitors buying or signing up. The minimum detectable effect is the smallest improvement worth detecting, such as a rise from 10 percent to 12 percent, a difference of 2 percentage points. That effect must be chosen from business value, not from hope: an improvement too small to matter is not worth the sample size needed to detect it, and an improvement chosen after seeing results is not a fair target. Power is the chance of detecting a real effect of that size if it exists, usually set at 80 percent, meaning a one-in-five chance of missing a real effect of the stated size. Significance level, usually 5 percent, is the chance of declaring a difference when none exists. These are conventions, and a test for a high-stakes decision may justify higher power or a stricter significance level, at the cost of a larger sample.
The relationship between effect size and sample size is steep, and it surprises most first-time testers. Halving the effect you want to detect roughly quadruples the required sample, because sample size grows with the inverse square of the effect. That is why a test designed to detect a 1-percentage-point improvement on a 10 percent baseline needs a far larger audience than a test designed to detect a 5-point improvement, and why small sites often cannot run finely targeted tests in a reasonable time. In that case, the honest options are to test a larger change, to test on a higher-traffic page, or to combine evidence over a longer period without peeking and stopping early, which would invalidate the result.
A worked example
Suppose a page converts at a baseline of 10 percent, and you want to detect a rise to 12 percent, a difference of 0.02, with the usual conventions. A common rough approximation for the sample per group is 16 multiplied by the average rate multiplied by one minus that rate, divided by the squared difference. The average rate between 10 and 12 percent is 11 percent, or 0.11, and one minus that is 0.89. Their product is 0.11 multiplied by 0.89, which is 0.0979. Multiply by 16 to get 1.5664. The squared difference is 0.02 multiplied by 0.02, which is 0.0004. Dividing 1.5664 by 0.0004 gives 3,916 visitors per group, or 7,832 visitors in total, because 3,916 multiplied by 2 equals 7,832. That approximation is a planning aid, not an exact power calculation, and a proper calculator uses the exact binomial or normal formulas with your chosen power and significance level, so treat 3,916 as an order-of-magnitude guide and use the calculator's figure for the final plan. If your site receives 500 visitors a day split between two versions, reaching 7,832 visitors takes 7,832 divided by 500, which is 15.66 days, or about 16 days, before any allowance for weekends or traffic variation.
That duration check is where many test plans fail. A sample size that looks modest on paper can mean weeks of running, during which prices, seasons, or campaigns change and contaminate the comparison. If the required duration exceeds the period over which conditions stay stable, redesign the test: choose a bigger effect, a higher-traffic surface, or a different question. Running a test for a few days and declaring a winner because the numbers looked good is not a shortcut; it is a different, invalid procedure, because early stopping on a favourable peek inflates the false-positive rate.
Common mistakes
The first mistake is starting a test without a sample size, and stopping when the result looks pleasing. The second mistake is changing the test mid-way, such as altering a headline or redirecting traffic, and still reporting the result as if conditions were constant. The third mistake is testing many variants at once without adjusting for the many chances to find a spurious winner. The fourth mistake is ignoring segmentation: an overall winner can hide a loser for an important segment, such as mobile visitors, so check major segments before rolling out. The fifth mistake is reporting a winner without its uncertainty: a 12 percent versus 10 percent result on small samples may be consistent with no real difference, and the confidence interval, not the headline rates, shows that.
Randomisation and isolation also matter. Visitors must be assigned randomly and consistently, so the same visitor sees the same version on return, and no other change, such as a price update or a campaign launch, should coincide with the test unless it affects both versions equally. Record the start and end dates, the traffic split, and the sample sizes, so the result can be audited later. A test whose method cannot be reconstructed is an anecdote, not evidence.
A practical routine
A routine that works for most A/B tests has five steps. First, state the baseline rate from recent, stable data. Second, choose the smallest effect worth detecting, from business value. Third, calculate the sample size per group and the expected duration at your traffic level. Fourth, run the test unchanged until the planned sample is reached, without peeking for a stopping decision. Fifth, report both rates, the difference, and its uncertainty, and check major segments before rolling out. For the companion significance check, the Statistical Power Calculator estimates detection chance, and the Research Sample Size Calculator estimates samples for surveys. For a percentage walkthrough, the guide Attendance Percentage Calculator Guide shows how rate calculations are checked step by step.
Sample size is decided before a test, not discovered during it. Choose the effect that matters, do the arithmetic, and let the planned sample, not impatience, end the test.