Statistical Power Calculator Guide: Can Your Study Detect an Effect

A study can fail in two ways. It can claim an effect that is not real, or it can miss an effect that is real. The first failure is controlled by the significance level, usually set at 5 percent. The second is controlled by statistical power, which is the probability that a study will detect an effect that genuinely exists. A study with low power is like a net with large holes: real effects slip through it.

The Statistical Power Calculator estimates power from effect size, sample size, and significance level, or works backwards to the sample size needed for a target power. This guide explains the four quantities involved, the usual target of 80 percent power, and a worked example. Note that this guide is educational only and is not a substitute for advice from a qualified statistician when designing a real study.

The four quantities that trade off

Power analysis involves four quantities, and fixing any three determines the fourth. The first is the effect size, which measures how large the effect is in standardised units. A small effect is harder to detect than a large one, just as a whisper is harder to hear than a shout. The second is the sample size: larger samples give more power, because they average out random noise. The third is the significance level, usually 0.05, which is the threshold for calling a result statistically significant. The fourth is power itself, the probability of detecting the effect.

The standard target for power is 80 percent. That means that if the effect is real, the study has an 80 percent chance of detecting it, and a 20 percent chance of missing it. The 20 percent miss rate is called the Type II error rate, denoted by the Greek letter beta, and power equals 1 minus beta. Eighty percent is a convention, not a law of nature: some fields use 90 percent for important studies, and pilot studies sometimes accept less, but 80 percent is the usual starting point.

These quantities trade off against each other. To detect a smaller effect, you need a larger sample. To achieve higher power, you need a larger sample. To use a stricter significance level, such as 0.01 instead of 0.05, you need a larger sample. Understanding these trade-offs is the main skill in power analysis, because it forces researchers to decide, before collecting data, what effect they care about and how sure they want to be.

Effect sizes in plain terms

Effect size is often measured with Cohen's d, which is the difference between two group means divided by their pooled standard deviation. A d of 0.2 is conventionally called small, 0.5 medium, and 0.8 large. These labels are rough guides: a small effect can matter enormously in medicine, where a treatment that helps slightly across millions of people saves many lives, while a large effect in a laboratory setting might be routine.

The key point is that effect size should come from the research question, not from optimism. Choosing an unrealistically large effect size makes the required sample look small, which makes the study look cheap, but then the study is underpowered for the effect that actually exists. Better sources for an expected effect size are previous studies, pilot data, and the smallest effect that would be practically meaningful. If a difference smaller than a certain size would not change any decision, there is little point designing a study to detect it.

Worked example: sample size for a medium effect

Suppose you plan a study comparing two groups, you expect a medium effect size of 0.5, you set the significance level at 0.05, and you want 80 percent power. How many participants do you need per group? A well-known approximation for a two-group comparison is that the sample size per group is about 16 divided by the square of the effect size. Here, the square of 0.5 is 0.25, and 16 divided by 0.25 is 64. So you need about 64 participants per group, or about 128 in total.

Check the arithmetic: 0.5 times 0.5 is 0.25, and 16 divided by 0.25 equals 64, because 0.25 times 64 is 16. The approximation is close to the exact calculation, which gives a similar figure in the low sixties per group, depending on whether the test is one-sided or two-sided. The exact number matters less than the order of magnitude: a medium effect needs dozens of participants per group, not a handful.

Now see what happens if the effect is small instead. With an effect size of 0.2, the square is 0.04, and 16 divided by 0.04 is 400 participants per group. A small effect needs a much larger study, which is why small effects are expensive to study well. And with a large effect of 0.8, the square is 0.64, and 16 divided by 0.64 is 25 participants per group. Large effects are cheap to detect, which is why early studies of dramatic treatments can be small and still convincing.

Common mistakes to avoid

The most common mistake is calculating power after the study, using the observed effect size, and treating the result as informative. This so-called post hoc power adds nothing beyond the p-value itself, because it is mathematically tied to it. Power analysis belongs before data collection, when it can still change the design.

Another mistake is confusing statistical significance with practical importance. A very large study can find a tiny effect statistically significant, even though the effect is too small to matter. Power analysis helps here too, because it forces the question of what effect size is worth detecting, before the data arrive.

A third mistake is ignoring dropouts and missing data. If you need 64 participants per group and expect 10 percent to drop out, you should recruit about 71 per group, because 64 divided by 0.9 is about 71.1. Planning for attrition is part of honest sample size planning, not padding.

Finally, remember that power calculations depend on assumptions: the effect size, the variability, and the statistical test. If those assumptions are wrong, the power estimate is wrong too. Treat the result as a planning guide with uncertainty, and run sensitivity checks with a range of plausible effect sizes.

Power analysis sits alongside other study-design calculations. The Average Calculator helps with the means that effect sizes are built from, and the Percentage Calculator is useful for expressing dropout rates and proportions. For a related guide on planning sample sizes, see the Research Sample Size Calculator Guide.

Power is a promise you make to yourself before a study: if the effect is real and at least this large, I have a good chance of finding it. Making that promise explicit, with a realistic effect size and a sample to match, is what separates a study designed to learn something from one designed merely to hope.