How to Choose a Sample Size: Power Analysis That Does Not Lie to You (2026)

Quick answer: Sample size is not something you guess after you have already collected most of the data. It is a planning decision based on the smallest effect you would care about, how much risk of a false negative you will tolerate, how much noise and variability you expect, and the design you will actually use. A power analysis that ignores any of those pieces can give you a precise-looking number that is still wrong in practice.

Part 2 of a practical statistics series for researchers. Start with p-values, effect sizes, and confidence intervals if those terms are new.

What you are actually deciding

When researchers ask, “How many participants do I need?” they are usually asking a cluster of related questions:

• How large an effect do I want to be able to detect?
• How sure do I want to be that I will detect it if it is really there?
• How much noise or variability should I expect?
• What analysis will I run?
• What design will I use?
• What can I realistically afford?

Those are not the same question. A sample size calculation is only as good as the assumptions behind it.

The smallest effect that matters

The most important and most neglected input is the effect size you would treat as meaningful.

There are two common traps.

• Using the smallest effect you can imagine. If you power for a trivially small effect, you may end up requesting more data than is feasible or necessary.
• Using the largest effect you hope to see. If you power for an optimistic estimate from a small pilot or a prior paper, your study may be underpowered for the effect sizes that are realistic.

A better target is the smallest effect you would interpret as worth acting on, reporting, or following up. That is a scientific judgment, not just a statistical one.

For example, in a clinical trial, a 0.2-point drop on a 10-point scale may be statistically detectable with enough subjects but clinically uninteresting. In another setting, a 1% change in a high-volume process could matter a great deal. Context decides.

Power is a planning concept, not a promise

Power is the probability of obtaining a statistically significant result, under a specific assumed effect and model, if that model is true.

That definition contains the usual cautionary notes:

• Power depends on assumptions.
• Power refers to a particular analysis and design.
• Power is not the probability that your study will “work” in every sense.
• Power does not guarantee the estimate will be precise enough to matter.

Many researchers quote “80% power” as if it were a complete justification. It is not. It is one input. You still need to say what effect you powered for, how uncertain that effect estimate is, and whether the study will be informative even if it does not reach significance.

The four ingredients most calculations need

For many standard designs, a sample size calculation needs at least these ingredients:

• the target effect size,
• the significance level, often 0.05,
• the desired power, often 80% or 90%,
• the expected variability or noise,
• and the design structure: one-sample, two-sample, paired, multi-group, repeated measures, regression, and so on.

Change any one of those and the required sample size changes. That is not a flaw. It is the point.

Variability matters more than beginners expect

Sample size is heavily influenced by variability.

If the outcome is noisy, you need more data to distinguish signal from noise. If the measurement is imprecise, you may need more data even when the effect is large. If the population is heterogeneous, the effective sample size can feel smaller than the headcount suggests.

This is one reason pilot data can be helpful and also dangerous. A pilot can give a realistic sense of variability, but small pilots are often unstable and can underestimate or overestimate the true spread. Use them as a guide, not as gospel.

A useful habit is to ask: if the real variability is worse than I think, what happens to my power? If the answer is “it drops badly,” the design may be fragile.

The design changes the math

The same research question can require very different sample sizes depending on the design.

• A paired design can be much more efficient than an independent-groups design when the pairing removes meaningful variability.
• Repeated measures can increase information if the correlations and timing are handled correctly.
• Cluster designs often need more total subjects because observations within a cluster are not independent.
• Covariates can help if they explain real variation, but they do not rescue a bad design.

Do not assume that a bigger number of measurements is the same as a bigger number of independent units. In many repeated-measures settings, the number of subjects still drives the effective information.

A better way to think about precision

Power is about the chance of reaching a decision threshold. Precision is about how tightly the estimate is pinned down.

Both matter.

A study can be “powered” to detect an effect and still produce an estimate with a wide confidence interval if the effect is smaller than expected or the noise is larger than expected. In that case, the study may end up inconclusive even if it was technically well planned.

If your real goal is to learn a useful quantity, plan for the precision you want as well. For example, you may want the final confidence interval to be narrow enough that it excludes effects you would consider uninteresting. That is often a better planning target than a bare significance test.

A practical way to justify a sample size

A defensible sample size justification usually says:

• what effect you would treat as meaningful,
• what power you aimed for,
• what variability or noise you assumed,
• what analysis you planned,
• what assumptions you were uncertain about,
• and what you would conclude under different outcomes.

That last point is important. A good sample size plan includes a plan for the result you did not expect.

For instance:

• If the result is significant and the estimate is close to the target effect, what will you conclude?
• If the result is significant but the estimate is much smaller, how will you interpret it?
• If the result is not significant but the estimate is in the expected direction, will you treat it as inconclusive, disappointing, or suggestive?
• If the result is not significant and the estimate is near zero, what will you conclude?

Planning for those cases keeps the study from becoming a rhetorical exercise after the fact.

Why a power analysis can mislead

Power analysis is helpful, but it can go wrong in predictable ways.

• Wrong effect target. If the target effect is unrealistic, the sample size will be too.
• Overly optimistic variance. If you assume too little noise, the study may be underpowered.
• Wrong analysis assumptions. If the planned model does not match the data, the nominal power may not reflect reality.
• Ignoring multiplicity. If you will test many outcomes, the sample size for one comparison may not protect the familywise error rate you care about.
• Treating power as precision. A study can be powered to reject a null and still leave you with a wide interval.
• Ignoring attrition or missing data. If participants drop out or measurements fail, the effective sample can shrink.

A careful plan names these risks instead of burying them.

Sample size and “significance fishing”

One subtle danger is that a study designed only to pass p < 0.05 can encourage threshold thinking.

If the main goal becomes “get significance” rather than “estimate the effect well enough to answer the question,” the design can lead to brittle conclusions. That is especially true when the sample is barely enough, the analysis is flexible, and the researcher keeps trying variations until something crosses the line.

A better design aims to produce an estimate that is useful whether or not it is significant.

A simple planning habit

Before you compute a number, write down:

• the question,
• the smallest effect you care about,
• the analysis you will use,
• the variability you expect,
• the power you want,
• and the precision you would find useful.

Then compute the sample size for the primary target. Then check how sensitive that number is to changes in variability, effect size, and attrition. Then decide whether the result is feasible.

If the required sample size is unrealistic, do not just shrug and proceed. Either change the design, reduce measurement error, narrow the question, or accept that the study will be inconclusive for the effect you care about.

A worked sketch

Imagine a two-group comparison of a new training method on test scores.

You decide:

• a 3-point difference on a 20-point test would be worth detecting,
• the standard deviation is around 6 points based on earlier work,
• you want 80% power at a 5% significance level,
• and you expect a simple independent-groups analysis.

That setup gives you a concrete target. You can then ask:

• What if the standard deviation is 8 instead of 6?
• What if the real difference is 2 points?
• What if 10% of participants drop out?
• What if the test scores are skewed and you need a different analysis?

Each question changes the picture. The final sample size should reflect the most plausible and the most damaging plausible scenarios, not just the optimistic one.

Common mistakes

• Powering for the effect you hope to see rather than the smallest effect you would care about.
• Treating a single power number as a full justification.
• Ignoring measurement error and dropout.
• Using a design that is inefficient for the question.
• Planning for significance but not for precision.
• Assuming that “adequately powered” means “definitely informative.”
• Forgetting that the analysis model and the sample size must match.

Key takeaways

• Sample size should be driven by the smallest effect worth detecting, not by hope.
• Power is the chance of a significant result under a specific assumption, not a guarantee.
• Variability, design, and analysis method all matter.
• Precision is often a better planning target than significance alone.
• A good justification states the effect, the assumptions, and the decision plan.
• A fragile design can produce a statistically significant result that still does not answer the question.

Challenge

Take a study idea you are working on and write a short sample-size justification that a skeptical colleague could challenge. Include the smallest meaningful effect, the planned analysis, the assumed variability, the desired power, and at least two assumptions that could change the result if they are wrong. Then note what you would conclude if the study ends up non-significant but with a wide interval.

Planning and analyzing studies with real data? Ampersand Academy offers one-to-one mentoring on statistics and data analysis — part of Mahadhi, a development and digital marketing company.

—

FAQ

Power is the chance that a study will produce a statistically significant result for a given effect size and analysis, assuming that effect and model are true. It is a planning tool, not a guarantee.

• What is power in simple terms?

Because if the true effect is smaller, the study may be underpowered. Powering for an optimistic effect can leave you unable to detect realistic differences.

• Why not just power for the biggest effect I can imagine?

It is the smallest effect you would treat as scientifically, clinically, or practically meaningful in the context of the question. Choosing it is partly a statistical decision and partly a domain judgment.

• What is the smallest effect worth detecting?

Not necessarily. It is a common convention, but the right level depends on the stakes, the cost of a false negative, and the precision you need. Some studies need more; some cannot justify that much data.

• Is 80% power always enough?

Yes. More noise means more data to detect the same effect. Variability is often the biggest driver of required sample size.

• Do I really need to consider variability?

Use the best available evidence, but check how sensitive the sample size is to worse-case variability. Pilot data can help if you treat it cautiously.

• What if I do not know the variability in advance?

Larger samples usually improve precision and power, but they do not fix a bad design, a weak measurement, or a question that is too vague. A well-designed smaller study can be more informative than a poorly designed larger one.

• Does a larger sample always make a study better?