Quick answer: If you run one test, a p-value has one meaning. If you run many tests, that meaning changes because you have given yourself many chances to find a small p-value by luck. Multiple-comparisons methods try to limit how often you are fooled. The two most useful ideas are familywise error control, which protects against any false positive in a defined set of tests, and false discovery rate control, which limits the proportion of flagged results you expect to be false leads.
Part 3 of a practical statistics series for researchers. If the basics are new, start with p-values, effect sizes, and confidence intervals.
Table of Contents
Why multiplicity is a real problem
A single test with a 0.05 threshold has a 5% chance of a false positive if the null is true and all assumptions hold. That sounds small.
Now imagine 20 independent tests, all null, each at 0.05. The chance of at least one false positive is much larger than 5%.
That is not a trick. It is the basic logic of repeated chances. The more tests you run, the more opportunities there are for something to look significant even when nothing real is happening.
This matters because real research is often full of testing:
- many outcomes,
- many predictors,
- many subgroups,
- many time points,
- many pairwise comparisons,
- many variants in an experiment,
- many biomarkers, features, or items.
If you treat each p-value as if it were the only test in the world, you will overstate how strong the evidence is.
What “multiple comparisons” actually means
A multiple comparison is not any two numbers being compared. It means you have a family of tests or claims and you want to control how often errors appear in that family.
That word “family” matters. The right correction depends on what set of decisions you are making.
For example:
- If you test three pre-specified outcomes and report all of them, the family might be those three tests.
- If you test 50 exploratory biomarkers, the family might be all 50.
- If you test one primary outcome and then dig into many subgroups, the family for the subgroup claims may be much larger than the family for the primary claim.
The correction is not magic. It is a way to be honest about how many chances you gave yourself.
The two main goals
There are two common error goals.
- Control the familywise error rate. This limits the probability of at least one false positive in the family. It is stricter.
- Control the false discovery rate. This limits the expected proportion of positive findings that are false positives. It is usually less strict and often more useful when you are screening many candidates.
These are not the same goal, and they are not interchangeable.
Familywise control is appropriate when even one false positive is costly, such as a confirmatory claim you plan to build a conclusion on. False discovery rate control is often better when you expect many tests, want to flag a set of promising leads, and can afford to follow up some false positives later.
A simple way to think about FDR
Suppose you run 100 tests and 10 of them come out significant after an FDR procedure.
One interpretation is: among those 10 flagged results, the procedure expects about a certain fraction to be false discoveries, on average, under its assumptions. That fraction is the target FDR level.
This is not the same as saying “there is an X% chance this specific result is false.” FDR is about the behavior of the selection procedure across the set of flagged results, not a probability attached to one paper, one estimate, or one hypothesis.
That distinction helps avoid a common overinterpretation.
What FDR does well
FDR is useful when you have many tests and you want to keep the list of findings usable.
It tends to retain more true positives than strict familywise methods in high-dimensional settings, while still limiting the false-discovery proportion on average. That makes it popular for screening tasks: gene expression, feature selection, large sets of outcomes, and similar situations where you cannot treat every test as equally sacred.
But usefulness is not the same as automatic correctness. If you apply FDR to a mess of exploratory tests without defining the family, without reporting the total number of tests, and without distinguishing confirmatory from exploratory results, you can still mislead.
What FDR does not do
FDR does not:
- prove the flagged results are true,
- make a noisy experiment reliable,
- fix a small sample,
- replace good design,
- or give each result an individual posterior probability of being true.
It is a rule for limiting one kind of error across a set of decisions. That is valuable, but it is not a substitute for thinking about the question, the model, the measurements, and the analysis plan.
Bonferroni and why it is famous
The simplest familywise idea is the Bonferroni correction: if you run m tests and want overall significance level alpha, test each one at alpha/m.
Bonferroni is easy to explain and easy to apply, which is part of why it is famous. It is also conservative in many settings, especially when tests are correlated or when you care more about the proportion of false positives than about the chance of any false positive.
That does not make Bonferroni wrong. It makes it one tool among several, best used when strict familywise control is the goal and the cost of a false positive is high.
The right correction depends on the aim
The choice is not “correct everything” versus “ignore the problem.” It is: what are you trying to protect?
- If you want to protect a small set of confirmatory claims, familywise control may be appropriate.
- If you are screening a large set of candidates and will follow up later, FDR may be more useful.
- If some tests are primary and others are exploratory, treat them differently rather than forcing one rule on everything.
- If you have a structured set of comparisons, such as pairwise contrasts from a multi-group experiment, use a method suited to that structure rather than just sprinkling corrections everywhere.
The key is to match the correction to the decision problem.
Define the family before you test
One of the most common mistakes is deciding what counts as a family after looking at the results.
That is risky because the choice can be influenced by what looks interesting. If you only correct the tests that “looked borderline” or only report the ones that survived a correction chosen after the fact, the reported evidence is less trustworthy.
A better approach is to define the family in advance when possible:
- Which outcomes are primary?
- Which are secondary?
- Which are exploratory?
- Which comparisons belong to the same family?
- Which results will be used to support a main claim?
Pre-specifying the family does not guarantee correctness, but it reduces the chance that the correction becomes a post-hoc story.
Exploratory work is still useful, but say so
Exploratory analysis is not a sin. It is how you generate hypotheses.
The problem is calling an exploratory result confirmatory. If you tested many things, picked the best-looking one, and then reported it as if it were the only test, you have overstated the evidence.
A clean way to handle this is to label results clearly:
- “Primary analysis” for pre-specified claims.
- “Secondary analysis” for pre-specified but less central questions.
- “Exploratory analysis” for broader screening or post-hoc digging.
That labeling helps readers understand what kind of evidence they are getting.
A practical decision flow
When you have many tests, ask:
- What is the claim I want to support?
- How many tests belong to that claim’s family?
- What is the cost of a false positive?
- Am I screening for leads or making a confirmatory statement?
- Did I define the family before seeing the results?
- Will I report all the tests, not just the winners?
If the answer to the last question is no, the correction alone will not rescue the interpretation.
A worked sketch
Suppose you compare a new teaching method across 12 outcomes.
- One outcome is primary.
- Five are secondary.
- Six are exploratory.
If the primary outcome is significant, you can report it with its p-value and interval, understanding that it was a single planned test.
For the secondary outcomes, you may want to correct for the five tests if you want to guard against false positives among them.
For the exploratory outcomes, you might use FDR if you want a shortlist of leads, but you should be clear that those are leads, not confirmed effects.
That structure is more honest than collapsing all 12 outcomes into one “significance” statement.
What to report
When multiplicity is in play, the report should usually say:
- how many tests were run,
- what family each correction applied to,
- which method was used,
- which results were primary, secondary, or exploratory,
- and which findings would survive a stricter or looser rule.
That level of detail lets readers judge the evidence rather than just seeing a table of asterisks.
Common mistakes
- Treating every p-value as if it were the only one tested.
- Correcting some tests but not others without a clear reason.
- Choosing the correction method after seeing which results look best.
- Confusing FDR with the probability that a given finding is false.
- Using a correction to make a weak study look rigorous.
- Failing to distinguish confirmatory and exploratory claims.
- Reporting only the tests that survived correction.
Key takeaways
- Many tests create many chances for false positives.
- Multiplicity is about the family of claims, not just the number of numbers.
- Familywise control protects against any false positive in the family; FDR controls the expected proportion of false discoveries among the flagged results.
- The right method depends on the goal, not on a universal rule.
- FDR is useful for screening; it is not a claim that each finding is likely true.
- Define the family before testing when you can.
- Label primary, secondary, and exploratory results clearly.
- Report the total number of tests and the correction used.
Challenge
Take a set of results you have seen or generated with many outcomes or comparisons. Identify the primary claim, the family of tests that should share a correction, and whether familywise control or FDR is more appropriate for that claim. Then write two sentences: one for the confirmatory interpretation and one for the exploratory shortlist.
Analysing many outcomes or predictors and unsure how to report them? Ampersand Academy offers one-to-one mentoring on statistics and data analysis — part of Mahadhi, a development and digital marketing company.
—
FAQ
Not always, but you do need to account for multiplicity whenever you make several related claims and want to limit false positives. Ignoring it can make a set of results look stronger than they are.
- Do I always need to correct for multiple comparisons?
Familywise error controls the chance of at least one false positive in a family of tests. FDR controls the expected proportion of positive findings that are false discoveries across the flagged set.
- What is the difference between familywise error and FDR?
Not universally. Bonferroni is stricter and may be suitable for confirmatory claims. FDR is often more useful for large screening sets. The right choice depends on the goal.
- Is FDR better than Bonferroni?
No. FDR is about the behavior of the selection procedure over the set of flagged results, not a direct probability that any one result is true.
- Does FDR mean a significant result is probably true?
A family is the set of tests tied to the same decision problem. It might be a few pre-specified outcomes, many exploratory biomarkers, or all pairwise comparisons from one experiment. The definition matters.
- What counts as one family of tests?
You can call it a lead or an exploratory result. Calling it a confirmed finding is misleading if many other tests were also examined.
- Can I call an exploratory result a finding?
Report how many tests were done, which family each correction applied to, which method you used, and which results were primary, secondary, or exploratory.
- What should I report when I run many tests?
Do I always need to correct for multiple comparisons?
Not always, but you do need to account for multiplicity whenever you make several related claims and want to limit false positives. Ignoring it can make a set of results look stronger than they are.
What is the difference between familywise error and FDR?
Familywise error controls the chance of at least one false positive in a family of tests. FDR controls the expected proportion of positive findings that are false discoveries across the flagged set.
Is FDR better than Bonferroni?
Not universally. Bonferroni is stricter and may be suitable for confirmatory claims. FDR is often more useful for large screening sets. The right choice depends on the goal.
Does FDR mean a significant result is probably true?
No. FDR is about the behavior of the selection procedure over the set of flagged results, not a direct probability that any one result is true.
What counts as one family of tests?
A family is the set of tests tied to the same decision problem. It might be a few pre-specified outcomes, many exploratory biomarkers, or all pairwise comparisons from one experiment. The definition matters.
Can I call an exploratory result a finding?
You can call it a lead or an exploratory result. Calling it a confirmed finding is misleading if many other tests were also examined.
What should I report when I run many tests?
Report how many tests were done, which family each correction applied to, which method you used, and which results were primary, secondary, or exploratory.