ANOVA and Post-Hoc Tests: Comparing Three or More Groups Properly

You have four treatment groups and you want to know which differ. Running all six pairwise t-tests feels natural and is one of the most common errors in applied statistics: with six tests at α = 0.05, the chance of at least one false positive is roughly 26% if nothing is really going on.

ANOVA is the standard answer, but it answers a narrower question than most people expect, and the post-hoc step that follows is where the real decisions live.

When you need ANOVA rather than repeated t-tests

The core problem is the family-wise error rate: the probability of at least one false positive across a set of tests. Testing each comparison independently controls only the per-test rate, and those compound.

Number of groups	Pairwise tests	Family-wise error at α = 0.05
2	1	5%
3	3	14%
4	6	26%
6	15	54%

ANOVA avoids this by testing one omnibus hypothesis — all group means are equal — with a single test. Only if that test rejects do you proceed to look at which groups differ, and you do so with a method that accounts for the multiple comparisons.

What the F test actually tests

The F statistic is a ratio of two variances: the variance between group means, and the variance within groups. If group means differ more than the internal variability of the groups would explain, F is large.

F = (between-group variance) / (within-group variance)

Two consequences follow immediately. First, a significant F tells you that not all means are equal — it does not tell you which differ, or how many. Second, F grows when within-group variance shrinks, so a real but small difference becomes detectable with a large enough sample. Statistical significance here is still not effect size.

# R
fit <- aov(score ~ treatment, data = experiment)
summary(fit)

# Python
import statsmodels.api as sm
from statsmodels.formula.api import ols
model = ols("score ~ C(treatment)", data=df).fit()
sm.stats.anova_lm(model, typ=2)

Following up: which post-hoc test

There is no single right post-hoc test; the choice follows from what you want to compare and what your variances look like.

Method	Compares	Controls	Use when
Tukey HSD	All pairs	Family-wise error	Equal variances, all pairwise comparisons matter
Dunnett	Each group vs one control	Family-wise error	There is a natural control arm
Bonferroni / Holm	Any chosen set	Family-wise error	A small, specific set of planned comparisons
Games-Howell	All pairs	Family-wise error	Variances clearly unequal, unequal n
Benjamini-Hochberg	Any set	False discovery rate	Many comparisons, exploratory screening

Dunnett is worth singling out. If your design has a vehicle or placebo arm and three doses, you have three comparisons of interest, not six. Using Tukey there tests pairs nobody cares about and gives up power for no reason.

library(emmeans)

# All pairwise, Tukey-adjusted
pairs(emmeans(fit, ~ treatment))

# Control comparisons only
emmeans(fit, ~ treatment) |>
  contrast(method = "trt.vs.ctrl", ref = 1, adjust = "dunnett")

Note that emmeans with trt.vs.ctrl is usually preferable to TukeyHSD() for control designs, because it handles unbalanced groups correctly and reports confidence intervals you can interpret directly.

Planned versus unplanned comparisons

If you specified comparisons before collecting data — dose 1 versus dose 2, treated versus control — they are planned contrasts and deserve the more powerful approach. Searching all pairs after seeing the means is unplanned, and needs the stricter correction. Deciding which one you did after the fact is how post-hoc rationalisation enters the literature.

Two-way ANOVA and interaction

With two categorical factors you get three tests: the main effect of each factor and their interaction. The interaction is usually the most interesting question, because it asks whether the effect of one factor depends on the level of the other.

fit2 <- aov(score ~ treatment * sex, data = experiment)
summary(fit2)

There is a real trap here. When an interaction is present, main effects are not interpretable in isolation: an “average effect of treatment” may describe no group in the study, because treatment helps one sex and harms the other. In that case report the interaction and the simple effects within each level, not the main effects alone.

# Simple effects inside the interaction
emmeans(fit2, ~ treatment | sex)

Type I, II and III sums of squares

For unbalanced designs the three types give different answers, and software defaults differ — R’s aov uses Type I, which depends on term order. For unbalanced factorial data with an interaction, Type III with sum-to-zero contrasts is the conventional choice; Type II is preferred when you have decided to ignore an interaction.

# Type III with proper contrasts, the reproducible route
options(contrasts = c("contr.sum", "contr.poly"))
library(car)
Anova(fit2, type = 3)

Checking assumptions

ANOVA assumes independent observations, approximately normal residuals within groups, and equal variances across groups. The first is a design property, not a test — no diagnostic rescues clustered or repeated observations, and those need a mixed model.

plot(fit)                      # residuals vs fitted, Q-Q
library(car)
leveneTest(score ~ treatment, data = experiment)

ANOVA is robust to mild non-normality with reasonable group sizes, largely because of the central limit theorem acting on the means. It is much less robust to unequal variances when group sizes differ, which is the case where Games-Howell or a Welch correction is warranted.

Situation	Better choice
Variances unequal, n unequal	Welch ANOVA, then Games-Howell
Ordinal outcome or severe non-normality	Kruskal-Wallis, then Dunn with correction
Repeated measures on the same subjects	Mixed model, not repeated t-tests
One continuous covariate plus groups	ANCOVA, checking slopes are parallel

Effect sizes

A p-value from an ANOVA tells you something differs; eta-squared tells you how much of the variance the factor accounts for. Report it, and report the group means with confidence intervals so a reader can see the pattern rather than a significance asterisk.

library(effectsize)
eta_squared(fit2, partial = TRUE)

Partial eta-squared is standard in factorial designs but is not comparable across studies with different designs, which is a reason to also report raw means and differences with intervals.

What to report

State the design, group sizes, means and standard deviations, the F statistic with degrees of freedom and p-value, the effect size, which post-hoc method you used and why, and the adjusted confidence intervals for the comparisons. If the interaction was significant, say so before any main effect.

Design choices — balanced groups, a prespecified control, adequate within-group precision — determine how much any of this can tell you. If you want to work through that planning side, Ampersand Academy covers it alongside the analysis.

Practice

1. Simulate four groups drawn from the same distribution, run all pairwise t-tests, and count false positives over 1,000 repetitions.
2. Run the same simulation with one ANOVA and Tukey follow-up and compare.
3. Re-analyse a two-factor dataset and state whether the interaction changes how you read the main effects.
4. Compare Tukey and Dunnett on a control-versus-doses design and explain the power difference.

Related tutorials

• How to Choose the Right Statistical Test: A Decision Chain That Follows Your Design — where an ANOVA sits in the wider decision chain
• Multiple Comparisons and False Discovery Rate: What Changes When You Test a Lot — why a significant F test still needs corrected pairwise comparisons
• Effect Size in Statistics: Cohen d, r and Eta-Squared — reporting eta-squared or partial eta-squared with the result