Quick answer: You choose a statistical test by walking a short decision chain, not by matching a table row. The chain is: what is your actual question (compare, associate, predict), what kind of outcome did you measure (continuous, count, category, ordinal, time-to-event), how are the observations structured (independent groups, paired, repeated, clustered), and how many groups are you comparing. Design comes first, data type second, and only then does the familiar list of named tests fall out.
Part 8 of a practical statistics series for researchers. If the basics are new, start with p-values, effect sizes, and confidence intervals.
Table of Contents
Start with the question, not the test
Most statistical-testing confusion starts at the wrong point: someone finds a menu of test names and tries to pick one. The healthier order of operations is the reverse.
Every inferential situation has four ingredients, and each narrows the choices:
- The research question. Do you want to compare groups, quantify an association, model dependence on predictors, or estimate a proportion?
- The outcome type. Continuous and roughly smooth? A count? A category with names or an order? A time until an event?
- The design. Independent observations? Paired measurements? The same subject measured several times? Sites that produce clusters of subjects?
- The conditions. Roughly symmetric distributions with similar spreads, or data that clearly violate those ideas?
Two different researchers can have data that look superficially identical and need completely different tests, because their designs differ. Ten patients measured before and after treatment are not twenty independent data points. Thirty villages with five farmers each are not one hundred and fifty independent observations.
The decision chain, step by step
Here is the chain as a sequence of questions. Answer them in order; stop when you reach a named procedure.
Step 1: What is the estimand? Ask what quantity you want to compare or estimate: a difference between group means, a difference in medians or ranks, an association between two variables, a proportion, a rate, or a hazard. The statistical literature is explicit on this point: the scientific question and the scale of the outcome determine the test, not the other way around (du Prel and colleagues make this the backbone of their overview of test choice).
Step 2: What is the outcome type?
- Continuous outcomes (heights, concentrations, response times) open up the family of mean-comparison and regression methods.
- Counts and rates (detections per plot, visits per month) point toward Poisson-type models rather than t-tests on raw counts.
- Binary and categorical outcomes (recovered or not, the two options chosen in a forced-choice task) point toward proportions, chi-square tests, and logistic regression.
- Ordinal outcomes (severity scores, Likert items) are comparable only through their order, which is the natural habitat of rank-based methods.
- Time-to-event outcomes need survival machinery, not ordinary group comparisons.
Step 3: How are the observations structured? Independent groups, paired or matched pairs, a single factor with several levels, more than one factor, repeated measures, or clusters. The same continuous two-group question has two different answers depending purely on this step: independent groups call for a two-sample comparison, paired designs call for a paired analysis of the within-pair differences.
Step 4: How many groups? With two groups, a mean comparison is direct. With three or more groups, you are in omnibus-test territory, and a significant overall result only says that at least one group differs — it does not say which. The follow-up comparisons then drive the multiplicity questions covered in the multiple comparisons article.
Step 5: Are the model conditions plausible? Means-based tests do carry conditions — even and well-behaved variability, and an outcome distribution in which stray values are not doing all the work. Do not pretend this step does not exist; also do not let a mechanical check dominate it. The assumption-checking article explains what to plot and what to trust at this stage.
The two-group continuous case, carefully
This is the case where the decision chain pays off most visibly, because it is the one beginners skip fastest.
Given a continuous outcome and two groups, ask first whether the groups are independent or paired. Ask that before choosing, because the two situations use different procedures, and mixing them up invalidates the analysis in either direction.
Within independent groups, ask about the spread. If the two variances could plausibly differ, the Welch variant of the two-group comparison is the safer default, precisely because it does not bet on equality of variances that you have not earned.
Counts, categories, and the table tests
When the outcome is categorical, the unit of analysis is the cell count.
A chi-square test checks whether category membership is related to group membership by comparing observed counts to the counts expected under independence. Its approximation works well when expected counts are comfortable; in very small samples with two categories by two groups, the exact version of the test is the standard route.
Watch out for one common inversion of this principle: a chi-square test on raw counts that are actually per-subject repeated measurements, rather than independent subjects, is a design error, not a flavor choice. The denominator needed for the inferential statements comes from how many independent units contributed the counts.
More than two groups: the omnibus trap
Moving from two groups to three or more is not just a parameter change. It changes the shape of the claim.
An omnibus test across groups says one thing: that the pattern you observed is unlikely under the hypothesis that all the groups behave alike. It does not say which pairs differ. Reviewers who read a single omnibus p-value as a license to name any pair as the winner are reading more out of the test than it delivered.
The two honest responses are:
- use a follow-up procedure designed for the planned comparisons, with its multiplicity logic spelled out in advance;
- or report the omnibus result and treat pairwise exploration as exploratory, labeled as such, as the multiple-comparisons article recommends for family definition.
Where regression replaces the named tests
The named two-group and multi-group tests are special cases of a broader modeling idea. Once you want to adjust for a second variable, or you have a non-continuous outcome, plain group comparisons stop being enough, and you are in regression territory.
- Continuous outcome with meaningful predictors — linear modeling.
- Binary outcome with predictors — logistic modeling.
- Count outcome with predictors — Poisson-family modeling.
The regression model choice guide walks that space in detail. The point to keep in visible focus here: knowing the test name does not mean knowing the question. A two-sample comparison and a linear model with a two-level factor estimate the same substantive contrast, and the modeling view is often the more informative framing when the design has any structure at all.
What to do when no test seems to fit
Sometimes the honest answer is that the available data cannot support the claim being attempted.
- If the sample size is far too small for any comfortable approximation, say so, and consider what a wider interval or a more transparent report of raw values would add.
- If the design confounded two factors, no defensible test un-confounds it. Design problems are not solved downstream.
- If the outcome is heavily censored or truncated, specialized methods apply — not ordinary tests run anyway.
This is the quiet lesson of the decision chain: some cells of the menu are unreachable without changing the study.
A worked example of the chain
Suppose you want to know whether a new dryland crop variety differs from the farm-average yield, and you have thirty plots, fifteen of each variety:
- Question: compare a continuous outcome between two varieties → two-group mean comparison.
- Design: plots independent of one another, no pairing → independent-samples framing.
- Spreads: roughly comparable, no extreme outliers → the ordinary two-sample comparison is fine; when in doubt, the Welch variant is the safe default.
- Follow-up: if the comparison involves more than two varieties, first the omnibus test, then a planned contrast or corrected pairwise procedure.
Change any one ingredient and the chain changes route. Fifteen farmers who each measure one plot of both varieties? That is a paired design, and the analysis becomes the mean of within-farmer differences, tested by the paired procedure. Yields recorded as intervals rather than exact values? A different family appears.
Common mistakes
- Choosing the test from a cheat-sheet that starts with the test names instead of with the design.
- Treating matched or repeated observations as independent.
- Picking a non-parametric test purely because a normality check gave a small p-value, without asking whether the estimand matches (see the rank tests article on what those tests compare).
- Running pairwise tests after a significant omnibus result without handling multiplicity.
- Running a chi-square test over counts that came from clustered or repeated designs.
- Reading a two-group result to one question and then answering a different one.
Key takeaways
- The chain is: estimand first, outcome type second, design third, group count fourth, conditions fifth.
- Independent versus paired is the single most consequential design call in the standard comparisons.
- Counts, categories, and continuous measurements call for different inferential machinery, and that is not negotiable.
- Omnibus results do not name the winning pair on their own.
- A named two-group test is a special case of a linear model — think in that direction when the design has any structure.
- No test can rescue a design that could not express the question.
Challenge
Take one analysis you have recently planned or reviewed. Write out the five-decision chain explicitly — estimand, outcome type, structure, group count, conditions — and name the test that follows from those answers. Then deliberately change one ingredient (paired instead of independent, three groups instead of two, count instead of continuous) and verify how the route changes. If any step was decided only after seeing the data, flag that as data-dependent selection.
Choosing analyses on real datasets? Ampersand Academy offers one-to-one mentoring on statistics and data analysis — part of Mahadhi, a development and digital marketing company.
—
FAQ
Fastest is not the right goal. The shortcut that stays correct is to start from the design and the measurement scale of the outcome, then check feasibility conditions on the candidate that the chain selects. A reputable decision guide, and the chain in the article above, both apply.
- What is the fastest way to choose a statistical test?
If the claim is about the set of groups as a whole, yes. If the claim was always one specific planned comparison, that comparison plus its correction is the more direct answer.
- If I have three groups, do I automatically need an omnibus test?
It depends on how strong the approximation is for a given sample and how noisy the estimate of variability is. The chain does not change, but the applicability conditions of whatever the chain selects do.
- Does the choice of test depend on sample size?
No. A pretest step of that kind delegates the decision to a nudge from noise. Model conditions should be judged on domain understanding and diagnostics, as the assumptions article explains.
- Should I base the choice on the result of a normality test?
As soon as the design has more than one driver of variation worth accounting for, or when the outcome is not continuous, the regression framing is usually the clearer route.
- When do I stop choosing tests and start using a regression model?
Yes, but each claim then stands separately, and the multiplicity implications of running both should be acknowledged rather than hidden.
- Can the same dataset support two different tests for two different questions?
Interactive choosers exist, and they are helpful teaching aids for two-group and several-group basics, but they cannot capture your study design and your question. Treat them as a cross-check for the answer the chain already produced, never as the starting point.
- Is there one authoritative tool that automates this?
What is the fastest way to choose a statistical test?
Fastest is not the right goal. The shortcut that stays correct is to start from the design and the measurement scale of the outcome, then check feasibility conditions on the candidate that the chain selects. A reputable decision guide, and the chain in the article above, both apply.
If I have three groups, do I automatically need an omnibus test?
If the claim is about the set of groups as a whole, yes. If the claim was always one specific planned comparison, that comparison plus its correction is the more direct answer.
Does the choice of test depend on sample size?
It depends on how strong the approximation is for a given sample and how noisy the estimate of variability is. The chain does not change, but the applicability conditions of whatever the chain selects do.
Should I base the choice on the result of a normality test?
No. A pretest step of that kind delegates the decision to a nudge from noise. Model conditions should be judged on domain understanding and diagnostics, as the assumptions article explains.
When do I stop choosing tests and start using a regression model?
As soon as the design has more than one driver of variation worth accounting for, or when the outcome is not continuous, the regression framing is usually the clearer route.
Can the same dataset support two different tests for two different questions?
Yes, but each claim then stands separately, and the multiplicity implications of running both should be acknowledged rather than hidden.
Is there one authoritative tool that automates this?
Interactive choosers exist, and they are helpful teaching aids for two-group and several-group basics, but they cannot capture your study design and your question. Treat them as a cross-check for the answer the chain already produced, never as the starting point.