Quick answer: A p-value is not the probability that your hypothesis is true, and it is not a measure of how important a result is. It is the probability of data at least this unusual assuming a specific null model. The effect size tells you how large the difference or association is. The confidence interval tells you how precisely you estimated it. If you report only “p < 0.05,” you have told the reader almost nothing about the result that matters scientifically.
First in a practical statistics series for researchers. The next article covers how to choose a sample size around the smallest effect that would matter.
Table of Contents
The mistake most papers still make
A result is often written up as if the p-value carried the whole story:
- “The groups differed significantly (p = 0.03).”
- “There was no significant difference (p = 0.08).”
Both sentences are statistically thin. The first sounds like a discovery even when the effect is tiny. The second sounds like evidence of no effect even when the estimate is noisy. Neither tells a reader what changed, how much, or how uncertain the estimate is.
A better sentence usually looks more like this:
- “The intervention lowered systolic blood pressure by 4.2 mmHg (95% CI 1.1 to 7.3), p = 0.008.”
Now the reader knows the direction, the size, the uncertainty, and the p-value as supporting information rather than the headline.
What a p-value actually is
A p-value answers one narrow question: if a particular null model were true, how surprising is data this extreme or more extreme?
That definition has three parts people often miss.
- It is conditional on a model. The p-value is not about your theory directly. It is about a null model, usually “no difference” or “no association,” plus assumptions about the data generating process.
- It is about the data, not the hypothesis. A small p-value does not mean the alternative is probably true. It means the observed data would be unusual under the null.
- It is not a measure of size or importance. A huge sample can produce a tiny p-value for a trivial effect. A small sample can produce a non-significant p-value even when the estimated effect is large.
This is why “p = 0.03” without an estimate and uncertainty is such an incomplete result.
What the effect size is for
The effect size answers a different question: how much?
That “how much” can take many forms:
- a mean difference,
- a standardized mean difference such as Cohen’s d,
- a risk difference,
- a ratio such as an odds ratio or hazard ratio,
- a correlation or regression coefficient.
The right choice depends on the question and the outcome. For a clinical outcome, the raw mean difference may be the most interpretable. For a count outcome with different baselines, a ratio may make more sense. For a psychological scale, a standardized difference can help compare across studies, but it should not replace the original units when those are meaningful.
The point is not to worship one effect size metric. The point is to report the quantity that answers the research question, not just whether the result is “significant.”
What the confidence interval actually tells you
A confidence interval gives a range of plausible values for the estimate given the model and data. It reflects precision.
A narrow interval means the estimate is fairly precise. A wide interval means the data do not pin it down well. That can happen for many reasons: small sample, high variability, weak design, or a rare outcome.
A common misinterpretation is to say, “There is a 95% probability the true value is in this interval.” That is not what a frequentist confidence interval means in the standard sense. The interval either contains the unknown parameter or it does not. The confidence level describes the procedure over repeated sampling, not a probability attached to one realized interval.
That distinction matters in practice because it keeps you from treating the interval as a literal probability statement. Even so, the interval is still one of the most useful reporting tools available: it shows both the estimate and the uncertainty around it.
A simple way to read a result
When you see a result, ask three questions:
- What is the estimate?
- How uncertain is it?
- Is the uncertainty small enough to matter for the question at hand?
A p-value answers none of those directly. It can be a useful diagnostic, but it should not be the main thing you report.
Consider two imagined results:
- Estimate = 0.10, 95% CI -0.02 to 0.22, p = 0.10
- Estimate = 0.45, 95% CI 0.38 to 0.52, p < 0.001
The second is both larger and more precise. The first is weak not just because p > 0.05, but because the interval includes values near zero and values that could matter. Those are different scientific situations, even if both get labeled “non-significant.”
Why “significant” and “non-significant” are bad labels
The 0.05 cutoff is a convention, not a law of nature. Treating it as a bright line creates several problems:
- Results just under 0.05 get celebrated as if they are more certain than they are.
- Results just over 0.05 get treated as failures even when the estimate is large and imprecise.
- Readers confuse statistical significance with practical significance.
- Researchers are tempted to chase the threshold rather than answer the question.
A better habit is to report the estimate, the interval, and the p-value, then interpret them together. If you need a decision rule, make it explicit and tie it to the context, not just to the p-value.
A worked interpretation
Suppose a study compares task completion time between two interfaces.
- Mean difference = 2.3 seconds, 95% CI 0.4 to 4.2, p = 0.02
What can you say?
- The estimated difference is 2.3 seconds.
- The data are compatible with differences from about 0.4 seconds to about 4.2 seconds, under the model.
- The p-value is small, meaning such data would be fairly unusual if the null of no difference were true.
- Whether 2.3 seconds matters depends on the context: user experience, task frequency, safety, cost, or the research question.
Now suppose the same study had:
- Mean difference = 2.3 seconds, 95% CI -1.1 to 5.7, p = 0.18
The point estimate is identical, but the uncertainty is much larger. The result is less conclusive, and the correct response is not “there is no effect”; it is “the data are too weak to decide confidently.”
The reporting rule that prevents most confusion
Use a three-part result statement whenever you can:
- the estimate,
- the uncertainty interval,
- the p-value or other supporting evidence,
and then add the scientific interpretation.
For example:
- “The odds of remission were higher with treatment A than treatment B (OR 1.62, 95% CI 1.18 to 2.22, p = 0.003). The size of the association is modest but consistent across the range of plausible values.”
That sentence tells the reader what the result is, how precise it is, and why it might matter.
Confidence intervals are not a substitute for thinking
A confidence interval does not tell you whether the effect is clinically, economically, or scientifically meaningful. That judgment needs context.
A very narrow interval around a tiny effect can be “statistically significant” and still unimportant. A wide interval around a large estimate can be inconclusive but still worth following up. Interpretation requires both numbers and substance.
Common misunderstandings to avoid
- “P = 0.05 means there is a 5% chance the result is due to chance.” Not accurate. The p-value is computed under a null model; it is not the probability the result is “due to chance” in any simple sense.
- “Non-significant means no effect.” Not accurate. It often means the data are too weak to distinguish signal from noise.
- “Significant means important.” Not accurate. Importance depends on the size and context.
- “The confidence interval is the range where the truth probably lies.” Loosely appealing, but not the formal meaning, and it can mislead if you treat it as a literal probability statement.
- “If two intervals overlap, the comparison is non-significant.” Not always true. Overlap and significance are related but not identical, especially for pairwise comparisons of separate estimates.
A quick checklist for results
Before you report a result, confirm you can answer these:
- What is the estimate?
- What is the uncertainty interval?
- What is the p-value, if you report one?
- What is the smallest effect that would matter in this setting?
- Does the interval exclude that meaningful effect, include it, or remain too wide to decide?
If you cannot answer those, the result is probably underreported.
Key takeaways
- A p-value is about data under a null model, not the probability that a hypothesis is true.
- An effect size tells you how large the result is.
- A confidence interval tells you how precisely the effect was estimated.
- Statistical significance is not the same as practical importance.
- “Non-significant” is not proof of no effect.
- Report estimate, interval, and p-value together, then interpret.
Challenge
Take any result you have read recently that was described only as “significant” or “non-significant.” Rewrite it with an estimate, a confidence interval, and a short interpretation of whether the plausible range matters for the research question. If the original paper did not report the interval, note that omission as a limitation in your own summary.
Want a second pair of eyes on how you are reporting results? Ampersand Academy offers one-to-one mentoring on statistics and data analysis — part of Mahadhi, a development and digital marketing company.
—
FAQ
No. It can be useful as one piece of evidence under a clearly stated model. It becomes misleading when treated as the main summary of a result or as a verdict on truth.
- Is a p-value useless?
Report the estimate, the confidence interval, and the p-value, then interpret the size and precision in context.
- What should I report instead of just “p < 0.05”?
Statistical significance concerns how incompatible the data are with a null model. Practical significance concerns whether the estimated effect is large enough to matter for the real-world question.
- What is the difference between statistical significance and practical significance?
Large samples make estimates more precise, so even small differences can be detected. That does not make the effect important; it makes the estimate precise.
- Why does a very large study often give a tiny p-value?
Not in the usual frequentist sense. It is a range from a procedure with known long-run behavior. For a single interval, the parameter is either inside it or not.
- Does a confidence interval give the probability the true value is inside it?
That usually means the data are too noisy to be confident. The right response is to acknowledge uncertainty and, if relevant, collect more data or improve the design.
- What if my interval includes zero but the point estimate looks large?
Not necessarily. The problem is over-reliance and poor interpretation. A p-value can support a result when reported alongside the estimate and interval, and when the model and question are clearly stated.
- Should I stop using p-values entirely?
Is a p-value useless?
No. It can be useful as one piece of evidence under a clearly stated model. It becomes misleading when treated as the main summary of a result or as a verdict on truth.
What should I report instead of just p 0.05?
Report the estimate, the confidence interval, and the p-value, then interpret the size and precision in context.
What is the difference between statistical significance and practical significance?
Statistical significance concerns how incompatible the data are with a null model. Practical significance concerns whether the estimated effect is large enough to matter for the real-world question.
Why does a very large study often give a tiny p-value?
Large samples make estimates more precise, so even small differences can be detected. That does not make the effect important; it makes the estimate precise.
Does a confidence interval give the probability the true value is inside it?
Not in the usual frequentist sense. It is a range from a procedure with known long-run behavior. For a single interval, the parameter is either inside it or not.
What if my interval includes zero but the point estimate looks large?
That usually means the data are too noisy to be confident. The right response is to acknowledge uncertainty and, if relevant, collect more data or improve the design.
Should I stop using p-values entirely?
Not necessarily. The problem is over-reliance and poor interpretation. A p-value can support a result when reported alongside the estimate and interval, and when the model and question are clearly stated.