Outliers in Statistics: Detect Them, Then Decide Deliberately

Every dataset has a value that looks wrong. Sometimes it is a decimal point that moved, sometimes it is a genuine extreme case, and sometimes it is the most informative observation in the study. All three look identical in a sorted column, which is why the handling of outliers is a judgement problem rather than a cleaning step.

The default behaviour — delete anything beyond two standard deviations and rerun — is indefensible. It removes the tail of the distribution, weakens every inference that follows, and, if the outlier was real, answers a question nobody asked.

Extreme is not the same as influential

These two ideas get merged constantly, and separating them is the first useful step. An extreme value is unusual on one variable. An influential value changes the answer: remove it and the estimate or its standard error moves materially.

Case	Unusual in X?	Unusual in Y?	Effect on the fit
Outlier in Y only	No	Yes	Large residual; pulls the line toward itself
High-leverage point	Yes	No	Little residual, but large potential to rotate the line
Influential point	Yes	Yes	Rotates the line; removal changes the slope noticeably
Typical point	No	No	None

A point far out on the x-axis with an ordinary y value has a small residual, so a residual-only screen misses it, yet it can move a slope more than any residual outlier. Leverage and influence statistics exist precisely because residual magnitude is not enough.

Detection: what each method actually finds

No single rule is correct, and the popular ones measure different things. Use at least two, then look at the observations themselves.

# R: univariate rules
x <- dat$value
range(x)
boxplot.stats(x)$out                       # IQR rule: beyond 1.5 * IQR
median(x) + c(-3, 3) * mad(x)              # MAD: resistant to the outlier

The IQR rule flags values beyond 1.5 interquartile ranges from the box. It is resistant to masking, unlike the standard-deviation rule, which is computed from the mean and standard deviation — both of which the outlier itself inflates, so the very extreme value that should be flagged widens the fence that is supposed to catch it.

Method	Finds	Weakness
IQR rule (1.5 × IQR)	Univariate extremes	Assumes roughly symmetric tails; flags many values in skewed data
z-score > 3	Univariate extremes	Mean and SD are themselves affected by the outlier
Modified z-score (MAD)	Univariate extremes, robustly	Needs a sensible scale; not model-aware
Studentised residuals	Poorly fit by the model	Misses high-leverage points with small residuals
Hat values (leverage)	Unusual predictor combinations	Says nothing about the outcome
Cook’s distance	Influence on the fitted coefficients	Thresholds are conventions, not laws

# R: model-based diagnostics
fit <- lm(y ~ x, data = dat)
plot(fit, which = 5)                     # residuals vs leverage
car::outlierTest(fit)                    # Bonferroni-corrected residual test
car::influenceIndexPlot(fit)             # Cook's D, hat, DFBETAS
car::influencePlot(fit)

# Python
import statsmodels.api as sm
infl = fit.get_influence()
cooks = infl.cooks_distance[0]
lev   = infl.hat_matrix_diag
resid = infl.resid_studentized_external

Cook’s distance combines residual size and leverage into one number estimating how much the fitted values change when an observation is removed. The common cutoffs — above 1, or above 4/n — are conventions borrowed from practice rather than theory. Treat a list of flagged points as a reading list, not a verdict.

The decision tree that matters

Once a value is flagged, there are three possibilities and only three. Each has a different correct action, and none of them is silent deletion.

Diagnosis	Evidence	Action
Data error	Impossible value, unit mismatch, duplicated row	Correct if the true value is recoverable; otherwise exclude and document
Genuine extreme case	Plausible value, verified against source	Keep; consider a model that accommodates it
Influence of the model choice	Heavy tail or skew, not a single bad point	Change the model, not the data

The third row is where most analyses go wrong. A single value that is not an error and not typical is not necessarily the problem — often the model’s distributional assumption is. A log transformation, a count model, or a robust estimator can make the same data fit well without touching a single observation.

Robust alternatives that keep the data

Robust statistics exist to answer the question without excluding anything. The median and interquartile range are the obvious univariate examples; trimmed means, Winsorised means and Huber-type estimators extend the idea to estimation and regression.

# R
mean(x, trim = 0.1)                      # 10% trimmed mean
median(x)

library(MASS)
rlm(y ~ x, data = dat)                   # robust regression, Huber weights

library(robustbase)
lmrob(y ~ x, data = dat)

Winsorising — replacing the most extreme values with the nearest retained value — is popular in some fields and should be reported explicitly, because it changes the estimand. A Winsorised mean is not an estimate of the mean; it is an estimate of a trimmed quantity that must be named.

For inference rather than estimation, rank-based tests are the most familiar robust option. They compare distributions using ranks, so no single observation can dominate the result, at the cost of no longer testing a hypothesis about means.

When outliers carry the finding

In rare-event research, the extreme observations are the subject. Adverse drug reactions, financial crashes, and epidemic superspreaders are defined by their tails, and an analysis that removes them has removed the phenomenon. The same is true of measurement studies where the outlier is the instrument failure you were trying to characterise.

Ask what the study is about before deciding what counts as anomalous. If the tail is the topic, report it prominently and use methods designed for it rather than engineering it away.

How to report a decision

A reviewer should be able to reproduce your judgement without contacting you. State the rule used to flag values, how many were flagged, what you found when you inspected them, what you did, and what happened to the estimates when you did it.

A sensitivity analysis is the strongest form of this. Report the primary analysis on all data, then the same analysis with flagged values excluded, and show that the conclusion is stable — or state plainly that it is not. An analysis whose conclusion flips when one observation is removed is a finding about fragility, not a result to be tidied up.

Reporting element	Example
Rule applied	IQR rule on the outcome; Cook’s D above 4/n for influence
Count flagged	Six of 412 observations exceeded the IQR fence
Inspection outcome	Four verified against source records; two were data-entry errors
Action taken	Two corrected; all six retained, with a sensitivity analysis excluding them
Effect on conclusions	Slope changed by 0.03 (95% CI -0.01 to 0.07); conclusions unchanged

What not to do

• Do not remove values because they weaken a result you wanted.
• Do not apply a fixed rule mechanically across variables with different distributions.
• Do not report a trimmed analysis without saying it was trimmed.
• Do not confuse a failed normality test with evidence of a data error.
• Do not silently impute an excluded value as if it never existed.

Handling extremes well is part of a wider discipline of honest modelling, and the same instincts apply to assumption checks and sensitivity analyses. Ampersand Academy covers those workflows alongside robust estimation.

Practice

1. Apply the IQR rule and the MAD rule to a skewed variable and compare which values each flags.
2. Fit a regression, list the three most influential points by Cook’s distance, and inspect them individually.
3. Refit with those points excluded and report how much the slope and its interval moved.
4. Choose a robust or transformed alternative and argue why it is preferable to exclusion.

Related tutorials

• Checking Statistical Assumptions in Practice: What to Plot, What to Trust, What to Do Next — where outlier checks sit among the other diagnostics
• Non-Parametric Tests Explained: What Rank Tests Actually Compare — rank-based tests that no single observation can dominate
• Correlation Coefficient: How to Read r Correctly — how one extreme point moves a correlation coefficient