Bootstrapping and Resampling: Confidence Intervals Without Distributional Assumptions

The textbook confidence interval for a mean needs a formula and an assumption. The same for a difference in proportions, a correlation, or a regression coefficient — there is a formula, but it assumes something about the distribution and quietly stops being trustworthy when the data are skewed or the sample is small.

The bootstrap takes a different route. If your sample is a reasonable stand-in for the population, then resampling from your sample approximates sampling from the population. Repeat that a thousand times and you have an empirical distribution for any statistic you can compute — no formula required.

The idea, in one paragraph

Draw n observations with replacement from your n observations. Some rows appear twice, some not at all — on average about 63% of the original rows appear at least once. Compute your statistic on that resample. Repeat 1,000 to 10,000 times. The spread of those statistics estimates the sampling distribution of your statistic, and its percentiles give you a confidence interval.

# R, minimal and explicit
set.seed(42)
x <- c(12, 15, 9, 22, 40, 18, 11, 14, 19, 55)

boot_means <- replicate(5000, mean(sample(x, length(x), replace = TRUE)))
quantile(boot_means, c(0.025, 0.975))
sd(boot_means)          # bootstrap standard error

# Python
import numpy as np
rng = np.random.default_rng(42)
x = np.array([12, 15, 9, 22, 40, 18, 11, 14, 19, 55])

boot = rng.choice(x, size=(5000, x.size), replace=True).mean(axis=1)
np.quantile(boot, [0.025, 0.975])

Nothing about that code assumes normality. That is the entire appeal — and also the reason people apply it where it does not belong.

Percentile, basic and BCa intervals

The percentile interval is just the 2.5th and 97.5th percentiles of the bootstrap distribution. It is simple, transformation-respecting, and biased when the statistic’s sampling distribution is skewed — which is exactly the situation where you reached for the bootstrap in the first place.

Use BCa by default. It corrects for bias and for the fact that the standard error may change with the true value, which matters for ratios, correlations and anything near a boundary.

library(boot)

mean_stat <- function(d, i) mean(d[i])
b <- boot(x, mean_stat, R = 5000)
boot.ci(b, type = c("perc", "bca"))

# Python — BCa is available directly
from scipy.stats import bootstrap
res = bootstrap((x,), np.mean, n_resamples=5000,
                confidence_level=0.95, method="BCa")
print(res.confidence_interval)

Bootstrapping things that have no formula

This is where the method earns its place. Medians, ratios of two medians, trimmed means, the 90th percentile, differences in AUC between two models, R² from a nonlinear fit — none of these have tidy closed-form intervals, and all of them bootstrap easily.

# A ratio of medians — no closed-form CI, trivial to bootstrap
ratio_stat <- function(d, i) {
  s <- d[i, ]
  median(s$value[s$group == "a"]) / median(s$value[s$group == "b"])
}
b2 <- boot(paired_data, ratio_stat, R = 5000)
boot.ci(b2, type = "bca")

Note the resampling unit. When data are paired or clustered — repeated measurements per patient, students within classrooms — resample the whole cluster, not individual rows. Row-level resampling of clustered data understates the standard error and produces intervals that are too narrow.

Permutation tests: the sibling method

A permutation test answers a hypothesis-testing question rather than an estimation one. Under the null hypothesis of no group difference, group labels are exchangeable: shuffle them, recompute the statistic, and see how often the shuffled statistic is as extreme as the observed one. That proportion is your p-value.

set.seed(1)
obs <- mean(group_a) - mean(group_b)
pooled <- c(group_a, group_b)
n_a <- length(group_a)

perm <- replicate(10000, {
  s <- sample(pooled)
  mean(s[1:n_a]) - mean(s[(n_a + 1):length(s)])
})
mean(abs(perm) >= abs(obs))     # two-sided p-value

Permutation gives an exact-ish p-value with almost no assumptions, which is why it is the right default for small samples and unusual statistics. It does not give you a confidence interval — use the bootstrap for that.

When the bootstrap fails

The bootstrap is not assumption-free; it assumes your sample resembles the population. Four situations break it.

• Extreme values or heavy tails: a single outlier dominates every resample, and the interval is meaningless. The sample must contain the tail behaviour you are trying to infer.
• Statistics near a boundary: a proportion of 0 or 1, a correlation of ±1 — the sampling distribution is not resampling-shaped.
• Very small samples: with n below roughly 20 the empirical distribution is too crude to stand in for the population.
• Dependent data resampled naively: described above, and the most common practical error.

For maxima and minima the bootstrap is notoriously unreliable because the resampled maximum can never exceed the observed maximum. If your quantity is an extreme value, use a parametric extreme-value model.

How many replicates?

A few thousand is plenty for a 95% interval; BCa benefits from more because it estimates tail quantities. Ten thousand is cheap insurance and takes seconds on modern hardware. Set a seed and report it — a bootstrap you cannot reproduce is a random number you published.

What to report

Give the statistic, the interval type (percentile or BCa), the confidence level, the number of replicates, and the resampling unit if the data are clustered. If a permutation test was used for the p-value, say how many permutations. These four facts make the result checkable, which a bare p-value never is.

Resampling is one tool in a broader problem: making an estimate honest about its uncertainty. If you want that reasoning laid out from design through inference, Ampersand Academy covers the whole sequence.

Practice

1. Compute a percentile and a BCa interval for the same median and compare their widths.
2. Bootstrap a ratio of two medians and explain why no formula was available.
3. Resample rows of a clustered dataset, then resample clusters, and compare the standard errors.
4. Run a permutation test on a small two-group comparison and compare its p-value to a t-test.

Related tutorials

• Confidence Intervals Explained: Read Them Like a Reviewer — the intervals that bootstrapping is usually replacing
• Checking Statistical Assumptions in Practice: What to Plot, What to Trust, What to Do Next — recognising the situations where resampling is the honest option
• Non-Parametric Tests Explained: What Rank Tests Actually Compare — the other approach when a distributional assumption fails