ROC Curves and AUC: Evaluating Diagnostic Tests and Classifiers

A classifier that labels 92% of cases correctly sounds good until you learn that 92% of the sample is negative — and it predicted negative for everything. Accuracy is a single number that hides the trade-off between missing cases and raising false alarms, and that trade-off is usually the entire decision.

ROC analysis makes the trade-off visible. It plots every achievable combination of sensitivity and specificity, and AUC summarises the whole curve in one number. Both are widely used and widely over-interpreted.

Sensitivity, specificity, and the threshold problem

Most models output a score, not a label. Turning a score into a decision requires a threshold, and every threshold makes a choice.

Sensitivity is the proportion of true positives you catch; specificity is the proportion of true negatives you correctly clear. They trade against each other at every threshold, and no amount of modelling removes that constraint — it is a property of the data, not the algorithm.

# R
library(pROC)
r <- roc(response = df$truth, predictor = df$score,
         levels = c("control", "case"), direction = "<")
r$auc
coords(r, "all", ret = c("threshold", "sensitivity", "specificity"))

# Python
from sklearn.metrics import roc_curve, roc_auc_score
fpr, tpr, thresholds = roc_curve(y_true, y_score)
auc = roc_auc_score(y_true, y_score)

How the ROC curve is built

Sweep the threshold from high to low. At each value, compute sensitivity (true positive rate) and 1 − specificity (false positive rate). Plot one against the other. The curve starts at (0, 0) — everything below the highest score is negative, so nothing is caught — and ends at (1, 1), where everything is called positive.

A curve hugging the top-left corner indicates a model that separates the two classes well. The diagonal is the performance of random guessing. A curve below the diagonal means the model is systematically inverted, which usually signals a sign error in the score rather than a bad model.

Reading AUC correctly

AUC has a precise and surprisingly simple interpretation: the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case. That is a statement about ranking, not about accuracy at any threshold.

Those bands are conventions, not thresholds for action, and their meaning depends entirely on the domain. Now the part that trips people up.

Three things AUC does not tell you

1. It does not tell you the accuracy at your chosen threshold. A high AUC with a badly chosen cut point still performs poorly in practice.
2. It does not tell you calibration. A model can rank perfectly and still predict 0.9 for an event that happens 40% of the time.
3. It is insensitive to class balance, which is a strength for comparison and a trap for deployment: with 1% prevalence, a 0.85 AUC can still mean a positive predictive value below 10%.

# Physical interpretation: proportion of correctly ordered pairs
# (identical to the Mann-Whitney U statistic)
from scipy.stats import mannwhitneyu
u, p = mannwhitneyu(y_score[y_true == 1], y_score[y_true == 0])
auc_equiv = u / (sum(y_true == 1) * sum(y_true == 0))

That equivalence is worth remembering: AUC is a rank statistic, which is why it is invariant to any monotone transformation of the scores. It also explains why it says nothing about the actual probability values your model emits.

Choosing a threshold from costs

The right threshold is a decision about consequences, not a statistical property. If missing a case costs 20 times more than a false alarm, the optimal threshold shifts far toward sensitivity — and the correct threshold can be derived rather than guessed.

# Youden's J: maximises sensitivity + specificity - 1
coords(r, "best", best.method = "youden",
       ret = c("threshold", "sensitivity", "specificity"))

# Cost-weighted optimal threshold: treat cost as the ratio
coords(r, "best", best.weights = c(1, 20),
       ret = c("threshold", "sensitivity", "specificity"))

Youden’s index treats false positives and false negatives as equally bad, which is a strong assumption made silently by almost everyone who uses the best shortcut. If the two errors genuinely cost the same, fine — say so. If they do not, pass the weights.

Report the confusion matrix at the threshold you chose

# R
pred <- ifelse(df$score >= 0.31, "case", "control")
table(predicted = pred, actual = df$truth)

caret::confusionMatrix(factor(pred), factor(df$truth), positive = "case")

A curve plus an AUC, with no stated threshold and no confusion matrix, leaves a reader unable to tell what the test would actually do to real people.

When precision-recall beats ROC

For rare outcomes — fraud, adverse events, rare disease screening — the false positive rate is a misleading denominator because there are so many true negatives that a large number of false positives barely moves it. Precision-recall curves show the cost directly: precision is the share of flagged cases that are real.

from sklearn.metrics import precision_recall_curve, average_precision_score

precision, recall, _ = precision_recall_curve(y_true, y_score)
ap = average_precision_score(y_true, y_score)   # better than AUC here

The baseline for a precision-recall curve is the prevalence itself, not 0.5 — a no-skill classifier achieves precision equal to prevalence. Comparing an average precision of 0.3 against a 1% base rate is a decent result; against 40%, it is not.

Comparing two models

An AUC point estimate without an interval invites over-reading. Two models differing by 0.02 AUC, with overlapping confidence intervals, are not distinguishable, and claiming otherwise is how small differences get published as improvements.

# DeLong test for two correlated ROC curves
roc.test(r_model1, r_model2)

# Or bootstrap the AUC difference
boot_auc_diff <- function(d, i) {
  s <- d[i, ]
  as.numeric(auc(roc(s$truth, s$m1, quiet = TRUE)) -
             auc(roc(s$truth, s$m2, quiet = TRUE)))
}
boot.ci(boot(df, boot_auc_diff, R = 2000), type = "bca")

Compare models on the same test set with the same observations — the DeLong test accounts for the correlation that comes from evaluating both on identical cases. And never compare AUCs computed on different cohorts; the differences in case mix make the numbers incomparable.

What to report

Report the AUC with a confidence interval, the ROC curve, the threshold you chose and the reasoning behind it, the confusion matrix at that threshold, sensitivity and specificity with intervals, and the calibration picture. If prevalence is low, add precision-recall and the positive predictive value.

Evaluation is where a model becomes a decision, and it is the step most often skipped. If you want to go from discrimination to a defensible threshold policy, Ampersand Academy covers that reasoning in depth.

Practice

1. Build an ROC curve for a real dataset and compute AUC from the Mann-Whitney U statistic to confirm the equivalence.
2. Pick a threshold with Youden’s index, then again with a 10:1 cost ratio, and compare the confusion matrices.
3. Compute positive predictive value at 1% prevalence for the same sensitivity and specificity.
4. Plot precision-recall for the same model and explain where the ROC curve was flattering.

Related tutorials

• Bootstrapping and Resampling: Confidence Intervals Without Distributional Assumptions — putting a confidence interval around AUC without a closed form
• Non-Parametric Tests Explained: What Rank Tests Actually Compare — the rank statistic that AUC is mathematically equivalent to
• Effect Size in Statistics: Cohen d, r and Eta-Squared — expressing the size of a group difference alongside discrimination