Measurement Reliability: ICC, Cohen’s Kappa, and Cronbach’s Alpha Done Right

Two raters score the same forty scans. A questionnaire measures the same trait on two occasions. A lab assay is run in duplicate. In each case the question is the same: how much of the variation in the numbers is signal, and how much is noise from the instrument, the rater, or the occasion?

Reliability quantifies that ratio. It matters more than most researchers treat it, because unreliable measurement does not merely add noise — it systematically shrinks the effects you go on to estimate, and it does so invisibly.

Reliability is not validity

Reliability asks whether a measurement is consistent across repetitions. Validity asks whether it measures the construct it claims to. A scale that consistently reports the same wrong number is perfectly reliable and entirely invalid; a scale that fluctuates wildly might still, on average, be centred on the truth.

That last row is the one worth internalising: correlation measures whether two measurements move together, not whether they agree. Two assays can correlate at 0.98 while one reads systematically 15% higher, which matters enormously if you plan to swap them.

Cohen’s kappa for categorical agreement

For two raters assigning categories, raw percentage agreement is misleading because some agreement happens by chance — badly so when one category dominates. Kappa corrects for that chance agreement.

# R
library(irr)
kappa2(ratings[, c("rater1", "rater2")], weight = "unweighted")

# Python
from sklearn.metrics import cohen_kappa_score
cohen_kappa_score(rater1, rater2)

Two cautions. The kappa paradox: when one category is very common, kappa can be low even when raw agreement is high, because chance agreement is high — so report the prevalence and the raw agreement alongside it. And for ordered categories, use weighted kappa so a one-step disagreement counts less than a three-step one.

kappa2(ratings[, c("rater1", "rater2")], weight = "squared")   # ordinal

For three or more raters, use Fleiss kappa. Note that it requires the same number of ratings per subject and, unlike Cohen’s kappa, does not assume the same raters evaluated everyone.

ICC for continuous measures

The intraclass correlation coefficient compares between-subject variance to total variance. An ICC of 0.90 means 90% of the observed variance is real between-subject difference and 10% is measurement noise.

ICC is not one statistic but a family, and the choice among them changes the number. Reporting ICC without stating the model is a genuine reporting defect.

library(psych)
ICC(ratings[, c("rater1", "rater2")],
    model = "twoway", type = "agreement", unit = "single")

The single-versus-average distinction is practically important. Averaging three raters raises reliability substantially, so a study reporting ICC(2,3) of 0.92 is not evidence that a single rater is that reliable — and if your clinical workflow uses one rater, the single-measure ICC is the one that matters.

For method comparison specifically, Bland-Altman limits of agreement are usually more informative than ICC: they tell you the range within which most differences between two measurements fall, in the original units.

library(blandr)
blandr.statistics(method1 = assay_a, method2 = assay_b, sig.level = 0.05)

Cronbach’s alpha and its misuse

Alpha estimates the internal consistency of a set of items intended to measure one construct. It is the most reported statistic in psychometrics and the most routinely misinterpreted.

• Alpha is not a measure of unidimensionality. A scale can have high alpha while containing two distinct factors.
• Alpha rises mechanically with the number of items, so a 40-item scale can hit 0.90 with weak inter-item correlations.
• Alpha assumes tau-equivalence — that all items load equally on the construct — which is rarely true.
• A high alpha can reflect redundant items that add length without adding information.

Omega (McDonald’s) does not assume equal loadings and is generally the better default, which is why methodological guidance increasingly recommends reporting it instead of, or alongside, alpha.

library(psych)
psych::alpha(scale_items)      # raw and standardised alpha
psych::omega(scale_items, nfactors = 1)

A related trap: alpha is for items measuring one construct. Computing it across a set of variables that are not intended to be a scale — assay results, demographic fields — produces a number that means nothing at all.

Standard error of measurement: the practical number

Reliability coefficients are abstract. The standard error of measurement translates them into the units you actually use, telling you how much a single observed score is likely to deviate from the true value.

# SEM = SD * sqrt(1 - reliability)
sem <- sd(scores) * sqrt(1 - 0.88)
# A 95% band around one measurement is roughly +/- 1.96 * SEM
c(lower = -1.96 * sem, upper = 1.96 * sem)

That calculation is where reliability stops being a checkbox. If a patient’s score is 62 and the 95% band is ±11 points, then a follow-up score of 68 is not improvement — it is noise. Minimum detectable change follows directly from this and is what clinical reporting should quote.

Attenuation: how unreliable measurement hides real effects

Measurement error in a predictor biases its estimated association toward zero. This is attenuation, and its size is governed by the reliability of the measure: an observed correlation is roughly the true correlation multiplied by the square root of the reliability product.

The consequence is a specific and common failure: a study with a crude measure finds a weak association and concludes the effect is small, when the true effect is moderate and the instrument was simply noisy. Correction is possible — disattenuation formulas, or latent-variable models that model the measurement error explicitly — but only if reliability was reported in the first place.

What to report

Name the statistic, the model, and the unit (single or average). Report the confidence interval around the reliability estimate — reliability is itself estimated from a finite sample. For agreement between methods, include limits of agreement in the original units. And if your measure is a scale, report omega alongside or instead of alpha, with the number of items.

Reliability is the ceiling on every conclusion downstream of a measurement, which makes it worth more attention than it usually gets. If you want to see how it constrains study design and sample size, Ampersand Academy covers that reasoning.

Practice

1. Compute Cohen’s kappa and raw agreement for one dataset and explain the gap between them.
2. Compute ICC(2,1) and ICC(2,3) on the same ratings and state which one your workflow should use.
3. Calculate the standard error of measurement and a minimum detectable change from a published reliability figure.
4. Estimate the attenuation for a correlation of 0.4 when reliability is 0.6, and compare with 0.9.

Related tutorials

• Correlation Coefficient: How to Read r Correctly — why a high correlation does not mean two measures agree
• Effect Size in Statistics: Cohen d, r and Eta-Squared — how unreliable measurement attenuates the effect you report
• data-types-in-statistics — choosing an agreement statistic that fits the measurement scale