Agreement and the kappa statistic
- Measure
Kappa measures how much two observers agree beyond what they would agree on by pure chance.
What it is
When two people classify the same cases (for example, two radiologists reading the same films), some of their agreement would happen by chance alone. Cohen's kappa compares observed agreement with the agreement expected by chance: it is 1 for perfect agreement, 0 when agreement is no better than chance, and can be negative when it is worse. It measures inter-observer reliability, not validity.
Formula and reading
- Po = (positive agreements + negative agreements) / total.
- Pe = (A's share of positives × B's) + (A's share of negatives × B's).
- κ = (Po − Pe) / (1 − Pe).
- In health care, some authors consider a kappa below 0.60 inadequate.
Worked example
- Data: two radiologists read 100 films. Both positive 30, both negative 50, A+ B− 12, A− B+ 8.
- Po = (30 + 50) / 100 = 0.80.
- A calls 42 positive (30 + 12) and B 38 (30 + 8); negatives: A 58 and B 62.
- Pe = 0.42 × 0.38 + 0.58 × 0.62 = 0.1596 + 0.3596 = 0.5192.
- κ = (0.80 − 0.5192) / (1 − 0.5192) = 0.2808 / 0.4808 ≈ 0.58.
- Reading: although they agree on 80% of films, agreement beyond chance is moderate (κ ≈ 0.58).
The two radiologists' readings in the example
| Radiologist A | B positive | B negative | Total A |
|---|---|---|---|
| Positive | 30 | 12 | 42 |
| Negative | 8 | 50 | 58 |
| Total B | 38 | 62 | 100 |
Markers
- κ
- Po
- Pe
How to read it
- Many physical examination signs and image readings have moderate kappas: worth knowing before basing a decision on a single finding.
- Before a study, raters are trained and their kappa measured to make sure they classify consistently.
Common pitfalls
- Reporting only percentage agreement, which looks high even when much of it is chance.
- Using kappa as if it measured accuracy against the truth.
- Forgetting that with extreme prevalence kappa can be low despite high observed agreement.
Test yourself
Think of the answer, then tap the question to see it.
Po = 0.90 and Pe = 0.60. What is kappa?
κ = (0.90 − 0.60) / (1 − 0.60) = 0.30 / 0.40 = 0.75: substantial agreement.
What does a kappa of 0 mean?
Observed agreement is the same as would be expected by chance.
Two pathologists have a kappa of 0.9. Does that guarantee their diagnoses are correct?
No. Kappa measures agreement between them (reliability); they can share the same error.
Connections
Educational content on epidemiological and statistical methods; the examples use made-up numbers to teach the calculation. Not a clinical or legal guide: current Colombian rules are in the official sources.
Sources: Interrater reliability: the kappa statistic (Biochemia Medica, 2012); Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests - Guidance for Industry and FDA Staff (U.S. Food and Drug Administration, 2007).
Spotted a mistake?We are always improving for the community. Tell us what is wrong and our team will review it.Report a mistake
Representative model, for educational purposes only.