[Durgical](https://durgical.com/en/) › [DurgiPedia](https://durgical.com/en/durgipedia/) › [Epidemiology](https://durgical.com/en/epidemiology/) › [Biostatistics](https://durgical.com/en/epidemiology/areas/stats/) › Multiple comparisons

# Multiple comparisons

- Concept

The more tests are run, the easier it is for one to come out significant by chance alone. Correct for it, or fix in advance what will be tested.

Updated Nov 5, 2026

## Key points

- Probability of at least one false positive with k independent tests = 1 − (1 − α)^k.
- With 20 tests at α = 0.05: 1 − 0.95²⁰ ≈ 64%.
- Bonferroni: use α / k for each test; simple but conservative (it loses power).
- The primary outcome and subgroups are registered before the study (protocol and trial registry).
- After a significant ANOVA, pairwise comparisons use adjusted methods (Tukey, Bonferroni).

## What it is

Each test at α = 0.05 carries a 5% risk of a false positive when there is no effect; across many tests, the chance of at least one false positive grows quickly. This happens when many outcomes or subgroups are analysed, or when data are looked at repeatedly. It is controlled with a primary outcome fixed in advance, with corrections such as Bonferroni and with caution in reading subgroup findings.

## Worked example

- Data: a study compares two groups on 20 independent outcomes, none with a real effect. α = 0.05 per test.
- Probability that one test gives no false positive = 0.95.
- Probability that none of the 20 does = 0.95²⁰ = 0.358.
- Probability of at least one false positive = 1 − 0.358 = 0.642, about 64%.
- With Bonferroni, α per test = 0.05 / 20 = 0.0025; the overall risk falls to 1 − 0.9975²⁰ ≈ 4.9%.
- An outcome with p = 0.01 would look significant, but it does not pass the corrected threshold of 0.0025.
- Reading: in a study with many outcomes, a single isolated p
- α / k
- Bonferroni

## Common pitfalls

- Reporting only the outcomes or subgroups that came out significant.
- Changing the primary outcome after seeing the results.
- Applying Bonferroni to dozens of highly correlated tests, which cuts power more than needed.

## Memory aid

‘Buy enough tickets and something wins’: many tests, some ‘finding’ by chance.

## Test yourself

Think of the answer, then tap the question to see it.

## What is the chance of at least one false positive with 10 independent tests at α = 0.05?

1 − 0.95¹⁰ ≈ 0.40, about 40%.

## How does the Bonferroni correction work?

It divides α by the number of tests: with 5 tests at α = 0.05, each is judged at 0.01.

## What is the main drawback of Bonferroni?

It is conservative: demanding very low thresholds loses power and increases type II errors.

## Why should the primary outcome and subgroups be fixed in advance?

To avoid choosing afterwards the analysis that gave p

[Add to my cards (4)](https://durgical.app/estudiar/?add=/en/epidemiology/multiple-comparisons/)

## Connections

## Related 4

- [Type I and type II errors](https://durgical.com/en/epidemiology/type-i-and-type-ii-errors/)
- [P value](https://durgical.com/en/epidemiology/p-value/)
- [Analysis of variance (ANOVA)](https://durgical.com/en/epidemiology/analysis-of-variance/)
- [Hypothesis testing](https://durgical.com/en/epidemiology/hypothesis-testing/)

[← All of epidemiology](https://durgical.com/en/epidemiology/)

Educational content on epidemiological and statistical methods; the examples use made-up numbers to teach the calculation. Not a clinical or legal guide: current Colombian rules are in the official sources.

Sources: [How can we make multiple comparisons?](https://www.itl.nist.gov/div898/handbook/prc/section4/prc47.htm) (NIST/SEMATECH e-Handbook of Statistical Methods, National Institute of Standards and Technology, 2012); [Bonferroni's method](https://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm) (NIST/SEMATECH e-Handbook of Statistical Methods, National Institute of Standards and Technology, 2012).

## **Spotted a mistake?** We are always improving for the community. Tell us what is wrong and our team will review it. Report a mistake

Representative model, for educational purposes only.

---
Source: https://durgical.com/en/epidemiology/multiple-comparisons/
License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Cite as: DurgiPedia (durgical.com), https://durgical.com/en/epidemiology/multiple-comparisons/