Courses/Behavioral Research: Statistical Methods

Behavioral Research: Statistical Methods

CG3.402

Vinoo Alluri•Monsoon 2025-26•4 credits

Revision Notes/Unit 8 — Multiple Comparisons (FWER, FDR)

Unit 8 — Multiple Comparisons (FWER, FDR)

Family-wise type-I inflation, Bonferroni and Holm corrections (FWER), Benjamini-Hochberg (FDR), permutation tests, and how multiple comparisons drive the replication crisis.

FWER vs FDR; Bonferroni, Holm, BH

10 min

Run m independent tests at α = 0.05 each → P(at least one false positive) ≈ $1 - 0.95^m$. For m = 20: 64%! **Multiple comparisons inflate Type I error** far beyond nominal α. Two control strategies: **FWER (Family-Wise Error Rate)** — P(any FP) ≤ α, conservative, Bonferroni and Holm; **FDR (False Discovery Rate)** — E[FP/rejections] ≤ Q, less conservative, Benjamini-Hochberg. **Use FWER for few costly tests; FDR for many exploratory tests.**