Category: Multiple comparisons. The use of multiple statistical tests
in a wide range of contexts, raises serious concerns. The proposed solutions to
these concerns are very controversial. These pages discuss some of the concerns
and the debate over the appropriate remedies. Articles are arranged by date with the most recent entries at the top. You can
find the theme and closely related
categories and other resources at the bottom of this page.
Stats: When a client asks for a bad
analysis (March 24, 2008). I received an email from someone who was being
asked to perform a subgroup analysis that is likely to produce confusing and
counter-intuitive results. I was asked to help draft some language to convince
the client that this was a bad idea.
Stats: Subgroup analysis
(December 21, 2004). A recently published trial shows a logical approach
for establishing the validity of a subgroup comparison.
Stats: Bonferroni correction (September 3,
1999). Dear Professor Mean: I keep reading about something called a
Bonferroni correction. Somehow this method keeps researchers from going on a
fishing expedition. Could you explain what a Bonferroni correction is and why
we want to keep scientists from fishing? -- Judicious John
Theme and closely related categories:
Other resources:
- Carlo Emilio Bonferroni
Description: This website provides information about the mathematician whose
work led to the Bonferroni correction.
- Decision theoretic designs for
Phase II clinical trials with multiple outcomes. Description: This
article provides a Bayesian approach to handling multiple comparisons in a
trial where with multiple safety and efficacy endpoints.
- Do multiple outcome measures
require p-value adjustment?. Description: This article criticizes the
use of Bonferroni corrections. It suggests that readers should instead
assess the quality of the study, the magnitude of the effect, and consider
results from similar studies. Researchers should select a primary outcome or
use a single composite measure rather than relying on Bonferroni.
- Empirical-Bayes adjustments for
multiple comparisons are sometimes useful. Description: This article
proposes situations where adjustments for multiple comparisons are
appropriate. The authors offer Empirical-Bayes and fully Bayesian approaches
and describe their advantages over the traditional Bonferroni approach.
- False positive outcomes and
design characteristics in occupational cancer epidemiology studies.
Description: This article reviews a series of false positive conclusions in
epidemiologic research. The authors find that failure to develop a specific
a priori hypothesis led to a three fold greater risk of producing a false
positive conclusion.
- Methods of correcting for
multiple testing: operating characteristics. Description: This
article reviewed 17 different methods for adusting p-values, including the
Bonferroni correction, in a computer simulation. There was no uniformly best
approach, but as a group, four methods did appear to be better than the
rest.
- Multiple significance tests and
the Bonferroni correction Description: This website provides a simple
introduction to the Bonferroni correction. It is an excerpt from An
Introduction to Medical Statistics, Third Edition.
- Multiple Comparisons with
Repeated Measures. Excerpt: One of the commonly asked questions on
listservs dealing with statistical issue is "How do I use SPSS (or whatever
software is at hand) to run multiple comparisons among a set of repeated
measures?" This page is a (longwinded) attempt to address that question. I
will restrict myself to the case of one repeated measure (with or without a
between subjects variable), but the generalization to more complex cases
should be apparent.
- No adjustments are needed for
multiple comparisons. Description: This article argues strongly
against the use of Bonferroni adjustments. The author derides the concept of
a global null hypothesis and notes the serious increase in Type II errors
that occur with Bonferroni adjustments.
- Permutation Tests for Joinpoint
Regression with Applications to Cancer Rates. Description: This
article provides an illustrative example of a regression model with an
unknown transition points and controls the probability of a Type I error
using a Bonferroni correction.
- Problems in defining cutoff
points of continuous prognostic factors: example of tumor thickness in
primary cutaneous melanoma. Description: This article shows how
examining an optimized cutpoint for dichotomizing an independent variable
effectively produces multiple hypothesis tests and leads to an inflation of
the Type I error rate.
- Quantitative evaluation of
multiplicity in epidemiology and public health research. Description:
This article reviews 173 randomly selected epidemiology articles and
demonstrates an increase in the Type I error rare when multiple statistical
tests are run without any adjustment.
- Some comments on frequently
used multiple endpoint adjustment methods in clinical trials.
Description: This article uses a computer simulation to examine the
performance of several adjustments for multiple comparisons.
- There must be something buried
in here somewhere Description: This webpage uses a simulation to
illustrate what happens with twenty simultaneous independent tests of
significance.
- What's wrong with Bonferroni
adjustments. Description: This article criticizes the use of
Bonferroni adjustments, arguing that they create more problems than they
solve. The authors criticize the concept of a global null hypothesis and
point out the increase in the risk of Type II errors.
- Assessing cause and effect from trials: a cautionary note. D. Howel, R. Bhopal.
Control Clin Trials 1994: 15(5); 331-4.
[Medline]
- Invited Commentary: Re: "Multiple Comparisons and Related Issues in the Interpretation
of Epidemiologic Data". John R. Thompson. American Journal of Epidemiology 1998: 147(9);
801-811.
[Medline]
- The Method of Multiple Working Hypotheses. TC Chamberlin. The Scientific Monthly
1944: 59357 - 62.
- Multiple comparisons and related issues in the interpretation of epidemiologic data.
D. A. Savitz, A. F. Olshan. Am J Epidemiol 1995: 142(9); 904-8.
[Medline]
- Simultaneous Statistical Inference Second Edition. Rupert G. Miller (1981) New
York: Springer-Verlag.
- Multiple Comparisons Theory and Methods. Jason C. Hsu (1996) London: Chapman &
Hall.
[Return to full topic list]
[Read current weblog entries]
This webpage was written by Steve Simon on 2007-06-13, edited by Steve Simon, and was last modified on
2008-07-08. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page.