Category: Multiple comparisons. The use of multiple statistical tests in a wide range of contexts, raises serious concerns. The proposed solutions to these concerns are very controversial. These pages discuss some of the concerns and the debate over the appropriate remedies. Articles are arranged by date with the most recent entries at the top. You can find the theme and closely related categories and other resources at the bottom of this page.

Stats: When a client asks for a bad analysis (March 24, 2008). I received an email from someone who was being asked to perform a subgroup analysis that is likely to produce confusing and counter-intuitive results. I was asked to help draft some language to convince the client that this was a bad idea.

Stats: Subgroup analysis (December 21, 2004). A recently published trial shows a logical approach for establishing the validity of a subgroup comparison.

Stats: Bonferroni correction (September 3, 1999). Dear Professor Mean: I keep reading about something called a Bonferroni correction. Somehow this method keeps researchers from going on a fishing expedition. Could you explain what a Bonferroni correction is and why we want to keep scientists from fishing? -- Judicious John

Theme and closely related categories:

• Theme: Evidence based medicine
• Category: Statistical evidence

Other resources:

• Carlo Emilio Bonferroni Description: This website provides information about the mathematician whose work led to the Bonferroni correction.
• Decision theoretic designs for Phase II clinical trials with multiple outcomes. Description: This article provides a Bayesian approach to handling multiple comparisons in a trial where with multiple safety and efficacy endpoints.
• Do multiple outcome measures require p-value adjustment?. Description: This article criticizes the use of Bonferroni corrections. It suggests that readers should instead assess the quality of the study, the magnitude of the effect, and consider results from similar studies. Researchers should select a primary outcome or use a single composite measure rather than relying on Bonferroni.
• Empirical-Bayes adjustments for multiple comparisons are sometimes useful. Description: This article proposes situations where adjustments for multiple comparisons are appropriate. The authors offer Empirical-Bayes and fully Bayesian approaches and describe their advantages over the traditional Bonferroni approach.
• False positive outcomes and design characteristics in occupational cancer epidemiology studies. Description: This article reviews a series of false positive conclusions in epidemiologic research. The authors find that failure to develop a specific a priori hypothesis led to a three fold greater risk of producing a false positive conclusion.
• Methods of correcting for multiple testing: operating characteristics. Description: This article reviewed 17 different methods for adusting p-values, including the Bonferroni correction, in a computer simulation. There was no uniformly best approach, but as a group, four methods did appear to be better than the rest.
• Multiple significance tests and the Bonferroni correction Description: This website provides a simple introduction to the Bonferroni correction. It is an excerpt from An Introduction to Medical Statistics, Third Edition.
• Multiple Comparisons with Repeated Measures. Excerpt: One of the commonly asked questions on listservs dealing with statistical issue is "How do I use SPSS (or whatever software is at hand) to run multiple comparisons among a set of repeated measures?" This page is a (longwinded) attempt to address that question. I will restrict myself to the case of one repeated measure (with or without a between subjects variable), but the generalization to more complex cases should be apparent.
• No adjustments are needed for multiple comparisons. Description: This article argues strongly against the use of Bonferroni adjustments. The author derides the concept of a global null hypothesis and notes the serious increase in Type II errors that occur with Bonferroni adjustments.
• Permutation Tests for Joinpoint Regression with Applications to Cancer Rates. Description: This article provides an illustrative example of a regression model with an unknown transition points and controls the probability of a Type I error using a Bonferroni correction.
• Problems in defining cutoff points of continuous prognostic factors: example of tumor thickness in primary cutaneous melanoma. Description: This article shows how examining an optimized cutpoint for dichotomizing an independent variable effectively produces multiple hypothesis tests and leads to an inflation of the Type I error rate.
• Quantitative evaluation of multiplicity in epidemiology and public health research. Description: This article reviews 173 randomly selected epidemiology articles and demonstrates an increase in the Type I error rare when multiple statistical tests are run without any adjustment.
• Some comments on frequently used multiple endpoint adjustment methods in clinical trials. Description: This article uses a computer simulation to examine the performance of several adjustments for multiple comparisons.
• There must be something buried in here somewhere Description: This webpage uses a simulation to illustrate what happens with twenty simultaneous independent tests of significance.
• What's wrong with Bonferroni adjustments. Description: This article criticizes the use of Bonferroni adjustments, arguing that they create more problems than they solve. The authors criticize the concept of a global null hypothesis and point out the increase in the risk of Type II errors.
• Assessing cause and effect from trials: a cautionary note. D. Howel, R. Bhopal. Control Clin Trials 1994: 15(5); 331-4. [Medline]
• Invited Commentary: Re: "Multiple Comparisons and Related Issues in the Interpretation of Epidemiologic Data". John R. Thompson. American Journal of Epidemiology 1998: 147(9); 801-811. [Medline]
• The Method of Multiple Working Hypotheses. TC Chamberlin. The Scientific Monthly 1944: 59357 - 62.
• Multiple comparisons and related issues in the interpretation of epidemiologic data. D. A. Savitz, A. F. Olshan. Am J Epidemiol 1995: 142(9); 904-8. [Medline]
• Simultaneous Statistical Inference Second Edition. Rupert G. Miller (1981) New York: Springer-Verlag.
• Multiple Comparisons Theory and Methods. Jason C. Hsu (1996) London: Chapman & Hall.

[Return to full topic list] [Read current weblog entries]

This webpage was written by Steve Simon on 2007-06-13, edited by Steve Simon, and was last modified on 2008-07-08. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page.