Category: Systematic overviews. These pages discuss issues associated with a systematic overview (systematic review, meta-analysis). Articles are arranged by date with the most recent entries at the top. You can find the theme and closely related categories and other resources at the bottom of this page.

Stats: Criticism of random effects in a meta-analysis (June 14, 2008). There are two approaches to combining results in a meta-analysis. They are called the fixed effects model and the random effects model. The fixed effects model effectively weights each study by the sample size, or by a measurement that is closely related to the sample size, such as the inverse of the standard error of the estimate. A random effects meta-analysis, in contrast, will assume that an estimate from a single study has two sources of error. One error is the same as in the fixed effects analysis and varies by the sample size of the study. The other error is a random component that is independent of the sample size and represents uncertainties due to conditions in this particular study that differ from conditions in other studies.

Stats: Finding only the important studies (January 21, 2008). Someone wrote into the MedStats listserv asking about a process that they had chosen to select "important" articles in a particular research area. This was, I presume, a qualitative summary of interesting results in a broad medical area rather than a quantitative synthesis of all available research addressing a specific medical treatment. The reason I suspect this is that the person mentioned that they had used the statistical significance of the studies as a filter and eliminated any negative studies from further consideration.

Stats: Cherry picking the literature (December 20, 2006). I have a relative who loves to send me articles supporting a particular religious and political viewpoint that he endorses. While that viewpoint he espouses is usually conservative, the problems with the articles he cites are problems that plague both sides. These articles always have an impressive bibliography, as if to say "Look! It was published and peer-reviewed, so it must be true." The problem with these articles though is that the bibliography was created using a process called "cherry picking."

Stats: Meta-analysis and diagnostic tests (February 14, 2006). I will be giving a talk at the graduate seminar series for the department of Mathematics and Statistics at UMKC on February 23. The title of the talk will be "Meta-analysis and diagnostic tests."

Stats: A controversial meta-analysis (December 20, 2005). Back in August 2005, the Lancet published an interesting meta-analysis on homeopathy. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Shang A, Huwiler-Muntener K, Nartey L, Juni P, Dorig S, Sterne JA, Pewsner D, Egger M. Lancet 2005: 366(9487); 726-32. The researchers identified 110 placebo controlled homeopathy trials and matched them with 110 placebo controlled conventional-medicine trials. Both sets of trials showed that smaller studies showed stronger effects. Both also showed that lower quality studies showed stronger effects. But when the analysis was restricted to large trials of high quality, the effect of conventional medicine was still statistically significant (odds ratio 0.58, 95% CI 0.39 to 0.85) but the effect of homeopathy was not (odds ratio 0.88, 95% CI 0.65 to 1.19). The critics of this meta-analysis raise some interesting objections, and you can read some of them in the in correspondence section of the December 17, 2005 issue of the Lancet.

Stats: Responding to a critique of meta-analysis (October 10, 2005). A contributor to the Evidence-Based Medicine list offered a possible criticism of meta-analysis. The criticism was along the lines of (I am paraphrasing and summarizing): Suppose we have two randomized trials coming up with exactly the opposite conclusion. Assume that bias and confounding are not an issue here. Then one study may be wrong. When meta-analysis takes an average of a correct value and an incorrect value, you will get a meaningless result. Now assume that the two results differ because they were studying two very different patient populations. An average here is also misleading, unless you weight by the proportion of the true overall population that these two patient populations come from. As an aside this reminds me of the old joke that a statistician is the only person who could stick his head in an oven and his feet in a bucket of ice and say that he feels fine on average. Here's the gist of my response.

Stats: Some articles on meta-analysis (June 10, 2005). I found a couple of interesting articles on meta-analysis that are difficult to classify: Interpreting epidemiological evidence: how meta-analysis and causal inference methods are related. Weed DL. Int J Epidemiol 2000: 29(3); 387-90 and Systematic reviews: a cross-sectional study of location and citation counts. Montori VM, Wilczynski NL, Morgan D, Haynes RB. BMC Med 2003: 1(1); 2. The first relates how meta-analysis supports some, but not all of Hill's criteria for causation. The second compared systematic reviews with narrative reviews. Systematic reviews were cited more often (26 times on average versus 8) and included twice as many citations.

Stats: Hedge's G (May 13, 2005). Someone asked me today about Hedge's G. I had never heard of it before, but if you do a web search, you will find econwpa.wustl.edu/eps/prog/papers/0411/0411124.pdf which defines it as a variation on Cohen's D that corrects for biases due to small sample sizes.

Stats: Cumulative meta-analysis (March 11, 2005). This figure below, published in Erythropoietin, uncertainty principle and cancer related anaemia. Clark O, Adams JR, Bennett CL, Djulbegovic B. BMC Cancer 2002: 2(1); 23 shows cumulative meta-analysis, which is the cumulated effects over time of studies in the use of erythropoietin (EPO) to treat cancer related anemia.

Stats: Summary Receiver Operating Characteristic Curve (January 21, 2005). In the past week, I have had two inquiries about how to perform a meta-analysis of studies of a diagnostic test. An intriguing idea that I discovered in researching this is the use of the Receiver Operating Characteristic (ROC) curve to summarize the results of the studies. Each study will have its own sensitivity and specificity, and plotting the sensitivity/specificity pairs on the coordinates of an ROC plot, along with some reference lines, will help you to evaluate the degree of heterogeneity of the studies, among other things.

Stats: Forest plots (January 12, 2005). Many meta-analyses use a graph known as a forest plot. I was always confused by the funny squares in a forest plot, so I looked for a description.

Stats: Measuring heterogeneity in meta-analysis (November 29, 2004). While browsing through the archives of the British Medical Journal, I noticed an excellent article on measuring heterogeneity in meta-analysis. There is a new measure, I-squared, that measures the proportion of inconsistency in individual studies that cannot be explained by chance. It ranges between 0% and 100% with lower values representing less heterogeneity.

Theme and closely related categories:

• Theme: Evidence based medicine

• Category: Publication bias

Other resources:

• GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. Description: This article presents the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system for rating evidence in a systematic overview or a clinical guideline. The system examines the quality of evidence, uncertainty about the balance between desirable and undesirable effects, uncertainty or variability in values and preferences, and uncertainty about whether the intervention represents a wise use of resources.
• GRADE working group Excerpt: The Grading of Recommendations Assessment, Development and Evaluation (short GRADE) Working Group began in the year 2000 as an informal collaboration of people with an interest in addressing the shortcomings of present grading systems in health care. The working group has developed a common, sensible and transparent approach to grading quality of evidence and strength of recommendations. Many international organizations have provided input into the development of the approach and have started using it.
• How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis. Description: Systematic reviews summarize all the data up to a certain time, so they can become "stale" over time. The authors selected a sample of systematic reviews and noted how often a new review of the same topic presented a change in statistical significance or a large change in the estimated magnitude of the effect. Using classic measures in survival analysis, the authors estimated half of the studies did not see such a shift until 5.5 years, but a significant fraction saw such a signal after one or two years.

[Return to full topic list] [Read current weblog entries]

This webpage was written by Steve Simon on 2007-06-14, edited by Steve Simon, and was last modified on 2008-07-14. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page.