What's Wrong with Medical Research?

This is a draft version of a speech I gave at the May 4, 2000 meeting of Bluejacket Toastmasters.

Introduction

"I can't believe all the junk that gets published in top notch medical journals!" That's a comment I hear all the time. I work with some prominent researchers at Children's Mercy Hospital and they don't trust a lot of the stuff that is published in the Journal of the American Medical Society, Lancet, the New England Journal of Medicine, and other prominent medical journals.

That's just an opinion, of course. As a statistician, I always trust cold hard data to opinions. In this case, though, the cold hard data backs up what my colleagues are saying.

Thornley and Adams Overview

One piece of data about medical research is an article by Ben Thornley and Clive Adams that appeared in the British Medical Journal in October of 1998. The title of this article is "Content and Quality of 2000 Controlled Trials in Schizophrenia over 50 Years." Thornley and Adams work for the Cochrane Collaboration Group, a group that provides systematic reviews of the results of medical trials, so they are in a good position to write such an article.

Thornley and Adams actually found over 2,500 studies, but decided to summarize only the first 2,000 they uncovered. Only the first 2,000. I still am very impressed at the amount of work this must have taken.

The research covered fifty years, starting in 1948 through 1997. The research covered a variety of therapies: drug therapies, psychotherapy, policy or care packages, or physical interventions like electroconvulsive therapy.

What did Thornley and Adams find? It wasn't a pretty picture. For the most part, researchers in schizophrenia studied the wrong patients, they didn't study enough of them, they didn't study their patients long enough, and they didn't measure them properly. Let's look at each of these assertions in turn.

They studied the wrong patients.

First, the researchers studied the wrong patients. Only 14% of the studies of schizophrenia are community based, but that's where most patients are treated. Community based studies are hard to design, because you have so much less control than you would with a pool of hospitalized or institutionalized patients. But research in a hospital does not extrapolate well to other settings. It's like the story of a man who lost his wallet at night in a dark alley, but was searching for it out in the street underneath a street lamp. Why was he searching in the street, you might ask? Because the light is better out here.

They didn't study enough patients.

Second, the researchers didn't study enough patients. The precision of the research studies depends on the number of patients studied. Thornley and Adams showed that a typical study of schizophrenia would require a total of 200 patients to achieve adequate precision. What was the average number of patients studied in these 2,000 trials? 65. The researchers needed 300 subjects, but on average got only 65. Only 3% of the researchers met Thornley and Adams' goal of 300 patients.

They did not study these patients long enough.

Third, the researchers did not study these patients long enough. According to Thornley and Adams, a good research trial should last at least six months. More than half of the trials lasted six weeks or less. Clearly, it costs less to run a short trial than a long trial.  But short term improvements in any medical study are much less important than long term changes.

They did not measure their patients properly.

Finally, the researchers did not measure their patients properly. Measuring improvement in a schizophrenic patient is indeed difficult, but the researchers blew it badly here. In 2,000 trials, the researchers developed and used 640 measures of these patients. This shows a serious lack of standards in schizophrenia research. The use of 640 different measurements shows there is no consensus on how to measure the severity of schizophrenia or how much a patient might have improved under a certain therapy. This makes it very difficult to compare results across different studies. You might say that the research about schizophrenia is schizophrenic.

Conclusion

The problems that Thornley and Adams raise are not limited to schizophrenia or even psychiatry. There is plenty of objective evidence in many other medical and scientific areas that the reports published in research journals have many of the same problems.

It's hard to do good medical research, especially in an area like psychiatry. But as Thornley and Adams have shown, we could be doing a lot better.

1. We study the wrong patients. We grab the easy to get hospitalized and institutionalized patients when we should be doing more studies in the community.
2. We don't study enough patients. A good study would require 300 patients, but the average study only had 65.
3. We don't study patients long enough. A good study would require six months of monitoring, but more than half of the studies lasted only six weeks.
4. And we don't measure things well. There were 640 different ways of assessing how the patients in these studies were doing.

So the next time you hear news reports about the latest research findings, be sure to be a little bit skeptical. Just because it appears in the New England Journal of Medicine doesn't mean that it is true.

This webpage was written by Steve Simon on 2000-05-04, edited by Steve Simon and Linda Foland, and was last modified on 2008-07-08. This page needs minor revisions. Category: Statistical evidence