A contributor to the Evidence-Based Medicine list offered a possible criticism of
meta-analysis. The criticism was along the lines of (I am paraphrasing and summarizing):
Suppose we have two randomized trials coming up with exactly the opposite conclusion.
Assume that bias and confounding are not an issue here. Then one study may be wrong.
When meta-analysis takes an average of a correct value and an incorrect value, you will
get a meaningless result.
Now assume that the two results differ because they were studying two very different
patient populations. An average here is also misleading, unless you weight by the
proportion of the true overall population that these two patient populations come from.
As an aside this reminds me of the old joke that a statistician is the only person
who could stick his head in an oven and his feet in a bucket of ice and say that he
feels fine on average.
Here's the gist of my response.
The error that this person describes is the error not of meta-analysis, but rather the
error of a meta-analysis that ignores heterogeneity. Is heterogeneity present in all
meta-analyses and does it result in flawed conclusions in all meta-analyses? Arguing that it
is possible to make an error using meta-analysis naively does not imply that all
meta-analyses are flawed, especially considering that the folks performing meta-analysis have
been aware of the problems caused by heterogeneity and have several proposed solutions to
this problem.
Furthermore, if you have two contradictory RCTs, it may be that the patients that you see
are dissimilar from either of the two trials. Maybe there is a third reality that applies to
your particular patients. So you can't trust the first trial, you can't trust the second
trial, nor can you trust the meta-analysis.
So what can we conclude here? Meta-analysis can be misleading? That's hardly surprising.
All research can be misleading if it is conducted poorly. And all research can be misleading
if you fail to consider how the patients you see are different from the patients being
studied by the researchers themselves.
All of our research tools are flawed. The trick comes from understanding when the flaws
are trivial and when they are fatal.
Furthermore, this looks like a "straw man" argument to me. You describe an application of
meta-analysis that is so simplistic that it does not represent anything even remotely close
to how meta-analysis is practiced today. You cite a "flaw" of meta-analysis that has been
well known for at least a decade or two and for which there has been substantial research
efforts.
If meta-analysis is indeed a flawed process, cite a particular meta-analysis that has lead
to an incorrect conclusion and then explain to us why it has failed. That would lead to a far
more productive debate than some hypothetical speculation.
Finally, to say a research method is flawed is troublesome when you don't offer a serious
alternative. There are some folks who argue that a large scale randomized trial is the
definitive approach and that meta-analysis is only second best. There are others who argue
that a carefully done meta-analysis is superior. Ironically, heterogeneity is one of
meta-analyses strengths, as its proponents have noted. If a research finding is replicated
across a variety of patient populations under a variety of research designs, the
meta-analysis is the best way to show robustness of the finding. In other words, a uniform
finding across heterogenous studies is far more persuasive than a single finding in a single
population.
But rather than offer a serious alternative to meta-analysis, the author of these comments
seems to be saying that almost all research is flawed. That's a very dangerous thing to be
saying, because disregarding good research findings, no matter what the research design that
generated them, can be dangerous to our health.
Rather than asking if meta-analysis is flawed, I would ask "Are we better off or worse off
having the tool called meta-analysis in our research arsenal." And my answer is yes, we're
much better off as long as we keep these tools out of the hands of amateurs.
