I found a quote on the Skeptic's Dictionary web site that
is worth commenting on. The author, Robert Todd Carroll was describing the Vioxx controversy
and the lawyers who are now aggressively recruiting people for a lawsuit against Merck, the
manufacturer of Vioxx. In it, he repeats a common misconception about relative risk (RR):
According to mathematician John Brignell, "In observational studies, [scientists]
will not normally accept an RR of less than 3 as significant and never an RR of less
than 2." --
http://skepdic.com/refuge/funk43.html
This is a widely quoted rule, but it vastly oversimplifies the issue. A small RR is a
weaker form of evidence, because it is possible to be swamped by even small biases and flaws
in the research study. A large RR is a stronger form of evidence, because only a huge bias or
flaw could produce an alternate explanation for these findings.
Weak evidence, however, is not the same thing as no evidence. Weak evidence that is
replicated and backed up by a credible biological mechanism can become sufficiently
persuasive.
The original source of the rule cited above is a paper by Austin Bradford Hill published
in 1965
[Medline]. It listed nine criteria that you should consider when evaluating a research
finding from observational data. It was developed during a time when Epidemiologists had just
successfully built a case linking smoking and lung cancer on the foundation of numerous
research studies, none of which was perfect. The lessons learned in that situation became
guidelines for applying Epidemiologic principles to other toxic exposures.
The first criterion that Hill cited, strength of association, has unfortunately morphed
into a dichotomy that all RRs smaller than 2 are bad and all RRs larger than 2 are good, in
spite of numerous cautions, both by Hill himself in that article and by many Epidemiologists
since then.
In the classic textbook, Modern Epidemiology (Second Edition), Rothmann and Greenland
[BookFinder4U link] point
out (page 24) that the relationship of cigarette smoking and cardiovascular disease is a weak
association, because it is pretty hard to double the risk of a disease that is already
responsible for about
40% of all deaths, according to the American Heart Association. Nevertheless, pretty much
everyone accepts the link between smoking and cardiovascular disease. If you relied on John
Brignell's advice about relative risks less than 2 or 3, then your standard of proof would
require that 80% to 120% of all smokers would have to die from cardiovascular disease.
Nothing, not even cigarettes, can produce 120% mortality, and even 80% mortality from a
single cause is probably out of reach.
A good recent commentary on this issue is "The missed lessons of Sir Austin Bradford Hill"
by Carl Phillips and Karne Goodman.
That's my major quibble. A minor quibble is that this rule is usually restricted, even by
John Brignell, to observational studies. But the study that showed an increase in risk for
Vioxx was a randomized study.
This doesn't mean that the rest of Dr. Carroll's comments are incorrect. It is indeed
possible that we are all overreacting to weak data. I don't think the final chapter has been
written yet on Vioxx and other drugs in its class.
A chapter of the book I am writing, Statistical Evidence,
talks about Corroborating Evidence. This chapter mentions
strength of association, among other factors that can influence the quality of evidence in
research. I also have a weblog entry on side effects
which mentions Vioxx and other COX-2 inhibitors.
I sent an email along the same lines as this weblog entry to Dr. Carroll and he published
it on his web site at
http://www.skepdic.com/refuge/funk43.html#comment
References
