An email on the MedStats group outlines a new treatment that is:
1. without any significant competing treatments,
2. utilized in a heterogenous patient population,
3. difficult to study in a randomized trial.
There are a variety of alternatives to a randomized study, but I suspect that this person
wants to use a historical control study. It sounds like he wants an informal endorsement from
a group of professional statisticians to use a historical control study instead of a
randomized study. Professor Mean normally charges serious money for an endorsement, but he is
in a generous mood and will offer an endorsement for free.
I wrote a section in my book with the title "Randomization is overrated" and one of the
reviewers (bless his soul) gave me a lot of grief over this. I still believe what I wrote.
Although I would not agree with the sentiment (to quote from the original email) that
randomized trials "are often short-sighted and uninteresting and focused at simple instead of
interesting questions." I would say that randomized trials are often performed in an
artificial environment which does not reflect patient care as it is practiced in the real
world.
You should recognize, however, that historical control studies are not well accepted in
the research community and the effort you expend on this may not pay off in terms of
convincing others to adopt this new therapy that you seem to like. The only time that
historical controls are found to be acceptable is when:
a) there is no competing therapy available; and
b) the disease causes close to 100% mortality or morbidity.
Some have added a third condition
c) the disease is well characterized in a highly homogenous population.
This latter condition would exclude any disease defined as a "syndrome" (a disease that is
defined by a constellation of symptoms rather than a positive diagnostic test and/or a
disease that defined as effects that cannot be explained by any other cause).
If patients for the most part refuse to agree to randomization, then you don't need
anyone's endorsement for using a non-randomized approach. I make a bad joke about this. Most
patients will refuse to be part of randomized trial that involves birth control methods,
because they already have strong opinions about the types of birth control that they find
acceptable or unacceptable. And they really get upset when they find out that one of the arms
in the study is a placebo birth control pill.
If randomization is truly impossible, then I would still encourage you to consider
alternatives to a historical control group. There are more rigorous designs that don't
require randomization at the patient level.
If randomization is possible, but it just takes too much time and effort, then you need to
make a value judgment. The reversion to a weaker form of evidence has a cost associated with
it. Randomization has a cost associated with it. Which cost is greater? In most situations
(but not all situations), I believe that the cost of reversion to a weaker form of evidence
is greater. In other words, anytime you can randomize, you should randomize.
The question is not, what evidence do you need to justify what you are currently doing,
but rather what evidence do you need to convince a skeptical colleague (not a cynical
colleague, of course, as cynics can never be persuaded). What you will find is that most of
your skeptical colleagues will not change their practice on the basis of a single historical
control study. So why are you doing research that is essentially unpersuasive?
07/08/2008.
