Stats
IRB review of a pilot study (March 26, 2007) Category: Ask
Professor Mean, Category: Ethics in research,
Category: Sample size justification
Dear Professor Mean: I am the new chair of the IRB at a county hospital. Many of the
studies we review are pilot studies with small samples. I have been trying to locate criteria
for the scientific review of pilot studies, but have not found a consensus in the literature
that I have seen. Is a pilot study merely a "dry run" of the procedures that will be used in
a later, larger-scale study? Or, is it reasonable for the IRB to demand that the investigator
provide specific criteria for determining whether the pilot has been a success? And, should
the IRB furthermore demand that specific hypotheses be formulated? My impression is that many
investigators declare their studies to be pilots in order to avoid more rigorous scrutiny of
their proposals.
Professor Mean was thinking that we need more pilot studies. In particular studies of
commercial airline pilots. Pilots who fly regularly to the Caribbean. And it is important for
Professor Mean to be on some of those flights while the data is being collected. Could you
arrange for a nice hotel suite when I arrive?
Short definition--a pilot study is a study that offers no direct benefits of its own, but
only indirect benefits through helping the planning of a larger scale study. Some people
might quibble with that definition, but it is a good starting point for discussion. Sometimes
people use the term "pilot study" when they should be using the term "exploratory study"
instead.
It's a myth that all research must have a formal hypothesis, and this is especially true
of pilot studies. I would not encourage researchers to try to force fit a hypothesis driven
research model onto a type of research where it does not fit well.
If I were on an IRB, I would instead demand details about the larger scale study and why
that larger scale study could not proceed effectively without information from the pilot
study. An economic argument would not fly (we're running a pilot because we don't have enough
money to run a large study), because no large scale study will occur and no benefit will
therefore accrue from the pilot study.
You might want to assess the sincerity of the researchers. Do they (or their colleagues)
really plan to do a trial ten times larger and ten times more expensive than the pilot if the
pilot is successful? I generally dislike assessing things like sincerity because it
presupposes that you are able to look inside the soul of another human being. So I would only
apply the sincerity test if there was major evidence that something fishy was going on.
Another important point to remember is that if the goals of a pilot study are qualitative,
then it is reasonable to use a qualitative justification of the sample size. A common
application of a pilot study is to identify Murphy's Law while the stakes are still low. If
something does go wrong (especially something that would have been easily fixable), find out
before you invest fifty thousand dollars. Typically, you would want a sample size large
enough to insure that the full spectrum of research subjects are included, because problems
are likely to occur at one end of the spectrum or the other but not both. Purposive sampling
or quota sampling can be very helpful here.
Suppose the goal of a pilot study is to establish reliability of a measurement system
prior to using it in a large scale trial. Reliability is typically measured by some type of
correlation, so just show that the confidence interval for this correlation is reasonably
precise--something fairly easy to do, even with a small sample size.
Note this was actually a question posted on the IRBForum by DD, but the question was so
well written that I quoted it directly.
Closely related web pages:
07/08/2008.
