A colleague sent me a very nice paper,
Design and analysis of pilot studies: recommendations for good
practice. G. A. Lancaster, S. Dodd, P. R. Williamson. J Eval Clin Pract
2004: 10(2); 307-12.
[Medline]
[Abstract]
that covers some of the same ideas in my web page,
Stats: Designing a pilot study.
This is a very well researched article and has some excellent
recommendations. Right in the abstract, the authors warn against just
slapping on the label of pilot study when your sample size is too small. The
authors also state in the abstract that
A well-conducted pilot study, giving a clear list of aims and
objectives within a formal framework will encourage methodological rigour,
ensure that the work is scientifically valid and publishable, and will lead
to higher quality RCTs.
They mention seven major objectives of a pilot study
- sample size calculation
- integrity of study protocol
- testing of data collection forms or questionnaires
- randomization procedure
- recruitment and consent
- acceptability of intervention
- selection of most appropriate outcome measure
Sample size calculation. Before you can calculate power for a
randomized control trial, you need to know the variability of your outcome
measure if it is continuous, or the proportion(s) you expect to see in the
control population if the outcome measure is categorical. The authors cite
Browne 1995 to support the use of a sample size of at least 30 patients
in this scenario and suggest that the 80% upper one side confidence limit be
used rather than the estimate itself.
Integrity of study protocol. If you are planning a large trial,
especially a trial that will recruit from several different centers, a small
scale run of the full protocol will help you assess the logistics of the
study. The authors explain that the pilot can help you evaluate things like
the storage and testing of equipment and materials.
Testing of data collection forms or questionnaires. I heartily
endorse this use for a pilot study. The authors point out that testing is
especially important for forms that the patients themselves have to complete,
or data collection forms used by several different people. The authors say
you should evaluate comprehensibility and consistency. They also point out
that pilot testing of a survey is not a substitute for assessing validity and
reliability for that survey.
Randomization procedure. This is a bit surprising, and I would have
lumped it in with integrity of the study protocol. I do have to admit,
though, that randomization has become more complicated than just the simple
flipping of a coin. The need to conceal the randomization list from those who
recruit patients into the study leads to the use of sealed envelopes and 24
hour randomization hot lines.
Recruitment and consent. It is often harder to recruit subjects into
a research study than you think. The authors point out that a pilot study
will give you a good estimate of how long it will take to recruit the
subjects you need for your research. They cite
Ross 1999, which indicates that slower than expected recruitment of
subjects is one of the most common reasons why studies are abandoned early.
Acceptability of intervention. The intervention you have planned for
your research subjects might not be palatable. The authors point out that
this may especially be a problem with drugs given to children, who often
cannot or will not tolerate medications that adults take with no difficulty.
Selection of most appropriate outcome measure. This is also a bit
surprising. The authors claim that when you need to choose between two or
three different outcome measures, a pilot study can help you make that
choice. I suspect, however, that most if not all of the considerations for
selecting an outcome measure would come from external information and not
from a pilot study.
The authors then warn against including pilot data in the full study. I
would tend to disagree. It's hard enough to get good data, why toss away the
extra data that you get in a pilot study. You do need to be careful about
this, especially if you sharply curtail the planned study based on findings
in the pilot study.
This paper is a very good effort to outline the scientific rationale for
pilot studies and should provide helpful guidance for committees that have to
review these types of research.
