Pilot study (September 3, 1999)

Dear Professor Mean, I am proposing a research study that will examine a complex intervention of diet, exercise, and behavioral modification for some of my pediatric patients who need to lose weight. I want to collect some data from a pilot study before I start the research study. How do I describe the pilot study in my protocol? -- Sophisticated Sarah

Dear Sophisticated,

That reminds me of a cute joke. How many statisticians does it take to light a gas stove? I don't know because we haven't run the pilot study yet.

Short answer

Think of the pilot study as a model of your full research study, but on a smaller scale. Run the pilot study for a briefer time frame and/or on fewer subjects. Focus the pilot study on those aspects of your full study that are novel, untested, complex, or innovative.

Do you remember all the controversy in Florida last year after the presidential election? We could have avoided a few of these problems if someone had run a pilot study on that butterfly ballot in Palm Beach County.

It helps if you can clarify the reasons that you want a pilot study. These reasons can be loosely classified into two categories

• To obtain data to help you plan the full study
• To see where "Murphy's Law" will strike

There are other reasons to run a pilot study. A pilot study helps everyone on your research team get familiar with the procedures in your protocol. A pilot study can also help you decide between two competing approaches (e.g., collecting data in an interview versus using a self-administered survey).

Perhaps the most critical piece of data from a pilot study is the standard deviation of your outcome measure. You cannot select an appropriate sample size for your study without knowing this value.

If your outcome measure is used very commonly, then you may already have an idea what your standard deviation is. Just look at some of the papers that you cited in your literature review. It takes a bit of hunting sometimes, but usually you can find some estimate of variation, such as a standard error, coefficient of variation, or confidence interval. Any of these can be converted into a standard deviation.

Whether you need to estimate the standard deviation in a pilot study depends on the degree of uniqueness and innovation in your experiment. Every experiment is unique, of course, but examine whether the use of this outcome measure has little or no precedent. Also examine how much different your subject population is. An outcome measure that has only been used in adults, for example, may make it more important for you to get pilot data for your pediatric study.

If your outcome measure is the probability of some event, then your sample size depends on how often the event occurs in your control population. You can use a pilot study to estimate this probability, but usually you can get a pretty good estimate of probabilities from previous research.

You should also try to estimate participation rates with your pilot study. How many people do you encounter per month that are eligible for your study? How many agree to participate? How many drop out in the middle of the study?

Finally, information from the pilot will help you estimate resource requirements. How much time do you spend per subject? How much money do you spend per subject? Both pieces of information are critical for preparing the budget for your full study.

Murphy's Law

Murphy's Law says that anything that can go wrong, will go wrong. The reason you run a pilot study is to ensure that the things that do go wrong, go wrong during the pilot study so you can fix them before you start the full study.

It's impossible to list all ways that a study could go wrong, but here are some areas that you should focus on.

Recruitment and retention problems.

• Do you get the types of subjects that you think you will get?
• Are important segments of your population being left out?
• Do a lot of people turn down the opportunity to participate in your study?
• Do a lot of people fail to finish your study?
• Do a lot of people fail to comply with your protocol requirements?

Ambiguous situations.

• Is it obvious who meets and who does not meet the eligibility requirements?
• Do your subjects provide no answer, multiple answers, qualified answers, or unanticipated answers to your survey?

Time and resource problems

• Does it take too long for your subjects to fill out all the survey forms?
• Will the study participants overload your phone lines or overflow your waiting room?
• How much time does it take to mail out a thousand surveys, and can your tongue lick that many stamps in one day?

Machinery problems.

• Is the equipment readily available when and where you need it?
• What happens when it breaks down or gets stolen?
• If the machine produces a stream of electronic data, can your computer software read and understand this data?

Data management problems.

• Is there enough room on the data collection form for all of the data you receive?
• Do you have any problems entering your data into the computer?
• Can you match data that comes in from different sources?
• Were any important data values forgotten about?
• Does your data show too much or too little variability?

Uninformative data

• Are most of your lab results are below the limit of detection?
• Does everybody gives the identical answer to a survey question?

Blind spots and oversights. Something will happen during your pilot study and you'll say "I never thought about that!" Better to have this oversight now than during the full study. Although you can and should show ask your colleagues whether there is there anything you overlooked in your protocol, it's still a good idea to run a pilot study. After all, your colleagues may have the same blind spots that you do.

Other considerations for a pilot study

Don't worry about the representativeness of your pilot subjects, unless you plan to include them in the total sample, or if the sampling procedure itself is complex and innovative. Just make sure that your pilot subjects cover the entire range of subjects in your full study. So if you plan to study this intervention in children ages 6-14, make sure that you have some 6 year olds and some 14 year olds in your pilot study as well as a bunch in between.

Also, don't confuse a pilot study with an exploratory study. An exploratory study will typically try to generate hypotheses for further research. Unlike a pilot study, an exploratory study can stand on its own. Furthermore, you should look for some justification of the sample size in an exploratory study. Since such a study does not have any pre-specified hypotheses, you justify the sample size by showing that some of the estimates produced by the study have reasonable precision.

There is no explicit justification of the sample size for a pilot study. It depends a lot on the complexity of the study. Be sure though, that you aren't just calling their research a pilot study just to get out of having to justify the sample size.

If you are presenting a pilot study to the IRB, I encourage you to cite the type of information that the pilot will provide. Also, please be sure to place the pilot study in the context of the full-blown study. You personally may not be the one who would conduct that full-blown study, but you still need to provide that context.

Summary

A pilot study is a model of your full research study but on a smaller scale. The pilot study helps by providing data needed to plan the larger study and by identifying areas where Murphy's Law will strike.

Further reading

The Lancaster et al publication is an excellent resource. I wrote a brief summary of this article for my weblog. Goodman et al and Omenn et al are nice published examples of pilot studies. Wittes et al is an argument in favor of including pilot data in the full research study.

1. Design and analysis of pilot studies: recommendations for good practice. G. A. Lancaster, S. Dodd, P. R. Williamson. J Eval Clin Pract 2004: 10(2); 307-12. [Medline] [Abstract]
2. The Carotene and Retinol Efficacy Trial (CARET) to Prevent Lung Cancer in High-Risk Populations: Pilot Study with Cigarette Smokers. Goodman G, Omenn G, Thornquist M, Lund B, Metch B and Gylys-Colwell I. Cancer Epidemilogy, Biomarkers & Prevention 1993:2(4);389-396. [Medline]
3. The Carotene and Retinol Efficacy Trial (CARET) to Prevent Lung Cancer in High-Risk Populations: Pilot Study with Asbestos-exposed Workers. Omenn GS. Cancer Epidemiology, Biomarkers & Prevention 1993:2(4);381-387. [Medline]

The role of internal pilot studies in increasing the efficiency of clinical trials. Wittes J and Brittain E. Stat Med 1990:9(1-2);65-71; discussion 71-2. [Medline]

07/14/2008. Category: Ask Professor Mean, Category: Pilot studies