Slipped deadlines and sample size shortfalls in a random sample of research studies (May 7, 2008).

There is a limited amount of data out there that suggests that many researchers overpromise on the planned sample size and completion date and underdeliver. For example,

I attended a web seminar about optimizing clinical trial enrolment, and they offered some additional evidence of the problem

About a year ago, I received a small grant to study the proportion of studies at Children's Mercy Hospital (CMH) that failed to meet the proposed completion deadlines, that failed to recruit the promised number of patients or both. Here is a brief summary of these results.

Records were reviewed for a stratified random sample of 130 IRB approved studies requiring full review which produced final reports between January 2001 and December 2005. Studies requiring full review were to a large extent studies that were prospective. Also included in the sample were 9 studies funded by an internal grant mechanism, which had to be prospective human studies. Studies were stratified by year of completion with 25 to 27 studies per year. Some years had different numbers of studies because of the additional internally funded studies. We excluded any studies from the Children's Oncology Group (COG). Childhood cancer is a thankfully rare disease, so the typical COG protocol would enroll only 1 or 2 patients at each site. We also excluded any retrospective studies or any studies not involving humans. If the study were part of a multi-center trial, we calculated the planned and actual sample sizes from CMH only.

For each study, the final report, any interim reports, and the original protocol submission were reviewed. The variables recorded were

  1. planned study start date,
  2. planned study end date,
  3. actual study start date,
  4. actual study end date,
  5. planned sample size,
  6. actual sample size.

If certain dates or sample sizes could be determined from the study, a code of U was entered. The primary goal of this research was to estimate

  1. the proportion of studies which lasted longer than planned,
  2. the proportion of studies which recruited fewer subjects than planned,
  3. the average size of these deviations.

A secondary objective was to measure the proportion of times that information about planned start and end dates were not included in the file. Finally, we wanted to see if certain features of the studies

  1. External sponsor (yes / no),
  2. Study coordinator (yes / no),
  3. Consent required (yes / no), and
  4. Randomization used (yes / no)

were associated with any of these measures.

A total of 117 studies (90%) failed to include a planned start date (n=2) a planned completion date (n=25) or both (n=90). The CMH IRB does not require applicants to include either date in their protocol submission, which is a major failing. For these 117 studies, the IRB was essentially signing a blank check and implied that approval was not contingent on the timely conduct of the study.

For the 13 studies with both planned start and end dates, there were two with missing information about actual dates, one missing actual start date and one missing actual end date. This leaves 11 studies where a comparison between the actual and planned duration. The mean planned duration was 18 months (range 4.6 to 45 months). In eight of these studies (73%), the actual duration was much longer than the planned duration. The average relative increase in duration among these 8 studies was 100% (range 6% to 286%). The most extreme case was a study that was planned to last 276 days (9.2 months) and actually lasted 1064 days (35.5 months). In the three studies that finished early, the average relative decrease in time was 26% (range 12% to 33%). If all 11 studies combined there would be an average increase of 65% (range -33% to 286%).

All of the studies except two included a planned sample size and two studies used ambiguous sample sizes (up to 40, up to 5), and in 17 studies (13%), we could not determine the actual number of subjects enrolled in the study. This left 109 studies where we could compare the planned and actual sample sizes. The mean planned sample size was 49.5 (range 1 to 830). There were 59 studies (54%) where the actual sample size was less than the planned sample size. The average shortfall in these studies was 55%. There were 8 studies where no patients were recruited. Six of these studies had planned sample sizes of five or less, but the other two had planned sample sizes of 25 and 50. The average shortfall among the 59 studies was 55% (range 2% to 100%). There were 50 studies that met or exceeded the planned sample size. 24 met the target exactly. Among the 26 studies that exceeded the sample size the average increase was 48% (range 6% to 433%) the largest change was a study that planned to recruit 3 patients, but instead recruited 16. When all studies were combined, the average discrepancy was -18% (range -100% to 433%).

Conclusions: The current IRB reporting mechanisms at CMH do not require researchers to report the planned duration of their research trials, nor do they require researchers to comment on any delays in their trials. I suspect that this is similar for many other IRBs. This represents a serious failing on the part of the IRBs to monitor the progress of these trials. While small delays are tolerable, the scientific validity of a trial comes into question if the proposed time frame or the actual time frame of the proposed research extends beyond a reasonable limit. Research delayed is research denied.

More than half of the IRB approved studies failed to enroll the promised number of patients and the average shortfall in these studies was 55%. Again, this failure to enroll an adequate number of patients calls into question the scientific validity of the research.

This webpage was written by Steve Simon and was last modified on 2008-07-14. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page. Category: Accrual problems in clinical trials