Stats #77: Bayesian tools for planning and monitoring accrual rates in clinical trials

Stats #77: Bayesian tools for planning and monitoring accrual rates in clinical trials

Content: This training class will discuss some Bayesian models for accrual (how rapidly patients volunteers for your research study).

Teaching strategies: Didactic lectures and small group exercises.

Abstract: Too many researchers overpromise and undeliver on the planned sample size and the planned completion date of their research. This leads to serious delays in the research and inadequate precision and power when the research is completed. Researchers need tools that will let them plan the pattern of patient accrual in their studies. These tools will also let the researchers carefully monitor the progress of their studies and let them take action quickly if accrual rates are suffering.

Objectives: In this seminar, you will learn how to:

define problems associated with slipped deadlines and sample size shortfalls,
describe a simple Bayesian model for patient accrual, and
use this model to predict the planned duration of the clinical trial.

Notes: There are no pre-requisites for this seminar. This class does not qualify for IRB Education Credits (IRBECs).

Contents:

Where can you find this handout?

Slipped deadlines and sample size shortfalls in a random sample of research studies

Why does a Bayesian approach make sense for monitoring accrual?

A simple Bayesian model for exponential accrual times

Case study of accrual in a clinical trial

Monitoring refusals and exclusions in a clinical trial

Where can you find this handout?

This handout and the handouts that I use for all of my seminars and training classes are a compilation of individual web pages at www.childrensmercy.org/stats. I use the "Include Page" feature of Microsoft FrontPage to combine these into a single page. You can always find the most recent version of this compilation by going to the web address listed at the bottom of this page. Links for the handouts for other seminars and classes appear at www.childrensmercy.org/stats/training.asp.

Why don't I use PowerPoint?

I stopped using PowerPoint for my presentations in the mid 1990's. This was based on Edward Tufte's advice that presenting information in a paper handout is more effective than presenting the information on a projected screen. I found this to be excellent guidance. I enjoy talking when I don't have to wrestle with a laptop computer. I look at my audience more and interact with them better. I elaborate on this in greater detail at www.childrensmercy.org/stats/weblog2004/powerpoint.asp.

Slipped deadlines and sample size shortfalls in a random sample of research studies (May 7, 2008).

There is a limited amount of data out there that suggests that many researchers overpromise on the planned sample size and completion date and underdeliver. For example,

"Globally, more than 80 per cent of clinical trials fail to enrol on time, and this recruitment problem is extremely costly for drug companies, contributing to 85-95 per cent of the lost days in a clinical trial." www.in-pharmatechnologist.com/news/ng.asp?n=68150-chiltern-india-cost-clinical-trial-regulatory.

I attended a web seminar about optimizing clinical trial enrolment, and they offered some additional evidence of the problem

More than 90% of clinical trial sites delay enrollment -- CenterWatch

72% of studies are delayed by more than a month -- CISCRP

Out-of-pocket costs are $1.2M on average per study for each month delayed -- Top 10 Pharma Company report

Delays in studies can cost life-sciences companies at least $800,000 a day in lost sales for a niche medication and $5.4 million a day for a blockbuster -- McKinsey

About a year ago, I received a small grant to study the proportion of studies at Children's Mercy Hospital (CMH) that failed to meet the proposed completion deadlines, that failed to recruit the promised number of patients or both. Here is a brief summary of these results.

Records were reviewed for a stratified random sample of 130 IRB approved studies requiring full review which produced final reports between January 2001 and December 2005. Studies requiring full review were to a large extent studies that were prospective. Also included in the sample were 9 studies funded by an internal grant mechanism, which had to be prospective human studies. Studies were stratified by year of completion with 25 to 27 studies per year. Some years had different numbers of studies because of the additional internally funded studies. We excluded any studies from the Children's Oncology Group (COG). Childhood cancer is a thankfully rare disease, so the typical COG protocol would enroll only 1 or 2 patients at each site. We also excluded any retrospective studies or any studies not involving humans. If the study were part of a multi-center trial, we calculated the planned and actual sample sizes from CMH only.

For each study, the final report, any interim reports, and the original protocol submission were reviewed. The variables recorded were

planned study start date,

planned study end date,

actual study start date,

actual study end date,

planned sample size,

actual sample size.

If certain dates or sample sizes could be determined from the study, a code of U was entered. The primary goal of this research was to estimate

the proportion of studies which lasted longer than planned,

the proportion of studies which recruited fewer subjects than planned,

the average size of these deviations.

A secondary objective was to measure the proportion of times that information about planned start and end dates were not included in the file. Finally, we wanted to see if certain features of the studies

External sponsor (yes / no),

Study coordinator (yes / no),

Consent required (yes / no), and

Randomization used (yes / no)

were associated with any of these measures.

A total of 117 studies (90%) failed to include a planned start date (n=2) a planned completion date (n=25) or both (n=90). The CMH IRB does not require applicants to include either date in their protocol submission, which is a major failing. For these 117 studies, the IRB was essentially signing a blank check and implied that approval was not contingent on the timely conduct of the study.

For the 13 studies with both planned start and end dates, there were two with missing information about actual dates, one missing actual start date and one missing actual end date. This leaves 11 studies where a comparison between the actual and planned duration. The mean planned duration was 18 months (range 4.6 to 45 months). In eight of these studies (73%), the actual duration was much longer than the planned duration. The average relative increase in duration among these 8 studies was 100% (range 6% to 286%). The most extreme case was a study that was planned to last 276 days (9.2 months) and actually lasted 1064 days (35.5 months). In the three studies that finished early, the average relative decrease in time was 26% (range 12% to 33%). If all 11 studies combined there would be an average increase of 65% (range -33% to 286%).

All of the studies except two included a planned sample size and two studies used ambiguous sample sizes (up to 40, up to 5), and in 17 studies (13%), we could not determine the actual number of subjects enrolled in the study. This left 109 studies where we could compare the planned and actual sample sizes. The mean planned sample size was 49.5 (range 1 to 830). There were 59 studies (54%) where the actual sample size was less than the planned sample size. The average shortfall in these studies was 55%. There were 8 studies where no patients were recruited. Six of these studies had planned sample sizes of five or less, but the other two had planned sample sizes of 25 and 50. The average shortfall among the 59 studies was 55% (range 2% to 100%). There were 50 studies that met or exceeded the planned sample size. 24 met the target exactly. Among the 26 studies that exceeded the sample size the average increase was 48% (range 6% to 433%) the largest change was a study that planned to recruit 3 patients, but instead recruited 16. When all studies were combined, the average discrepancy was -18% (range -100% to 433%).

Conclusions: The current IRB reporting mechanisms at CMH do not require researchers to report the planned duration of their research trials, nor do they require researchers to comment on any delays in their trials. I suspect that this is similar for many other IRBs. This represents a serious failing on the part of the IRBs to monitor the progress of these trials. While small delays are tolerable, the scientific validity of a trial comes into question if the proposed time frame or the actual time frame of the proposed research extends beyond a reasonable limit. Research delayed is research denied.

More than half of the IRB approved studies failed to enroll the promised number of patients and the average shortfall in these studies was 55%. Again, this failure to enroll an adequate number of patients calls into question the scientific validity of the research.

This webpage was written by Steve Simon and was last modified on 2008-07-14. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page. Category: Accrual problems in clinical trials

Why does a Bayesian approach make sense for monitoring accrual? (May 8, 2008).

I'm working with Byron Gajewski to develop some models for monitoring the progress of clinical trials. Too many researchers overpromise and undeliver on the planned sample size and the planned completion date of their research. This leads to serious delays in the research and inadequate precision and power when the research is completed. We want to develop some tools that will let researchers plan the pattern of patient accrual in their studies. These tools will also let the researchers carefully monitor the progress of their studies and let them take action quickly if accrual rates are suffering.

We've adopted a Bayesian approach for these tools. While a Bayesian approach to Statistics is controversial, we feel that there should be no controversy with regard to using Bayesian models in modeling accrual.

There are lots of humorous quips about Bayesian statistics. One of my favorites is from Stephen Senn.

"Bayesian: One who, vaguely expecting a horse and catching a glimpse of a donkey, strongly concludes that he has seen a mule." Statistical Issues in Drug Development, 2nd edition, Senn SJ (2007) page 46.

This quip alludes to the classic lay description of Bayesian Statistics. The Bayesian asks a researcher to summarize their state of belief about a statistical model prior to the collection of data. This produces probability distributions (prior distributions) for various parameters in the statistical model. After the data is collected, the Bayesian statistician will combine that data with the prior distribution to produce a posterior distribution and calculate expected values of the posterior distribution (along with other quantities from the posterior distribution) to draw conclusions from the data. Frequently, the expected values from the posterior distribution represent a weighted average of the data and of the prior beliefs. If the prior beliefs are strong relative to the data, more weight is placed on the prior distribution. If the sample size of the data is large relative to the degree of certainty provided by the prior distribution, then more weight is placed on the data.

There is a belief among some critics of Bayesian Statistics that the prior distribution allows subjective beliefs to be incorporated into an otherwise objective analysis. A true scientist should be disinterested in the results of the research so as to maintain the credibility of the research findings. There are several counterarguments to this criticism, of course, and others can make these counter-arguments better than I can.

From the perspective of accrual, however, there should be no debate. A researcher would never undertake a clinical trial unless he/she had a least an inkling of how quickly patients would volunteer for the trial. Soliciting such beliefs does no harm to the supposed objectivity of the final data analysis. In fact, you can use a Bayesian model for accrual for a clinical trial where all of the proposed data summaries are classical non-Bayeisan.

The big advantage to specifying a prior distribution is that when a researcher has extensive experience in given research arena and provides appropriately precise prior distributions, that will prevent the researcher from overreacting to a bit of early bad news about accrual. In contrast when a researcher provides only a vague prior distribution about accrual patterns, early evidence of problems is given greater weight, allowing for rapid interventions to correct the slow accrual.

A simple Bayesian model for exponential accrual times (May 26, 2008).

Here is a simple Bayesian model for exponential accrual times. This model will help researchers to plan the estimated duration of a clinical trial. The same model will also allow the researcher to monitor the accrual during the trial itself and develop revised estimates for the duration or the sample size.

This web page represents a direct translation of a poster presentation, Predicting Accrual in Clinical Trials: Bayesian Posterior Predictive Distribution, Stephen D. Simon and Byron Gajewski, presented at the 2007 Joint Statistical Meetings in Salt Lake City, Utah. Here is a PDF version of the original poster. These results were also adapted for a paper

Gajewski BJ, Simon SD, Carlson S. "Predicting accrual in clinical trials with Bayesian posterior predictive distributions" Statistics in Medicine (2008): 27(13); 2328-2340. DOI: 10.1002/sim.3128 [Medline] [Abstract].

A similar accrual model based on the Poisson distribution was developed independently and published a few months earlier in the same journal

Anisimov VV, Fedorov VV. "Modelling, prediction and adaptive adjustment of recruitment in multicentre trials" Statistics in Medicine (2007); 26: 4958�4975. DOI: 10.1002/sim.2956 [Medline] [Abstract].

Here is the notation that will be used in the Bayesian model for exponential accrual times.

Suppose that the trial director wishes to assess the accrual process after m patients have been recruited. Let t₀ represent the time that the study started and let t₁, t₂, �, t_m represent the times that each new patient enters the trial. Without loss of generality, we can assume that the study starts at time t₀=0. Compute waiting times

We will assume that

and specify an inverse gamma as the prior distribution for theta. Eliciting a prior distribution is a difficult task. We start our elicitation by asking two simple questions

How long will it take to accrue n subjects?

On a scale of 1-10, how confident are you in your answer to 1?

Let T represent the answer to the first question. Divide the answer to the second question by 10 to get P. This produces the following prior distribution.

You can then present various properties of this prior distribution to the researcher to assess how realistic this prior distribution is.

After the trial has started, assume that m patients have entered the trial at time t_m. The posterior distribution is

which has posterior mean

This is approximately a weighted average of the prior mean and the mean waiting time for the observed data. The relative weights depend on the prior sample size (nP) and the sample size of patients observed so far (m).

We are interested in predicting the next n-m waiting times W_m+1,�,W_n. A simple and easily generalizable approach is to randomly select θ₁ from the posterior distribution and then randomly select waiting time n-m random variables from W_m+1,1 ,�,W_n,1 from an exponential distribution with parameter θ₁. Repeat this process for θ₂, θ₃,�,θ_b, where b is a large number (typically 1,000). The sum of observed and simulated waiting times, S_b(n) = w₁ + w₂ + � + w_m + W_m+1,b + � + W_n,b will represent b estimates of the total duration of the clinical trial.

Example: To illustrate the proposed method, consider an unnamed current phase III clinical trial (randomized, double-blind, and placebo-controlled) used to examine the efficacy and safety of a dietary supplement. This study was planned and accrual started prior to our development of these methods, but still serves to illustrate how this approach would work. The current protocol requires n=350 subjects, with balanced randomization to either treatment or placebo control. In the previous study, investigators were able to recruit, from a similar population, 350 subjects across 3 years.

At the design phase of the study it was felt that the previous clinical trial offered strong prior information. Setting P=0.5 results in k=175 and V=1.5 years. This corresponds to a prior mean of V/(k-1)=0.0086 years (3.1 days). This means that the researcher expects to see a new patient every third day, on average.

With this prior distribution, the researcher can predict the trial duration and account for uncertainty in the specification of the parameters and uncertainty due to the random nature of the exponential accrual model.

The graph above shows the predicted trial duration based solely on the prior distribution. The gray region represents the range between the 2.5 and 97.5 percentiles. The white line in the middle of this region represents the 50th percentile.

For this particular prior distribution, the 50th percentile for trial duration is 3.0 years. There is a 0.025 probability that the trial could finish in 2.5 years or less and a 0.025 probability that it could last 3.6 years or longer.

After the study was funded and the protocol approved, the investigative team began recruiting subjects. After 239 days the project director compiled a report that displays enrolled dates of 41 subjects. This represents an average waiting time of 5.8 days, much longer than prior expectation (3.1 days).

The Bayesian predictive distribution appears above. Notice that the estimated completion time is not a simple linear projection of the data, as the prior distribution still exerts enough influence to bend the projection back slightly. The 50th percentile for trial duration is 3.7 years, a substantial increase over the original belief that the trial would last about 3 years. The 2.5 and 97.5 percentiles for the Bayesian predictive distribution are 3.3 and 4.3. so even allowing for the uncertainties in the accrual pattern, there is a very high probability that this trial will finish later than planned.

We can produce a Bayesian predictive distribution for a non-informative prior by setting P=0. The results are shown above. Note that the median trial duration (5.6 years) is now a simple linear projection of the accrual trend through the first 41 subjects. This approach, however, properly accounts for the uncertainty associated with this trend. The 2.5 and 97.5 percentiles are 4.2 years and 7.5 years respectively.

The R code for producing the Bayesian predictive distribution for the trial duration is shown below. this simulation could easily be done in most other reasonable statistical software packages or even in a spreadsheet.

duration.plot <- function(n,T,P,m,tm,B=1000,Tmax=2*T,sample.paths=0) { # n is the target sample size # T is the target completion time # P is the prior certainty (0 <= P <= 1) # m is the sample observed to date # tm is the time to date # B is the number of simulated duration times # Tmax is the upper limit on the time axis of the graph # # set P to zero for a non-informative prior # set m and tm to zero if no data has been accumulated yet. k <- n*P+m V <- T*P+tm theta <- 1/rgamma(B,shape=k,rate=V) simulated.duration <- matrix(NA,nrow=n-m,ncol=B) for (i in 1:B) { wait <- rexp(n-m,1/theta[i]) simulated.duration[,i] <- tm+cumsum(wait) } lcl <- apply(simulated.duration,1,quantile,probs=0.025) mid <- apply(simulated.duration,1,quantile,probs=0.500) ucl <- apply(simulated.duration,1,quantile,probs=0.975) layout(matrix(c(1,2,2,2))) par(mar=c(0.1,4.1,0.1,0.1)) duration.hist <- cut(simulated.duration[n-m,], seq(0,Tmax,length=40)) barplot(table(duration.hist),horiz=FALSE, axes=FALSE,xlab=" ",ylab=" ",space=0, col="white",names.arg=rep(" ",39)) par(mar=c(4.1,4.1,0.1,0.1)) plot(c(0,Tmax),c(0,n),xlab="Time", ylab="Number of patients",type="n") polygon(c(lcl,rev(ucl)),c((m+1):n,n:(m+1)), density=-1,col="gray",border=NA) lines(mid,(m+1):n,col="white") segments(0,0,tm,m) while (sample.paths>0) { lines(simulated.duration[,sample.paths],(m+1):n) sample.paths <- sample.paths-1 } pctiles <- matrix(NA,nrow=2,ncol=3) dimnames(pctiles) <- list(c("Waiting time","Trial duration") ,c("2.5%","50%","97.5%")) pctiles[1,] <- 1/qgamma(c(0.975,0.5,0.025),shape=k,rate=V) pctiles[2,1] <- lcl[n-m] pctiles[2,2] <- mid[n-m] pctiles[2,3] <- ucl[n-m] list(pct=pctiles,sd=simulated.duration) }

The function calls that produced the three figures shown above are

p0 <- duration.plot(n=350,T=3,P=0.5,m= 0,tm= 0,Tmax=10) p1 <- duration.plot(n=350,T=3,P=0.5,m=41,tm=239/365,Tmax=10) p2 <- duration.plot(n=350,T=3,P=0, m=41,tm=239/365,Tmax=10)

Predicting sample size if the trial has a fixed duration. We can also use this process to obtain a posterior predictive sample size. Let T represent the time point at which the study must end. Compute partial sums S_b(m+1), S_b(m+2), . . until the partial sum exceeds T. The values n_b which represent the largest values where the partial sums do not exceed T, provide a predictive distribution of sample sizes.

Example: Let's use the same prior distribution, but produce a simulation that estimates the final sample size under the assumption that the trial must end at exactly 3 years.

The figure above shows the Bayesian predictive distribution based only on the prior distribution. The median estimated sample size is 349 and the 2.5 and 9.5 percentiles are 291 and 413.

The figure above shows how the projected sample size changes after 239 days and 41 patients. The median is much lower now (274), and the 2.5 and 97.5 percentiles (233 and 321 respectively) are both well below the original expectation of recruiting 350 patients.

A Bayesian predictive sample size ignoring the prior information is shown above. The median sample size is 188, which is a simple linear projection of the data. The 2.5 and 97.5 percentiles are 143 and 241, respectively.

This simulation approach also allows you to examine more complex accrual patterns, such an accrual goal of recruiting until 50 patients have volunteered, or until 6 months have elapsed, whichever comes first.

The R code for producing the Bayesian predictive sample size appears below.

accrual.plot <- function(n,T,P,m,tm,B=1000,nmax=2*n,sample.paths=0) { # n is the target sample size # T is the target completion time # P is the prior certainty (0 <= P <= 1) # m is the sample observed to date # tm is the time to date # B is the number of simulated accrual times # nmax is the upper limit on the sample size axis of the graph # # set P to zero for a non-informative prior # set m and tm to zero if no data has been accumulated yet. k <- n*P+m V <- T*P+tm theta <- 1/rgamma(B,shape=k,rate=V) simulated.duration <- matrix(NA,nrow=nmax-m,ncol=B) for (i in 1:B) { wait <- rexp(nmax-m,1/theta[i]) simulated.duration[,i] <- tm+cumsum(wait) } tlist <- seq(tm,T,length=100) accrual.count <- function(x,t) {m+sum(x<=t)} simulated.accrual <- matrix(NA,nrow=100,ncol=B) for (i in 1:100) { time <- tlist[i] simulated.accrual[i,] <- apply(simulated.duration,2, accrual.count,t=time) } lcl <- apply(simulated.accrual,1,quantile,probs=0.025) mid <- apply(simulated.accrual,1,quantile,probs=0.500) ucl <- apply(simulated.accrual,1,quantile,probs=0.975) layout(matrix(c(2,2,2,1),nrow=1)) par(mar=c(4.1,0.1,0.1,0.1)) accrual.hist <- cut(simulated.accrual[100,], seq(0,nmax,length=40)) barplot(table(accrual.hist),horiz=TRUE, axes=FALSE,xlab=" ",ylab=" ",space=0, col="white",names.arg=rep(" ",39)) par(mar=c(4.1,4.1,0.1,0.1)) plot(c(0,T),c(0,nmax),xlab="Time", ylab="Number of patients",type="n") polygon(c(tlist,rev(tlist)),c(lcl,rev(ucl)), density=-1,col="gray",border=NA) lines(tlist,mid,col="white") segments(0,0,tm,m) while (sample.paths>0) { lines(tlist,simulated.accrual[,sample.paths]) sample.paths <- sample.paths-1 } pctiles <- matrix(NA,nrow=2,ncol=3) dimnames(pctiles) <- list(c("Waiting time", "Trial accrual"),c("2.5%","50%","97.5%")) pctiles[1,] <- 1/qgamma(c(0.975,0.5,0.025),shape=k,rate=V) pctiles[2,1] <- lcl[100] pctiles[2,2] <- mid[100] pctiles[2,3] <- ucl[100] list(pct=pctiles,sa=simulated.accrual) }

The function calls that produced the three figures shown above are

d0 <- accrual.plot(n=350,T=3,P=0.5,m= 0,tm= 0,nmax=500) d1 <- accrual.plot(n=350,T=3,P=0, m=41,tm=239/365,nmax=500) d2 <- accrual.plot(n=350,T=3,P=0.5,m=41,tm=239/365,nmax=500)

Future work: The model we propose is easily extended in a variety of ways:

Use of alternatives to the exponential distribution for modeling waiting times.

Examination of alternative prior distributions.

Use hierarchical models to predict accrual across multiple centers in a multi-center trial.

Conclusion: Predicting accrual across a fixed time period of a planned study is not an easy procedure. As demonstrated in this paper there are uncertainties that should be incorporated into this prediction. We hope that the method in this paper encourages investigators to account for these uncertainties when planning and monitoring accrual in a clinical trial.

Case study of accrual in a clinical trial, part 2 (October 9, 2007)

I received additional accrual data on a clinical trial I am monitoring. To review, the trial started on August 28, 2007 and will continue until January 31, 2008, for a total of 22 weeks. The researcher thinks that he might be able to get 3 patients per week over a 22 week trial (66 total), but he is very confident that he would get at least 2 patients per week (44 total). The confidence in the estimate of 3 patients per week was rated as 5 on a 10 point scale. After one week, a single patient has entered the study. No patients enter on weeks 2, 3, or 4. On week 5, three patients enter the study. On week 6, one more patient enters for a total of 5 patients.

What would the estimated accrual look like after each week of the study? For each graph, I will include a red line that shows what an accrual rate of 3 patients per week would look like and a blue line that shows what an accrual rate of 2 patients per week would look like. Let's start with uncertainty as reflected during the initial planning phase.

This is the prior estimate of final sample size. It is centered around 66, but there is substantial uncertainty about the final sample size. The 95% probability limits for the predicted sample size range from 42 to 95. Note that the pessimistic line in blue corresponds roughly to the lower limit of the predicted final sample size.

The first week produces only one patient rather than 3. The estimated overall sample size has shrunk from 66 to 60, and the 95% probability interval is also slightly smaller.

After two weeks and still only one patient, the median predicted sample size slips to 52.

After a third week with no additional patients, the median predicted sample size continues to slip.

After a fourth week of no activity, the medina predicted sample size falls all the way to 41. The estimated sample size is even less that the projection of the pessimistic rate of two per week.

In fifth week, three patients entered the study, but the predicted sample size stays pretty much the same. The researcher did get the back on track, but did not get the "catch up" growth that this study needed.

In week 6, with only one additional patient, the median predicted sample size declines again. If you compare this graph to the graph showing the uncertainty in predicted sample size prior to the start of the study, there is an important feature beyond the downward shift in the median predicted sample size. Notice that the 95% probability limits had a total width of 52 at the start of the study and only 32 after 6 weeks. As more time elapses and you gain more experience with understanding exactly how fast or slow patients accrue, you get more precision in your prediction about the final sample size.

This webpage was written by Steve Simon on 2007-10-09, edited by Steve Simon, and was last modified on 2008-07-08. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page. Category: Accrual problems in clinical trials

Monitoring refusals and exclusions in a clinical trial (May 1, 2008 with major update on May 29, 2008).

Someone sent me an email asking about the work that Byron Gajewski and I have done on monitoring accrual patterns in clinical trials. She had been doing something similar at her job and wanted to see if we could collaborate.

In her situation, the major issue was

the number of patients who made an initial contact but did not keep their first appointment,

the number of patients who kept the appointment, but refused to sign the consent form once they realized what the study was about, and

the number of patients who did sign the consent form, but who did not meet the inclusion criteria once the initial screening was done.

Her data is interesting because her company recruits to multiple centers. The accrual pattern is also interesting, because in her job, she sees a sudden burst of recruits once an ad is placed, this will settle down and then another burst will occur when another ad is placed.

The loss of subjects during the course of recruitment is quite severe. Only one in every 20 patients makes it from the initial contact through all of the steps to final enrolment and randomization in the study. Such a large loss is probably unavoidable, because you can't bypass the consent process and you can't let just anyone into these studies. But clearly this loss has tremendous economic implications and we need to carefully plan for these losses and monitor them during the course of the study.

The geometric distribution provides an excellent approach to model the losses of patients prior to enrollment. The geometric distribution can be characterized as the number of independent Bernoulli trials observed until the first success. The probability mass function of the geometric distribution is

for k=1, 2, 3, . . .

Assume that there are k stages to the screening process from initial contact to final enrollment. At the initial planning stage for a study, we would ask the researcher to estimate the rejection/refusal rate at each stage, R_i or alternately the rate of success at each stage, S_i. Of course

Let N_k represent the desired number of patients enrolled. The number needed at the initial contact would be

This presumes, of course, that our estimates of each rate is perfectly accurate. There are some advantages to using a Bayesian approach, so we would also solicit a prior distribution on each rate. The simplest prior distribution is a beta distribution. The actual process of soliciting a prior distribution is a topic for another time. Let us suppose that the prior distribution is

After n1 patients have been recruited into the trial, with s1 successes and r1 rejections/refusals at the first stage, s2 successes and r2 refusals at the second stage, etc., we would have a posterior distribution

The expected value of the posterior distribution is

which can be written as a weighted average

You can also simulate the number of initial contacts needed to enroll patients N_k by randomly drawing values from the posterior distributions and calculating sums of geometric distributions.

Example: This example is based on retrospective data, but will be treated as if the planning was done prospectively. A clinical trial needs to enroll 414 patients. The researcher notes that there are three stages at which rejection or refusal could occur.

Stage 1: A volunteer could makes an initial contact, but does not set up a first appointment, cancels the first appointment, or fails to show up at the first appointment.

Stage 2: At the initial appointment, the volunteer reviews the details of the study and does not want to join the study.

Stage 3: After the consent form is signed, the volunteer gets some medical tests done. These tests show that the patient is ineligible for the trial.

The researcher estimates the probability of success at each of the three stages to be 20%, 90%, and 50%. The overall rate of success (surviving all three stages) is simply the product of these three probabilities (9%). This is a very small number, but represents the realities of conducting clinical trials. We cannot coerce people to join these trials, and it may be dangerous to set too broad an inclusion criteria. The time that the trial will take clearly is critically dependent on these values and even small declines in the success probabilities could produce major delays. It, therefore, becomes critical to plan these values carefully and to monitor them closely.

Since this is retrospective data, the success probabilities were deliberately chosen to be inaccurate. The actual success probabilities at the end of the study were quite different. It took 8,138 initial contact to get 2,449 volunteers to show up at the initial appointment (30% success rate). There were 2,084 who signed the consent form (85% success rate). Only 414 of these met the inclusion criteria (20%). Two of the initial estimates are too high and one is too low, but overall the projection is unduly optimistic because the overall success rate is 5%, much lower than the prior estimate of 9%.

A Bayesian model for these rates provides some valuable insights, and it is worth some effort to try to elicit an informative prior. Clearly the researchers on this project are not total novices and can offer some insights about their degree of certainty about their initial estimates of the success probabilities.

The process of eliciting an informative prior is very important but it is also very complex. I will defer discussion of this and place it in a separate web page. For now, assume that this process produces three beta distributions: Beta(A1=10,B1=40), Beta(A2=45,B2=5), and Beta(A3=25,B3=25). With these prior distributions, you can simulate how the trial will behave.

The graph shown above represents a simulation using 1000 replications. The median number of patients needed at the initial stage in order to get 414 volunteers who are willing and able to enrol in the study is 4,690. There is substantial uncertainty. There is a 2.5% probabilities that it could be 2,673 or smaller and a 2.5% probability that it could be 9,964 or larger. The histogram at the top of the graph shows the actual distribution of these values. The prior distributions represent the degree of uncertainty about the initial estimates and 95% credible intervals (CrI) are one way of displaying this uncertainty. The 95%CrI for the success probability at stage 1 (20%) ranges from 10% to 32%. At stage 2 (90% success probability), the interval is 80% to 97%. At stage 3 (50% success probability), the interval is 36% to 64%. The overall success rate (9%) has an interval from 4% to 16%.

The median projection of 4,690 falls far short of what actually happened in this clinical trial, of course, because the initial success probabilities were in general too optimistic. The Bayesian model will provide revised estimates of this projection as data accumulates during the trial.

This trial started very rapidly and after seven days, the 18 patients were enrolled in the study. There were some losses along the way to get these 18 patients into the trial. 61 volunteers did not meet the inclusion criteria (18/79 or 23% success at stage 3). 14 patients did not sign a consent form (79/93 or 85% success at stage 2). 222 did not make or keep their initial appointment (93/315 or 30% success at stage 1). The overall success rate (18/315 or 6%) is much lower than 9%.

The graph above shows the Bayesian simulation combining the prior estimates with the observed data. Notice that the 95% CrI for Stage 3 has declined markedly as has the interval for the overall success rate. The interval for Stage 1 has increased. The projection for total number of patients needed to recruit initially to insure that 414 enroll in the trial is also much higher (though still short of the 8,139 observed at the end of the trial).

After 14 days, things have heated up even more. There are now 93 patients enrolled in the trial, though there were 340 lost at stage 3 (93/433 or 21% success rate), 66 lost at stage 2 (433/499 or 87% success rate), and 1070 lost at stage 1 (499/1569 or 32% success rate). The overall success rate (93/1569 or 6%) is about the same as before and still much lower than the initial estimate.

The graph above shows the Bayesian model incorporating the prior distributions and the 14 days worth of data. The number of initial contacts is now estimated to be 6,923 and the 95% CrI (5,909 to 8,392) is now much narrower, leading to more confident predictions.

Here is the code that produced these graphs.

projected.number.to.recruit <- function(nk,A,B,s,r, sim.size=1000,n0.max=2*nk/prod(A/(A+B))) { # nk is the desired number of patients who survive # all the rejection/refusal steps to get into the study. # A and B are the parameters of the prior beta # distribution(s). If more than one rejection/refusal # process is being monitored then A and B are vectors. # You can conceptualize A as a prior number of successes # and B as a prior number of failures (rejections/refusals). # s and r are the observed number of successes and failures # (rejections/refusals). Again, if more than one process # is being monitored, then r and s are vectors. # The output of this function will be a vector of # simulated values for n0, the number needed at the initial # stage to guarantee nk participant at the final stage. # The output also include p0, the overall success rate # across all k stages. k <- length(A) # If A, B, s, and r all do not have the same length, # the results of this function will be unpredictable. # When I get the chance, I want to check for this # before proceeding. n0 <- rep(NA,sim.size) p0 <- rep(1,sim.size) for (i in 1:sim.size) { n.step <- nk for (j in 1:k) { pj <- rbeta(1,A[j]+s[j],B[j]+r[j]) p0[i] <- p0[i]*pj n.shortfall <- n.step-s[j] n.step <- s[j]+r[j]+ sum(1+rgeom(n.shortfall,pj)) } n0[i] <- n.step } mk <- (s+r)[k] m0 <- s[1] pred <- quantile(n0,probs=c(0.025,0.5,0.975)) layout(c(1,2,2,2)) par(mar=c(0.1,4.1,0.1,0.1)) pnr.hist <- cut(n0,seq(0,n0.max,length=40)) barplot(table(pnr.hist),horiz=FALSE,axes=F,xlab=" ",ylab=" ", space=0,col="white",names.arg=rep(" ",39)) par(mar=c(4.1,4.1,0.1,0.1)) par(las=2) plot(0,0,type="n",axes=F, xlim=c(0,n0.max),ylim=c(0,nk), xlab="Number needed to recruit", ylab="Number enrolled") axis(side=1) axis(side=2) box() polygon(x=c(mk,pred[1],pred[3]),y=c(m0,nk,nk), density=-1,border=NA,col="gray") segments(0,0,mk,m0) segments(mk,m0,pred[2],nk, col="white") text(pred,nk,round(pred),srt=90,adj=1) text(0,m0,m0) text(0,nk,nk) CrI <- matrix(NA,nrow=k+1,ncol=2) for (j in 1:k) { CrI[j,] <- qbeta(c(0.025,0.975),A[j]+s[j],B[j]+r[j]) text(n0.max,nk*(k+1-j)*0.05, paste("95%CrI for Stage ",j," (", round(CrI[j,1],2),",", round(CrI[j,2],2),")", sep=""),adj=1) } CrI[k+1,] <- quantile(p0,probs=c(0.025,0.975)) text(n0.max,0, paste("95%CrI all stages (", round(CrI[k+1,1],2),",", round(CrI[k+1,2],2),")", sep=""),adj=1) list(n0=n0,p0=p0,CrI=CrI) } A <- c(10,45,25) B <- c(40, 5,25) s00 <- rep(0,3) r00 <- rep(0,3) s07 <- c( 18, 79, 93) r07 <- c( 61, 14, 222) s14 <- c( 93, 433, 499) r14 <- c( 340, 66,1070) prior.results <- projected.number.to.recruit(414,A,B,s00,r00,n0.max=15000) day07.results <- projected.number.to.recruit(414,A,B,s07,r07,n0.max=15000) day14.results <- projected.number.to.recruit(414,A,B,s14,r14,n0.max=15000)

The next logical step is to combine and exponential accrual model with a geometric model of rejection/refusal rates.