This is a major revision of a
March 14, 2007 weblog entry.
I have been helping a colleague who is interested in monitoring the safety of kidney
biopsy events. He was kind enough to let me use his data set on my web pages in order to
illustrate some new methods for monitoring adverse events. This data set allows you to see
some examples of the use of control charts to track adverse events. Here is the raw data.
2003-01-12 ---- 2003-01-28 ---- 2003-02-01 ---- 2003-02-14 ----
2003-02-14 ---- 2003-02-15 H-NO 2003-03-09 ---- 2003-03-17 ----
2003-03-22 ---O 2003-03-25 ---- 2003-03-30 H--- 2003-03-31 ----
2003-04-05 ---- 2003-04-13 ---- 2003-04-15 --N- 2003-04-19 H-NO
2003-04-22 ---- 2003-04-27 ---- 2003-05-11 ---- 2003-05-12 ----
2003-05-13 ---- 2003-05-20 ---- 2003-05-24 ---- 2003-06-02 ----
2003-06-08 ---- 2003-06-10 ---- 2003-06-22 -I-- 2003-06-23 ----
2003-06-24 ---- 2003-07-04 ---- 2003-07-06 ---- 2003-07-15 ----
2003-07-22 ---- 2003-07-25 ---- 2003-07-26 ---- 2003-07-26 ----
2003-08-01 ---O 2003-08-24 ---- 2003-08-26 ---- 2003-08-30 ----
2003-09-26 ---O 2003-09-26 ---- 2003-09-27 ---- 2003-09-27 H-N-
2003-09-28 ---- 2003-10-10 -I-- 2003-10-10 ---- 2003-10-12 --N-
2003-10-19 -I-- 2003-10-24 ---- 2003-10-24 -I-- 2003-10-26 ----
2003-10-31 H--- 2003-11-02 ---- 2003-11-07 -I-- 2003-11-07 ----
2003-11-09 ---- 2003-11-09 ---- 2003-11-15 ---- 2003-11-17 --N-
2003-11-29 -I-- 2003-12-12 ---- 2003-12-20 ---- 2004-01-03 ----
2004-01-04 ---- 2004-01-23 ---- 2004-01-25 --N- 2004-02-08 ----
2004-02-10 ---- 2004-02-14 ---- 2004-02-15 ---- 2004-02-15 ----
2004-02-17 ---- 2004-02-20 ---- 2004-02-22 ---- 2004-03-02 ----
2004-03-19 -I-- 2004-03-22 ---- 2004-03-26 ---- 2004-03-27 H-NO
2004-03-28 ---- 2004-04-10 ---- 2004-04-18 -I-- 2004-04-25 ----
2004-04-30 ---- 2004-05-02 ---- 2004-05-11 -I-- 2004-05-22 ----
2004-05-23 ---- 2004-05-28 ---- 2004-06-08 ---- 2004-06-15 ----
2004-06-20 -I-- 2004-06-26 ---- 2004-07-05 ---- 2004-07-09 ----
2004-07-09 ---- 2004-07-11 -I-- 2004-07-13 ---- 2004-07-24 ----
2004-07-30 ---- 2004-08-01 ---- 2004-08-01 -I-- 2004-08-06 H---
2004-08-07 --N- 2004-08-10 ---- 2004-08-13 ---- 2004-09-05 ----
2004-09-12 ---- 2004-09-21 ---- 2004-10-08 ---- 2004-10-12 ----
2004-10-13 ---- 2004-10-22 ---- 2004-11-02 ---- 2004-11-07 H-NO
2004-11-14 ---- 2004-11-28 -I-- 2004-11-29 ---- 2004-12-07 ----
2004-12-10 -I-- 2004-12-12 ---- 2004-12-13 ---- 2004-12-26 ----
2004-12-26 ---- 2005-01-03 ---- 2005-01-03 ---- 2005-01-09 -I--
2005-01-13 H--- 2005-01-15 ---- 2005-01-17 ---- 2005-01-17 H---
2005-01-20 ---- 2005-01-25 H--- 2005-01-28 ---- 2005-02-08 ----
2005-02-11 --N- 2005-02-11 ---- 2005-02-14 ---- 2005-02-18 ----
2005-02-21 ---- 2005-03-01 ---- 2005-03-07 ---- 2005-03-07 ----
2005-03-18 ---- 2005-03-18 --N- 2005-03-19 H-NO 2005-03-21 ----
2005-03-25 ---- 2005-04-10 ---- 2005-04-11 ---- 2005-04-11 ----
2005-04-15 -I-- 2005-04-23 ---- 2005-04-25 HI-- 2005-04-26 ----
2005-04-26 ---- 2005-04-29 ---- 2005-05-07 ---- 2005-05-09 ----
2005-05-13 ---- 2005-05-23 ---- 2005-06-06 ---- 2005-06-06 --N-
2005-06-10 ---- 2005-06-13 ---- 2005-06-19 ---- 2005-06-20 ----
2005-06-26 ---- 2005-06-30 ---- 2005-07-08 ---- 2005-07-18 ----
2005-07-22 ---- 2005-07-31 H-NO 2005-08-15 ---- 2005-08-19 ----
2005-08-21 H-N- 2005-08-22 ---- 2005-08-28 ---- 2005-08-29 ----
2005-08-29 ---- 2005-09-12 ---- 2005-09-12 ---- 2005-09-16 ----
2005-09-19 H--O 2005-09-23 H--- 2005-09-24 H--- 2005-09-25 ----
2005-09-26 ---- 2005-09-30 ---- 2005-10-09 ---- 2005-10-16 ----
2005-10-21 H--O 2005-11-04 H--- 2005-11-07 ---- 2005-11-14 --N-
2005-11-15 ---- 2005-11-15 ---- 2005-11-26 ---- 2005-11-28 ----
2005-12-02 ---- 2005-12-12 ---- 2005-12-16 ---- 2005-12-18 ----
2006-01-01 ---- 2006-01-02 ---- 2006-01-06 --N- 2006-01-16 ----
2006-01-16 ---- 2006-01-17 ---- 2006-01-20 ---- 2006-01-22 ----
2006-02-05 ---- 2006-02-06 H--- 2006-02-13 --N- 2006-02-24 ----
2006-02-26 ---- 2006-03-12 ---- 2006-03-19 ---- 2006-03-20 ----
2006-03-22 ---- 2006-03-27 --N- 2006-04-03 ---- 2006-04-03 ----
2006-04-14 ---- 2006-04-17 ---- 2006-04-30 ---- 2006-05-01 ----
2006-05-07 ---- 2006-05-09 ---- 2006-05-12 ---- 2006-05-12 ----
2006-05-13 ---- 2006-05-14 ---- 2006-05-19 ---- 2006-05-21 --N-
2006-05-22 --N- 2006-05-26 ---- 2006-05-29 ----
The dates represent dates of the kidney biopsies for 239 consecutive biopsies. I have
shifted these dates by an arbitrary constant to protect confidentiality. Those dates with an
H represent biopsies where gross hematuria was noted (n=21). An I represents a biopsy where
an inadequate amount of tissue was obtained (n=17). An N represents a biopsy where narcotics
were required to control the pain (n=22). An O represents any other adverse event
(perforation, hematoma, fistula, transfusion needed, prolonged hospitalization, re-admission,
or graft loss, n=11). Some of these events (perforation, graft loss) never occurred in this
particular data set.
The first 170 biopsies occurred prior to a major change in procedure, the use of real time
ultra sound to help with needle positioning. The three month period from July through
September was considered a transition period. There were 20 biopsies performed during this
transition. The biopsies from October 2005 onward were considered to be part of the post
implementation phase.
The plot below shows the time when certain biopsy events occurred.
Hematuria is one of the most common adverse events, occurring about 9% of the time or
roughly one in every 11 patients. There are no trends or patterns over time.
Inadequate tissue is another serious concern. This occurs about 7.5% of the time or
roughly once in every 13 patients. Notice that no instances of inadequate tissue occurred
after the changes were made. We can't really estimate the rate very well with this data, but
we suspect that it is small (less than 2%) and it represents a sudden drop in the rate from
what we observed prior to the intervention.
Another concern is a biopsy with pain sufficient to require the use of narcotics. This
occurs roughly once in every ten patients and shows no trends or patterns over time.
All other adverse events represents a smaller piece of the puzzle, with an average of one
other adverse event for every twenty patients. Although three events did occur after the
intervention, the most recent data suggests the possibility of an improvement as the last 40
or so patients have avoided any of these other adverse events. This drop in the adverse event
rate is not large enough to allow us to claim an improvement in the process of biopsies, but
as additional data accumulates, we may be able to draw a stronger conclusion.
How can you construct these graphs?
Here are the details on how to construct the control chart for hematuria. The table below
shows each individual biopsy with an H if the biopsy resulted in hematurian and a dash (-) if
the biopsy did not.
1 2 3 4 5 6 7 8 9 0
[ ] - - - - - H - - - -
[10] H - - - - H - - - -
[20] - - - - - - - - - -
[30] - - - - - - - - - -
[40] - - - H - - - - - -
[50] - - H - - - - - - -
[60] - - - - - - - - - -
[70] - - - - - - - - - H
[80] - - - - - - - - - -
[90] - - - - - - - - - -
[100] - - - H - - - - - -
[110] - - - - - H - - - -
[120] - - - - - - - - H -
[130] - H - H - - - - - -
[140] - - - - - - H - - -
[150] - - - - H - - - - -
[160] - - - - - - - - - -
[170] - - - H - - H - - -
[180] - - - - H H H - - -
[190] - - H H - - - - - -
[200] - - - - - - - - - -
[210] - - - H - - - - - -
[220] - - - - - - - - - -
[230] - - - - - - - - -
Note that hematuria occurred with the 6th, 11th, 16th, and 44th biopsy. That means we had
to wait 6, 5, 5, and 28 patients between each event. The full list of waiting times are
6 5 5 28 9 27 24 12 13
3 2 13 8 19 3 8 1 1
6 1 20
The last hematuria event occurs with the 214th patient. What do we do with these final
biopsies. It seems a shame to ignore them, because the long stretch without hematuria does
provide information. You know that the next hematuria event could only occur with the 240th
or later patient. So the date gap is at least 26 patients, maybe more.
You can plot these date gaps, including the 26, in sequence and draw a line connecting the
consecutive data points. Use a special symbol (X) and a dotted line for the last data value
(26) because we don't know exactly what this value is.
Plotting the waiting times on a log scale is useful because waiting times are often
skewed. The plots shown above also reverse the scaling so that long waiting times are shown
at the bottom of the graph and short waiting times are at the top. With this reverse scaling,
a sudden surge in bad events (represented by a series of short waiting times) is displayed at
the top of the graph and a sudden decline (represented by a series of long waiting times) is
displayed at the bottom.
The average waiting time has a special interpretation. The waiting times form a
telescoping sum, and the sum is equal to 240, one more than the total number of biopsies. The
average simply divides this sum by the number of biopsies with hematuria (22). If you swapped
the numerator and the denominator, you would get 22/240 which is an estimate of the
probability of observing hematuria in a randomly selected biopsy (actually the estimated
probability is 22/239, but that is not worth quibbling about).
The inverse of a probability of a harmful event is known as the number needed to harm. In
this example, NNH = 10.9. This tells you that you have to perform 11 biopsies on average in
order to experience one event of hematuria.
There are several ways to compute control limits. The approach I used for this data set
was to treat the waiting times as individual values in an XmR chart. This chart requires you
to compute a moving range, a range for each successive pairs of waiting times. The first
three moving ranges are
|6- 5|= 1
|5- 5|= 0
|5-28|=23
and the entire list of moving ranges is shown below.
1 0 23 19 18 3 12 1 10
1 11 5 11 16 5 7 0 5
5 19 6
The average moving range is 8.48. The formula for the control limits is
10.9 +/- 2.66*8.48
The upper control limit is 33.5. The lower control limit is negative, an impossible value
to achieve, and is not plotted on the graph. The warning limit (or the two sigma limit) is
computed as
10.9 +/- (2/3)*2.66*8.48
The upper warning limit is 25.9 and the lower warning limit is negative.
