This page lists some materials that I plan to incorporate into "Statistical Evidence."

Patients reactions to finding out they were in the placebo group (May 11, 2005)

A lot of people have written a lot of things about the use of placebos in research, but one group that hasn't been heard from nearly enough is the patients themselves. A recently published article has changed that trend.

This study was a semi-structured questionnaire given to 42 patients who received placebo in a study of corticosteroid for heel pain. The authors cite a publication that found that more than half of the research studies did not tell anything to their placebo patients after the study ended and that the most common reason given is that such a thought had never even occurred to the researchers.

This is rather sad when you think about it. Patients who participate in research give us a wonderful gift. They cede authority over what treatment they will get. And they give this authority, not to their doctor, but to a random device like a coin flip. These volunteers are willing to endure a fair amount of inconvenience, possibly endure additional pain (for getting extra blood samples, for example), and maybe even suffer an increased level of risk in the study. One way you can show your gratitude to them is by letting know at the earliest possible moment what treatment they did receive. Don't do this during the study, of course, because that destroys all the benefits of blinding and ruins the rationale for using a placebo group in the study. But after the study is over, you should offer this information.

The authors do cite one example, though, where some patients were on a placebo for treatment for depression. They responded well, as you might expect from the placebo effect, but when they were told afterwards that they were on a placebo, most of them relapsed. In another trial, patients who received a placebo surgery were angry when they found out afterwards and demanded the active surgery, even though the research showed that the active surgery was no better than placebo.

In the interviews, the researchers found that patients expectations about the trial varied but that some were hopeful to achieve pain relief. Is this an example of therapeutic misconception? The researchers did not discuss therapeutic misconception, which is the tendency for people participating in research to believe that participation in a research trial has to be beneficial to them, even when they are told otherwise. In an article about the special problems of research in rehabilitation medicine, the author notes that

These patients, as well, are likely to agree to participate in just about any type of research in the hope that it will provide some degree of benefit for them, even if they are told this is not likely to be the case. As with spinal cord injured patients, they often ask every time they come to the outpatient clinic whether or not there are any research projects under way that they can volunteer for, without any prompting by the physician. Obviously, the potential for therapeutic misconception is also quite high with this group, and this needs to be considered and addressed by the researcher. www.biomedcentral.com/1472-6939/4/2

The patients were asked to guess which treatment group they were in, and only 55% got it right. This is a question that you should ask at the end of any blinded trial to assure that something (such as a well known side effect of the active drug) did not tip off the patients prematurely.

Most of the participants (83%) welcomed the opportunity to be debriefed about the experiment. These patients also wanted to know about the study results. Again, this should be done in all research. If a subject is participating in research because they want to add to the body of knowledge, one way you can reward them is by telling them what exactly did the research showed.

Some patients were a bit embarrassed when they found out that the beneficial effects they saw were the result of a placebo, but others were amazed and fascinated at experiencing the placebo effect.

There is room for a lot more research like this. We speculate all the time about what practices are ethical and what practices are not. I'm as guilty of this as anyone. But the real test of what is ethical has to incorporate the values, expectations, and beliefs of the research subjects themselves.

Interesting web links and quotes for the month of May (updated May 6, 2005)

Improving Medical Statistics and the Interpretation of Medical Studies. Roehm E. Accessed on 2005-05-06. There is considerable variability in the information that the public and physicians receive regarding the results of medical trials. One contributing factor is the incorrect application of statistics in the medical literature. A more common source of error is that the conclusions of a study are not always a reasonable reflection of the data presented. The following includes examples of the misuse of statistics and inappropriate conclusions in the medical literature. Some suggestions for improvement are included. www.improvingmedicalstatistics.com/

Searching the literature (May 3, 2005)

Next week, our hospital will have a seminar on Reiki. I thought it would be useful to see what evidence there was on this technique. My search included access to several resources through a system called Ovid. Unfortunately, Ovid requires a fee, so it is not available to the general public. There are free sources of information, but these are scattered around and provide less comprehensive coverage than Ovid. For example, the site www.cohrane.org offers you the abstracts, but not the full text of the Cochrane Database of Systematic Reviews.

My search was a bit ill focused, unfortunately. Normally, when you do a search, you specify four things:

but since I am just curious about Reiki in general, I only have the I part of PICO defined. It would be better to ask a question like:

In adults with lower back pain (P), does the use of Reiki therapy (I) compared to physical massage (C) affect the patient's self report of pain?

There are several good guides to the PICO format:

It's a good idea to start at the highest level of the evidence hierarchy and work your way down. So I went first to the Cochrane Database of Systematic Reviews and looked for any review including the word "Reiki."

There were five reviews, but none of these looked helpful. Two of them used Reiki as an exclusion criteria and the other three allowed Reiki as a cointervention but did not evaluate its effects.

Then I went to an Ovid link to "All EBM Reviews - Cochrane DSR, ACP Journal Club, DARE, and CCTR." A search here yielded 16 links including the five I had found earlier.

Several results looked promising:

The first two are critical reviews of a paper:

that looks at several therapies including Reiki. You can get a pdf version of this paper but when you read this paper, it turns out that none of the papers selected for this systematic review incorporated the practice of Reiki.

All of the remaining links refer to single trials, so I put them on hold for now to see if I could something broader instead.

A peek at the National Guideline Clearinghouse (free to the public at www.guidelines.gov) yielded no hits on "Reiki." I did not hold out a lot of hope here, because this is a resource better suited when you have a well defined patient population that you need to manage. Since I am totally missing the P portion of PICO, I am at a serious disadvantage here.

A review of the Best Bets website (www.bestbets.org -- please don't put a .com here because it will lead to a gambling site), also yielded no publications relating to Reiki.

Then I went to PubMed. A search for "Reiki" with a limit of publication type to meta-analysis yielded two publications, both of which focused on therapeutic touch rather than Reiki.

A search without limits yielded 381 articles, too many to skim through.

It may be time to think about single randomized trials again, so I reran the "Reiki" search with publication type of randomized clinical trials. This yielded 28 references, most of which talked about therapeutic touch rather than Reiki.

I was curious why I was getting so many articles about therapeutic touch. It turns out that Medline apparently lumps Reiki in with therapeutic touch as a MeSH term. To look for references that use the word "Reiki" rather than the MeSH term associated with Reiki, you need to use [tw], the text word tag.

If you do a search on "Reiki [tw]" without any limits, you get 52 articles rather than 381. Now restrict this list to publication type of randomized controlled trial, and you get six references. Most of these are articles that we found earlier.

So what did I learn? There may be a few good systematic reviews out there, though they seem to focus on therapeutic touch rather than Reiki. There are a handful of randomized trials out there that mention Reiki directly instead of therapeutic touch. I have not had time to critically appraise any of these articles.

Postscript (May 4, 2005): I asked for some comments and suggestions from the Evidence Based Health mailgroup, and got two responses right away. One person pointed out an evidence based review

Meta-analysis talk (May 2, 2005)

Today I gave a talk on meta-analysis. I started with a general overview of meta-analysis, and then discussed some of the particular problems associated with a meta-analysis of a diagnostic test.

The discussion was quite lively, and wide-ranging. At one point, we were talking about how to respond to a bad referee report for a paper you are trying to get published. I mentioned how it is difficult to argue with a referee although a few times I have had to resort to a "my referee is an idiot" defense. Most of the time, I just try to go along with what the referee says unless their suggestions do violence to the statistics.

One of the other people mentioned that he would often cite a paper by Moye published in 2001 in Circulation when arguing with a referee about changing the analysis. It turns out that I had this paper in my files already. The full citation is

and it discusses the problems with unwarranted changes in a research protocol. I may have to try this the next time I have to respond to a referee report. Actually, it is usually not my paper. Most of the time, someone comes to me with a referee report on a paper I have never seen before and asks for help in responding to the referee's critical comments.

Dr. Moye has also written an excellent book:

which has a lot of lively and entertaining examples in its pages.

We had a lot of interesting comments during my talk, and I agreed to give another talk on meta-analysis at a later date. I would just give a non-technical overview based on my chapter:

in the book I am trying to get published. The link to this chapter will disappear when the book is published, so please do not bookmark it.

PubMed tags (April 28, 2005)

Searching in PubMed can be tricky. If you don't find what you want the first time, it may help to specify exactly what part of the PubMed record you want to search for.

For example, I was searching on Schiavo to see if there were any interesting commentaries about this case in PubMed Central. But when I searched simply on "Schiavo," PubMed gave me 10 articles where one of the authors had a last name of Schiavo. I could search instead for "Schiavo [ti]" which would limit my search to those articles where the word "Schiavo" appeared in the title of the publication.

Other useful tags in PubMed are:

The [ta] tag is very useful when the name of the journal (e.g., Circulation) is also a commonly used medical term. You can also search by the journal's ISSN number if you know it.

The [dp] tag uses the YYYY/MM/DD format and you can specify only the year or only the year/month. You can also specify a range using a colon between the two dates. Finally, you can search the last X days by specifying "last X days [dp]" in your search. This also works for the last X months and the last X years. The [tiab] tag allows you to search for words in either the title or the abstract.

The "free full text [sb]" tag will retrieve only those articles with free full text on the web. For example, searching on "L'Abbe plot" yielded seven references, but when I searched on "free full text [sb] L'Abbe plot" I got the two articles which had free full text on the web. When you are looking for good teaching examples, it is wonderful to search for publications that you can link to directly, knowing that everyone who reads your pages will be able to view the full article if they so desire.

You can also search for certain publication types such as Review, Clinical Trial, or Editorial using the [pt] tag.

The [tw] tag searches for words in the title, abstract, and MeSH terms.

MeSH, by the way, stands for "Medical Subject Headings" and is an attempt by PubMed to create a hierarchy of medical terms to help make searches more efficient. You can learn all about MeSH at

Further reading

Searching the Internet (April 26, 2005)

A response to a question on the stat-l discussion group provided a clever way of searching the Internet. Someone asked about finding references about how to analyze ordinal data. There were several good responses, but there was one that I liked best.

This person suggested doing a citation search using Google Scholar. The way you do this is to find a key reference book. In this case, he suggested the book by Alan Agresti, Analysis of ordinal categorical data (ISBN 0471890553). Find this book on Google Scholar by searching on Agresti ordinal. Here's the results of that search.

Each result in Google Scholar also includes a cited by link. The Agresti book was cited by 228 other sources. Click on that link to find a wealth of resources related to ordinal data.

Post hoc power is never justified (April 22, 2005)

Someone wrote in and was upset that a referee was insisting on post hoc power for all the outcome measures, and he only wanted to compute post hoc power for the negative outcomes (the outcomes that did not achieve statistical significance).

The references that I cite on my web page about post hoc power are very strongly against using post hoc power for ANY outcome measure. Post hoc power is inversely related to the p-value, so any large p-value is going to automatically have a small post hoc power. For any comparison of two groups, if the p-value is larger than 0.05, the post hoc power has to be smaller than 50%.

If you were silly enough to believe that post hoc power was measuring something useful, you would then have to accept the absurd conclusion that every single negative study that was ever published was underpowered. Certainly, some negative findings occur because the sample size is too small but sometimes they are negative because nothing is going on. Not every treatment being studied is going to be effective, and not every exposure being studied is going to be harmful.

If a referee asks you to include a post hoc power calculation, just say no. Include a sentence in your paper along the lines of

and then cite 2 or 3 of these references.

Where is the confidence interval? (March 31, 2005)

A recent letter to the editor,

complains about an article claiming that a drug, citalopram, can reduce depressive symptoms

The letter writers dispute (among other things) the claim of a statistically and clinically significant reduction. In the original paper, the authors show several results, and the one that is perhaps the most important is the proportion of patients who score 28 or less on the Children's Depression Rating Scale. By this criteria, 36% of the treated patients and 24% of the control patients showed improvement.

One way to see if the results of a study are clinically significant is to present a number needed to treat plus confidence limits. The difference in percentages is 12% and when you invert it, you get a value of NNT=8.1. This means that you have to treat 8 patients with the drug in order to see one additional improvement over placebo.

Is an NNT of 8 clinically significant? I can't answer this question because I am not a psychiatrist. I will point out that you have to balance the costs of the treatment (economic costs, the inconveniences caused by having to take the medication, and the side effects that these patients will have to endure) against the benefits.

The thing to keep in mind is that all eight patients pay the costs of the drugs to accrue benefits for a single patient. If the benefit of pushing one additional patient below 28 on the Children's Depression Scale is more than 8 times as valuable as the costs that each patient endures, then a change from 24% to 36% is worthwhile from a clinical viewpoint.

Another interesting thing to consider though is the confidence interval. The original paper presents no confidence intervals, and none are mentioned the reply to the letter.

The CONSORT guidelines for reporting of randomized clinical trials recommends the use of confidence intervals for presenting research results. There are 175 journals that have endorsed the CONSORT guidelines, but not the American Journal of Psychiatry.

A crude calculation for the 95% CI gives a NNT that could be as small as 4 or as large as infinity. An infinite NNT is equivalent to no effect whatsoever.

Perhaps the authors used a more sophisticated method to determine statistical significance, but even then, I suspect that the upper limit for the confidence interval for NNT would be in the hundreds or thousands.

I shouldn't pick on these particular authors. There is a general problem with reporting confidence intervals

and discussing clinical importance.

Allegations of scientific misconduct (March 28, 2005)

A recent email in the IRBForum highlight allegations of scientific misconduct and cites a news report in BMJ,

This report describes, Dr. Gretchen LeFever, a researcher at East Virginia Medical School, who is being fired. Dr. LeFever has published controversial research on the prevalence of attention deficit hyperactivity disorder (ADHD) disorders in children.

The article describes an allegation that may have lead to her firing:

In May 2004 an anonymous whistleblower charged Dr LeFever with “scientific misconduct.” The whistleblower pointed out a discrepancy between Dr LeFever’s published report of the wording of the survey question and the actual question used for her 2002 study. In the original survey parents were asked: “Does your child have attention or hyperactivity problems, known as ADD or ADHD?” The published version of the question was “Has your child been diagnosed with attention or hyperactivity problems known as ADD or ADHD?”

There is indeed a difference between the two questions, but normally this sort of thing is handled by publishing a letter in the same journal. There is also an allegation that Dr. LeFever jeopardized the rights of children in a survey study of parents that the Institutional Review Board (IRB) had originally declared exempt from consent requirements, a ruling that was later changed

Dr. LeFever alleges that she is being fired not for misconduct or for ethical violations but because her research criticized the overuse of drugs in children with ADHD. An interesting quote from one of her critics also appears in the article.

One of her main critics is Jeffrey Katz, a clinical psychologist in Virginia Beach and the local coordinator of the Children and Adults with Attention-Deficit/Hyperactivity Disorder group. Dr Katz questioned her claim that the condition had been diagnosed in 17% of children in grades 2 to 5. He said, “When somebody like Dr LeFever makes these claims that are apparently not based on good research, it minimises a very real problem. Parents won’t bring their children in for evaluation, because they are afraid that medication will be automatically prescribed. They think it’s a bad thing and the sole treatment. But medication can have significant benefits.”

It reminds me of a similar allegation of misconduct from 20 years ago. Dr. Herbert Needleman published a paper in 1979 that showed a relationship between lead levels in children's teeth and a decline in intelligence scores. There was a dispute about some of the methods that Dr. Needleman used that escalated into a charge of scientific misconduct.

It is not surprising that Dr. Needleman's work made him a frequent target of criticism by the lead industry, or that he was once forced to defend himself against charges of scientific fraud and misconduct. Not only was he exonerated, but he fought for and won the right for those accused of such charges to an open hearing with legal representation – a right that has benefited the entire scientific community. www.heinzawards.net/recipients.asp?action=detail&recipientID=29

Not everyone was impressed with Needleman’s work, though. Critics uncovered many problems. Needleman didn’t control for the confounding factor of child’s age. Factoring in age yielded few significant results. Needleman excluded from his analysis children who were “lead poisoned” but without impaired intelligence. Needleman omitted other results that didn’t support his conclusion. Needleman was subsequently accused of scientific misconduct. Though he was ultimately not convicted of scientific misconduct, he wasn’t vindicated either. www.junkscience.com/foxnews/fn031601.htm

Who are we to believe when such strong accusations are hurled from both sides? Allegations like this are difficult for an outsider like me to sort out. Certainly, allegations of misconduct provide a weapon that has the potential for abuse. If you don't like the results of the research, the best way to attack it is to get the researcher fired for misconduct. Not even tenure can protect your job if a charge of misconduct sticks.

On the other hand, we do need protection against research fraud. GlaxoSmithKline was sued in June 2004 for their failure to disclose unfavorable information about one of their drugs, Paxil.

The civil suit, filed by New York state's attorney general, Eliot Spitzer, charges the drug company with “repeated and persistent fraud” in concealing the results of studies that suggested that paroxetine was ineffective in treating depression in adolescents. It is the first time a US public authority has pursued a drug company for misreporting trial data. GlaxoSmithKline faces US lawsuit over concealment of trial results. Dyer O. Bmj 2004: 328(7453); 1395. [Medline] [Full text] [PDF]

Certainly you have to look at the source of the accusation because any allegation of fraud might be politically motivated. A better indication of problems would come from an independent assessment from a review panel.

What about fraudulent research that never gets uncovered? There is no simple way for person like you or me to detect fraud. There are reforms in the peer-review system that can help. For example, researchers who publish a paper should submit their protocol as well as the research publication. Any deviations from the research protocol would be open for review. Not all deviations are fraudulent deviations, of course, but a peer reviewer would be able to assess this.

Protection for whistle-blowers can lead to abuse of the system, but such protection is still important to allow colleagues and subordinates of a fraudulent researcher to voice concerns without fear of losing their jobs.

Further reading

Non-random samples (March 25, 2005)

Someone sent me an email asking about a project that involved interviews of women at higher levels of management in an organization. This is a rather small group, and might require a non-random selection process. What are the limitations of a non-random sample?

It helps to remember the definition of a random sample. This is a subset of a population where every individual in the population has an equal probability of being in the subset. This is an ideal that is almost never achieved, even with the most careful research.

If your sample is not random, you have to ask yourself "What type of women would be left out, or would have a much lower probability of being in my sample?" It could be that women who work at remote sites, work at home, or who work odd hours, would be the ones that you would have no chance, or a lower chance of selecting in your sample.

Next, you have to make an assessment (usually a subjective assessment) of how the outcome measure might change in the group of women who are underrepresented or unrepresented in your sample.

In medicine, we often require some level of stability in the lives of the patients that we study. They have to live in the same place for at least six months or a year. So patients who move a lot are much less likely to be in our sample. What sort of person moves a lot? Often (but not always) it is someone who is living on the lower rungs of the socioeconomic ladder. They are often marginally employed and they have to go where the jobs are. For children, they may have to move because they are being shuttled from relative to relative because of the difficult economic circumstances of their caregivers.

When you tend to exclude patients with lower socioeconomic circumstances, you are excluding patients who have, in general, a poorer prognosis. As a result, you may end up looking at your results through rose colored glasses.

Whether you choose a random or a non-random sample depends on the balance between the difficulty and cost associated with a random sample versus the limitations that you have to endure with a non-random sample. This is often a difficult choice and depends quite a bit on your subjective assessments and your values. Of course, when a random sample is impossible, your choice is made for you.

I have some definitions of various types of samples on my web pages:

More on conflicts of interest (March 23, 2005)

I need to write up something on my very incomplete page on Intellectual conflict of interest. A review in JAMA of the report, Complementary and Alternative Medicine in the United States, which I commented on in a January 24, 2005 weblog entry, has an interesting quote:

Because research funding is limited, the report also offered several criteria for choosing which alternative treatments should be studied. They include biological plausibility of the treatment, some existing evidence of safety and effectiveness, and treatments that are targeted toward prevalent conditions that cause a substantial burden of suffering. But the committee said no therapy will meet all the criteria, and that even treatments without obvious biological plausibility should not automatically be excluded from study. One critic of the report took a swipe at that notion. “Normally grants are given to study what [agent] looks most promising,” said Stephen Barrett, MD, a retired psychiatrist and operator of the Quackwatch.org Web site. “There is very little discrimination, none by this committee, used to determine which [agents] should have priority. Research ought to be assigned on the basis of promise.” Barrett also raised conflict-of-interest issues because several of the committee members have received research grants from the National Center for Complementary and Alternative Medicine, a cosponsor of the report. “It's extremely important to look at the composition of this committee,” he said. Rebecca Voelker, IOM Points to Need for More Research, Regulation in Alternative Medicine, JAMA 293(10); 1178-1180.

This is a common complaint is that researchers are dependent on grants, and so try to present their results in a way to exaggerate the importance of their findings and to build a case for the need for more research funding, some of which will hopefully return to those same researchers.

A similar claim of conflict has been made for the U.S. Environmental Protection Agency. They have been accused of overstating the problems with the environment in order to encourage Congress to give them more money. In a criticism of  the National Environmental Education Advancement Project (NEEAP), an environmental educational program funded by EPA, Michael Sanera asks

Regardless of the effects of the NEEAP program, the larger issue is this: Should an environmental agency which has a stake in the outcome of environmental policy debates be responsible for environmental education, especially when that environmental education includes a heavy dose of political action training? It would seem only common sense that this situation represents a serious conflict of interest. Does anyone truly expect that the EPA can support a dispassionate, objective, and unbiased educational program which examines both the strengths and weaknesses of the EPA’s implementation of the Clean Air or Water Acts or the Superfund program? www.cei.org/gencon/004,02412.cfm

Surrogate outcomes (March 23, 2005)

I'm on the mailing list of the Alliance for Human Research Protection, and they highlighted a book review:

for the book

Other reviews of this book appears on the Blogcritics.org, about.com, and medicalconsumers.org websites:

and in JAMA

The author has his own website:

I have not read this book yet, but I did read several reviews. The thing that caught my eye about the review in Health Affairs was the description of a drug, Fosomax (alendronate) which treats osteoporosis. Although the published literature on this drug looks quite impressive at first, it turns out that you have to treat a large number of women in order to prevent a single hip fracture. Furthermore, although Fosomax improves bone mineral density (BMD), it turns out that

BMD mainly measures the outer layer of bone (cortical bone), whereas much of the strength of bone lies in the inner structures of trabecular bone. Drugs such as Fosamax primarily strengthen cortical rather than trabecular bone, which improves the BMD score but may not contribute as much to fracture prevention.

This is a classic example of a surrogate outcome. Most patients don't care about cortical or trabecular bone density. What they care about is having to endure a broken bone or living with dowager's hump.

A collection of randomized and non-randomized studies (March 22, 2005)

I'm updating some of my training classes to use examples from open source journals, because it is easier for me to include content of these articles directly in the web pages. An example of this is practice exercises for my training class Statistical Evidence: Apples or Oranges?

But the previous practice exercise, which used a wider range of journals had some cute articles in the mix. I'll especially miss the article on episiotomy.

Selective reporting of research findings (March 14, 2005)

I have talked extensively about publication bias in my weblog

and address this issue in detail on my book on Statistical Evidence which I hope to finish sometime soon.

A related problem is when researchers decide to report or not report particular data analyses based on how impressive the results appear. This is called publication bias in situ (PBIS) by Phillips 2004.

Examples of this problem occur when researchers have discretion in

(1) Which exposures and outcomes to consider in datasets with many variables.
(2) Which functional forms to use to represent variables (e.g., how to divide continuous variables into categories).
(3) Whether to conduct separate analyses by subgroup, and which subgroup results to emphasize.

and they then focus their attention on that analyses which produce the more interesting results. The more interesting results, of course, are those results which tend to show that a new drug or therapy is better and the less interesting results are those that tend to show that the new drug or therapy is about the same as the standard drug or therapy. I make a joke about this sort of thing when I tell people how wonderful statistical software is. It allows you to run ten separate analyses and then choose the one that gives you the smallest p-value.

Evaluating PBIS is difficult because you normally don't have access to the original research protocol to see what the researchers had originally intended.

Quality of published research (March 14, 2005)

One of the big advantages of meta-analysis is that it allows you to review a series of publications in an area to see if there are any gaps or quality shortfalls. While I was doing a few PubMed searches, I found eight articles co-authored by LL Kjaergard that showed how this works in a variety of different areas.

The classic reference that I commonly cite in my training classes is

which documented problems with selecting the wrong patients, not studying enough of them, not studying them long  enough and not measuring them properly.

Cumulative meta-analysis (updated March 11, 2005 written March 9, 2005)

This figure below, published in

shows cumulative meta-analysis, which is the cumulated effects over time of studies in the use of erythropoietin (EPO) to treat cancer related anemia. The outcome variable, the odds ratio for whether a patient requires transfusion, showed a significant benefit for EPO and that sufficient evidence had already accumulated by 1995. If such a meta-analysis had been performed back then, there would have been no need to run the additional trials. These redundant trials are bad because they wasted scarce research dollars on a topic where sufficient information had already been accumulated to answer the research question. They are also bad because half of the patients in these post-1995 trials received no treatment or placebo, even though there was enough evidence at that time to show that this is an inferior option.

There has been a suggestion that any protocol submitted to an Institutional Review Board (IRB) should include a systematic overview or meta-analysis of the previous research (see Chalmers 1996), rather than just a simple literature review, to prevent future IRBs from making the same mistake of those that approved the post-1995 studies of EPO. In some situations, that is definitely overkill, but it is a suggestion worth serious consideration in other circumstances.

Since BMC Cancer is published with an open access license, I can freely reproduce this image without getting permission, as long as I cite the source. I try to preferentially cite such resources because they make it easy to include their content on my web pages and in my teaching.

Further reading

[FrontPage Include Component]

 

[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]
[FrontPage Include Component]