Statistical Evidence. Chapter 7. Where is the evidence? Searching for information.

In a book like this, it would be difficult to give a comprehensive overview of how to search the published literature to find all the research studies associated with a particular treatment or a particular disease.  You should discuss any serious literature search, though, with a professional librarian.

Here, I want to give you a quick overview of some of the issues associated with searching. I do a lot of searches to find good teaching examples. That's not quite the same thing as finding all the studies associated with a disease, but you may find some of the tricks I have learned to be helpful.

7.1 PICO format

When you are searching for information, it helps to use a structured format for your question. The PICO format works very well. To focus your question well, you should specify:

Not every question fits perfectly into this structure. You may also find that your question does not involve a comparision group.

My question about prenatal smoking and birthweights could easily be fit into the PICO format.

Does the use of smoking cessation programs (I) in women who are pregnant (P) lead to an improvement in birthweight (O) compared to simply offering advice and encouragement about the importance of quitting smoking (C)?

7.2 Search high level sources first

I was trying to track down an article that I remembered from several years ago. It was an evaluation of smoking cessation programs for pregnant mothers to try to improve the birth outcomes, especially an increase in birthweight.

Normally, it is best to go to high level sources, such as the Cochrane database, bestbets.org, or guidelines.gov first. There is a very nice Cochrane review, which we will see details about later. The search at bestbets.org was a bust, but guidelines.gov had a nice guideline

It wasn't quite what I was looking for, but it was still worth reading. This guideline had the following comments about smoking during pregnancy:

I-5 Screening for Tobacco Use - Offer Cessation -- Week: 6-8

The Working Group’s Recommendations for Women In Low Risk Pregnancy:

  1. Strongly recommend routine screening for tobacco use in pregnancy at the initial prenatal visit. For patients who smoke, recommend assessment of smoking status at each subsequent prenatal visit. (Lumley, Oliver, & Waters, 2001; Mullen et al., 1991) (QE: I; Overall Quality: Good; R: A)
  2. If the screening is positive, cessation should be strongly recommended. (Wisborg et al., 2000; Panjari et al., 1999; Dolan-Mullen, Ramirez, & Groff, 1994) (QE: I; Overall Quality: Good; R: A)
  3. There is insufficient data to recommend for or against pharmacologic therapy for tobacco cessation in pregnancy.

Notice the cryptic notes in parentheses after items 1 and 2. This a grading system on the quality of the evidence, and you have to be sure you understand the codes properly. Some systems use number codes and some use letter codes. Even worse, for some systems, low numbers or letters represent the best quality evidence and for other systems low numbers or letters represent the lowest quality evidence. So you have to read the fine print.

For this particular guideline, QE: I  represents "Evidence obtained from at least one properly randomized controlled trial." You can also discover that "Overall Quality: Good" means that the evidence is directly linked to the health outcome, as opposed to an intermediate or surrogate outcome. Finally, the phrase "R: A" means "A strong recommendation that the intervention is always indicated and acceptable."

7.3 Searching in PubMed

For a complex search in PubMed, it can sometimes help to break the search into individual pieces and then combine the pieces together. So you should  first took a look at how many references you would find if you looked at "smoking cessation" (10,621 references), and then pregnancy (570,261 references), and then birthweight (29,446 references). All of these numbers are large, which reassures you that you are searching using the right words. In contrast, if you had searched on "stopping smoking" rather than "smoking cessation," you would have only found 661 references, which is too small a base when you start combining it with the other terms.

Now combine the terms together.

The combination of all of these terms produces 69 references. When you still have a large number of references, you then have the luxury of looking for a meta-analysis of these studies to save yourself the effort of reading a bunch of individual studies. You can click on the LIMITS tab in the PubMed search, or you can add the term "Meta-Analysis [pt]" to your search criteria. This final search yields two meta-analyses

  1. Lumley J, Oliver SS, Chamberlain C, Oakley L.
    Interventions for promoting smoking cessation during pregnancy.
    Cochrane Database Syst Rev. 2004 Oct 18;(4):CD001055. Review.
    PMID: 15495004 [PubMed - indexed for MEDLINE]
  2. DiFranza JR, Lew RA.
    Effect of maternal cigarette smoking on pregnancy complications and sudden infant death syndrome.
    J Fam Pract. 1995 Apr;40(4):385-94.
    PMID: 7699353 [PubMed - indexed for MEDLINE]

 and the first one listed is the one I wanted.

PubMed tags

The "[pt]" is an example of a PubMed tag. These tags allow you to specify exactly what part of the PubMed record you want to search for.

For example, I was searching on Schiavo to see if there were any interesting commentaries about this case in PubMed Central. But when I searched simply on "Schiavo," PubMed gave me 10 articles where one of the authors had a last name of Schiavo. I could search instead for "Schiavo [ti]" which would limit my search to those articles where the word "Schiavo" appeared in the title of the publication.

Other useful tags in PubMed are:

The [ta] tag is very useful when the name of the journal (e.g., Circulation) is also a commonly used medical term. You can also search by the journal's ISSN number if you know it.

The [dp] tag uses the YYYY/MM/DD format and you can specify only the year or only the year/month. You can also specify a range using a colon between the two dates. Finally, you can search the last X days by specifying "last X days [dp]" in your search. This also works for the last X months and the last X years. The [tiab] tag allows you to search for words in either the title or the abstract.

The "free full text [sb]" tag will retrieve only those articles with free full text on the web. For example, searching on "L'Abbe plot" yielded seven references, but when I searched on "free full text [sb] L'Abbe plot" I got the two articles which had free full text on the web. When you are looking for good teaching examples, it is wonderful to search for publications that you can link to directly, knowing that everyone who reads your pages will be able to view the full article if they so desire.

You can also search for certain publication types such as Review, Clinical Trial, or Editorial using the [pt] tag.

MeSH terms

You could have refined this search using MeSH terms. MeSH terms are especially helpful for a term like "heart attack" that is too vague from a medical viewpoint. This vagueness causes two problems. First, "heart" and "attack" are common words with multiple meanings and uses. These words may appear far separated from each other in an article that is totally unrelated to heart attacks. Second, the words "heart attack" could describe conditions like "Myocardial Infarction" or "Coronary Arteriosclerosis" or "Coronary Thrombosis" or "Congestive Heart Failure" and would thus produce too broad a range of conditions.

PubMed filters

Professional who use PubMed regularly have developed specialized filters that

Other considerations

Some other considerations to improve your search include:

The search strategy I describe here works well on other systems besides PubMed.

Further reading

7.4 Searching the Internet

When you are trying to find information, your first choice should always be the peer-reviewed literature. The peer-review process is not perfect, but it does eliminate a large number of unsupported research claims. The same cannot be said about the Internet. Still, there are times when the Internet can provide fast and accurate answers. For example, if I am looking for a definition of an alternative medicine therapy that is acceptable to the people who practice that therapy, the Internet will link with various organizations that promote this practice. I also find the Internet helpful for tutorials on things like how to use PubMed.

There are a wide range of search engines on the Internet, and although I usually try Google first, I have found other search engines to work just as well.

You should learn a bit about Boolean logic-how to use words like AND, OR, and NOT to refine your search. Here are some brief tutorials

Gauging the quality of Internet resources

Anyone can publish on the Internet, and there is very little if any attempt to monitor for misleading or even fraudulent claims. Most of the people like me who publish information on the Internet do so without any overt bias or ill intentions, but you still need to be cautious. How do you evaluate a web site to see if it provides credible and reliable health information?

There are a variety of things that you should look at. Here are some guidelines loosely based on the Health on the Net Foundation's code of conduct at www.hon.ch/HONcode/Conduct.html

  1. Is the advice being offered by a medically trained professional?
  2. Is the advice intended to support rather than replace the care you get from your doctor?
  3. Is your confidentiality respected?
  4. Is the advice backed up by references and hyperlinks to the original sources?
  5. Are the claims presented in a fair and balanced manner?
  6. Is it obvious who wrote the material?
  7. Are commercial sponsors and non-commercial supporters clearly identified?
  8. Is the material on the web page developed independently of any advertising or other sources of revenue?

Be especially aware of material presented by professional organizations and advocacy groups. They do offer a lot of valuable and important information, but are unlikely to produce material that discusses limitations, side effects, and other problems.

Google Scholar

The Google web site introduced a new feature in 2004 called Google Scholar, scholar.google.com, in response to complaints that the Google search engine did not find a lot of information stored in library databases. The Google Scholar site will focus on scholarly resources that have been through peer review. It is also a nice way to supplement a PubMed search, because Google Scholar includes all of the PubMed files in its search. This allows you to use some advanced search capabilities that Google has developed which may not be easily available in PubMed. Of course, PubMed has certain search features not available in Google Scholar.

Often a paper published in the peer review literature can also be found at other locations, such as the web pages of the authors. Google Scholar will show you all of the locations of an article, and sometimes you can get the full text for free at the author's web pages.

Goggle Scholar offers a nice feature that allows you to search for papers that cite a classic or seminal reference. I was interested in finding some recent discussion about spectrum bias, the tendency for some research studies to overstate sensitivity and specificity because they fail to include the full spectrum of disease severity. A search on spectrum bias yields a variety of references. Here are the first five.

Spectrum bias in the evaluation of diagnostic tests: lessons from the rapid dipstick test for … MS Lachs, I Nachamkin, PH Edelstein, J Goldman, AR … - Cited by 57 - Web Search ... Spectrum bias in the evaluation of diagnostic tests: lessons from the rapid dipstick test for urinary tract infection. Lachs MS, Nachamkin ... Ann Intern Med, 1992 - ncbi.nlm.nih.gov

Body mass index compared to dual-energy X-ray absorptiometry: evidence for a spectrum bias F Curtin, A Morabia, C Pichard, DO Slosman - Cited by 22 - Web Search Click here to read Body mass index compared to dual-energy x-ray absorptiometry: evidence for a spectrum bias. Curtin F, Morabia A, Pichard C, Slosman DO. ... J Clin Epidemiol, 1997 - ncbi.nlm.nih.gov

Spectrum bias or spectrum effect? Subgroup variation in diagnostic test evaluation SA Mulherin, WC Miller - Cited by 14 - Web Search ... ACADEMIA AND CLINIC. Spectrum Bias or Spectrum Effect? Subgroup Variation in Diagnostic Test Evaluation. ... Origins of the Concept of Spectrum Bias. ... Ann Intern Med, 2002 - annals.org - annals.org - annals.org - ncbi.nlm.nih.gov - all 5 versions »

The effect of spectrum bias on the utility of magnetic resonance imaging and evoked potentials in … PW O'Connor, CM Tansay, AS Detsky, AI Mushlin, W … - Cited by 10 - Web Search ... ARTICLES. The effect of spectrum bias on the utility of magnetic resonance imaging and evoked potentials in the diagnosis of suspected multiple sclerosis. ... Ann Intern Med, 2004 - neurology.org - neurology.org - ncbi.nlm.nih.gov

Problems of spectrum and bias in evaluating the efficacy of diagnostic tests DF Ransohoff, AR Feinstein - Cited by 220 - Web Search Original Article from The New England Journal of Medicine -- Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. ... N Engl J Med, 1978 - content.nejm.org - ncbi.nlm.nih.gov

The fifth article is obviously a classic, since it was cited by 220 other papers. If you click on the link text "Cited by 220" you will get all of these papers.

You can also do a similar thing with web pages. For example, you can find all the pages that link to a key page. For example, I routinely refer to the Skeptic's Dictionary (www.skepdic.com) to find material critical of various alternative medicine therapies. It's an easy way to get a different perspective from all the web sites that promote alternative medicine. If you wanted to find additional skeptical resources, you could search for all the web pages that link to skepdic.com. Most search engines will let you do this. In google, you would just search on "link:www.skepdic.com."

Sensitive searches versus specific searches

Whenever you search for information, you have to worry about false positives and false negatives. A false positive is an article that appears on your list but it isn't relevant to what you are looking for. A false negative is an article that does not appear on your list, but that is relevant. There is a cost for both false positives and false negatives and you need to think carefully about which is more of a problem for you.

I often search through PubMed or the Internet for interesting teaching examples. If I miss a good example, that's usually okay because there are plenty of others out there. So I find that false negatives are not a serious concern. False positives, though, are more of a problem because they take a lot of time to sort through.

The people conducting a systematic overview, however, have the opposite problem. They don't want to leave out an important study so they try their hardest to get an all inclusive list. If that means having to sort through a long list, that's just part of the price of assuring a comprehensive search.

Before you conduct your own search, decide what is most important to you. If false positives are the more serious concern, strive for a narrow, specific search. Put in a lot of qualifiers and limitations, and make sure you use "AND" a lot. This helps insure that you get a short list that is easy to work through. If false negatives are the more serious concern, then strive for a broad, specific search. Be loose with your search limits and include a wide range of synonyms connected by "OR."

Other tips:

On your own

1. You have a teenager who is trying to quit smoking and you want to see how group therapy might help compared to a nicotine replacement therapy like a patch or nasal spray. Write out a well focused question using the PICO format.

2. I recently attended a seminar on Reiki therapy. To prepare for the meeting, I wanted to see if there was any published evidence about whether this therapy works. Search for any peer-review articles about Reiki. Try first to find a systematic overview or meta-analysis. If you cannot find such an article, try to find a randomized trial.

3. Repeat this process using Therapeutic Touch.

4. Starting in late 2004, a widely publicized series of randomized trials showed that certain drugs known as Cox-2 inhibitors had an  increased risk of cardiac side effects. Was there any published data prior to 2004 that might suggest that these drugs had an increased risk? Write out an inquiry using the PICO format and search on PubMed for any randomized clinical trials on Cox-2 inhibitors that might answer this question.

5. Perform a web search on the phrase "Reiki therapy" or "Cognitive Behavioral Therapy" and examine the first ten sites that appear. How many of these pages meet the standards of the HON Code? How far down in the search list do you have to go before you find a source about Reiki therapy that you feel is fair and balanced?

This page was written. Do not bookmark this page--it will disappear by the end of 2004. This page was last modified on 08/18/2005. dot edu or click on the email link at the top of the page.