Cherry picking the literature (December 20, 2006) Category: Systematic overviews

I have a relative who loves to send me articles supporting a particular religious and political viewpoint that he endorses. While that viewpoint he espouses is usually conservative, the problems with the articles he cites are problems that plague both sides. These articles always have an impressive bibliography, as if to say "Look! It was published and peer-reviewed, so it must be true." The problem with these articles though is that the bibliography was created using a process called "cherry picking."

Cherry picking is selecting those articles that support your viewpoint and ignoring the ones that do not support your viewpoint. How can you tell when someone is cherry picking the literature?

I dislike examining motives because it is subjective, but before you do any evaluation, at least make a cursory attempt to discern the prevailing political or religious viewpoint of the author. It happens most often with people who have a strong motivation to demonstrate a particular outcome. Strongly motivated people are still capable of producing objective findings, but the tendency to be selective seems to be associated with extreme viewpoints. The more extreme the political or religious persuasion of the author, the more important it is for you to look for evidence of objectivity as described below.

Second, look for evidence of a systematic effort to identify ALL of the relevant research. There are various publicly available tools like PubMed that allow you to "google" the peer-reviewed literature. Beware if the author fails to explain the process of article selection.

Third, look for an objective discussion of the strengths and weaknesses of each article found. Beware if the author reserves his/her criticisms only for contradictory research, and fails to note the weaknesses that are present in the supportive studies.

Finally, beware of anyone who accepts all research findings uncritically or someone who nitpicks to excess. The truth is somewhere in the middle. All research has flaws and limitations, but much of the research is still informative in spite of these flaws. The latter flaw, nitpicking, is the more common one. Someone who is motivated to discredit the entire research enterprise is usually up to serious mischief.

So don't be fooled by an impressive bibliography. The process that was used to create that bibliography is important.

07/08/2008.