Stats
Selective reporting of research findings (March 14, 2005)
Category: Publication bias
I have talked extensively about publication bias in my weblog
and address this issue in detail on my book on
Statistical Evidence which I hope to finish sometime soon.
A related problem is when researchers decide to report or not report
particular data analyses based on how impressive the results appear. This is
called publication bias in situ (PBIS) by Phillips 2004.
Examples of this problem occur when researchers have discretion in
(1) Which exposures and outcomes to consider in datasets with many
variables.
(2) Which functional forms to use to represent variables (e.g., how to
divide continuous variables into categories).
(3) Whether to conduct separate analyses by subgroup, and which subgroup
results to emphasize.
and they then focus their attention on that analyses which produce the more
interesting results. The more interesting results, of course, are those
results which tend to show that a new drug or therapy is better and the less
interesting results are those that tend to show that the new drug or therapy
is about the same as the standard drug or therapy. I make a joke about this
sort of thing when I tell people how wonderful statistical software is. It
allows you to run ten separate analyses and then choose the one that gives
you the smallest p-value.
Evaluating PBIS is difficult because you normally don't have access to the
original research protocol to see what the researchers had originally
intended.
07/08/2008.
