When a client asks for a bad analysis (March 24, 2008).

I received an email from someone who was being asked to perform a subgroup analysis that is likely to produce confusing and counter-intuitive results. I was asked to help draft some language to convince the client that this was a bad idea.

When I was younger, I had an image of myself as a crusading statistician who would fight for truth, justice, and the American way. One of my professors at the University of Iowa described statisticians as "guardians of the scientific method."

It all sound quite alluring but as I have gotten older, I have mellowed quite a bit. Part of it comes from the bruises and scrapes of some of my "battles" on behalf of some unnamed ideal. Part of it comes from a recognition that scientists understand and appreciate the limitations of many statistical approaches almost as well as we statisticians do.

So my advice was to run the subgroup analysis, include all the appropriate cautionary language and (if you have the authority) insist that the cautionary language be included in any publication. In my mind, there is no sin in running an analysis that has serious limitations as long as you are honest about those limitations in any presentation or publication.

That doesn't mean that I don't try to guide clients towards the analysis that I think is best. I try to be persuasive without being authoritarian. And there are some analyses that I just won't do. But my general tendency these days is to give the client what he/she wants. As long as they understand what they are getting, I don't see a problem with this.

2008-07-14. Category: Human side of statistics, Category: Multiple comparisons